Python memory management & Impact to memory-hungry application

Slides:



Advertisements
Similar presentations
allow v. wrong adj. What’s wrong? midnight n. look through guess v. deal n. big deal work out 允许;准许 有毛病;错误的 哪儿不舒服? 午夜;子夜 快速查看;浏览 猜测;估计 协议;交易 重要的事.
Advertisements

第七课:电脑和网络. 生词 上网 vs. 网上 我上网看天气预报。 今天早上看了网上的天气预报。 正式 zhèngshì (报告,会议,纪录) 他被这所学校正式录取 大桥已经落成,日内就可以正式通车 落伍 luòw ǔ 迟到 chídào 他怕迟到,六点就起床了.
IFY Parents Meeting 3 December 年12月3日家长会
Presented By: 王信傑 Ricky Wang Date:2010/10/6
台北市漢口街一段85號8樓 TEL: (02) ILTEA國際英語認證 ILTEA English Proficiency Tests 符合教育部CEF採認國際標準 符合人事行政局陞任計分標準 台北市漢口街一段85號8樓 TEL: (02)
Healthy Breakfast 第四組 電子一甲(電資一) 指導老師:高美玉 組長:B 侯昌毅
Unit 4 (1) *HAllOWEEN*.
神愛世人, 甚至將他的獨生子賜給他們, 叫一切信他的, 不至滅亡, 反得永生。 約翰3:16.
Unit 10 Section B Period 2 (3a-Self Check). What’s your biggest worries? a lot of homework quarry with my friends.
老人健康促進之 成效評估與展現 高雄市政府衛生局 李素華 2014/05/15.
专题八 书面表达.
海南软件职业技术学院 网站美工之 调 色 主讲人:潘萍 2015年11月.
我们会赞叹生命之花的绚丽和多姿,也会歌颂生命之树的烂漫和青翠,但是生命是如此脆弱……
報告即將開始.
中鸣虚拟搜救比赛项目 (一人) 现场主题创作(40%)(一人) 3D虚拟搜救(60%)(一人).
第五章 要怎麼收穫先那麼栽 教育發展與職業選擇
初中进阶 (2346 期 ) 1 版. 1. What types of bullying do you know about? Physical hitting, tripping, stealing and hair pulling Social telling other kids.
How can we be a member of the Society? You should finish the following tasks if you want to be a member of the Birdwatching Society.
院公共选修课 Leisure English 黄瑛瑛.
Unit 7 How much are these socks? Section A 第一课时.
Could you please clean your room?
Homework 4 an innovative design process model TEAM 7
Unit 4 I used to be afraid of the dark.
Module 5 Shopping 第2课时.
Unit 2 What should I do?.
U3 Where are you from?. Module 2 City life 制作:花都新华第四小学 姚婉贞
Fun with English 7A Unit 2 Main task.
GPU分散式演算法設計與單機系統模擬(第二季)
Logistics 物流 昭安國際物流園區 總經理 曾玉勤.
创建型设计模式.
但是如果你把它发给最少两个朋友。。。你将会有3年的好运气!!!
常见问题解答 II. App上重置并清空数据库之后,手机app找不到圣诞灯怎么办? I. 打开APP,发现并连接不了圣诞灯怎么办?
助動詞(Auxiliary Verbs) 重點1~9英文助動詞有: do must shall/should/ought to
Remember the five simple rules to be happy 快樂的五個簡單常規
SpringerLink 新平台介绍.
邹佳恒 第十八届全国科学计算与信息化会议 • 威海,
Lesson 44:Popular Sayings
Chapter 3 Nationality Objectives:
海豹寄給人類的一封信 自動播放.
Unit 4.
基于课程标准的校本课程教学研究 乐清中学 赵海霞.

A SMALL TRUTH TO MAKE LIFE 100%
IBM SWG Overall Introduction
Have you read Treasure Island yet?
Remember the five simple rules to be happy 快樂的五個簡單常規
2 GROUP 身體殘障人士 傷健共融—如何改善肢體殘障人士的生活素質
Guide to a successful PowerPoint design – simple is best
Imperative Sentence 祈使句
情态动词.
中国科学技术大学计算机系 陈香兰 2013Fall 第七讲 存储器管理 中国科学技术大学计算机系 陈香兰 2013Fall.
中国科学技术大学计算机系 陈香兰 Fall 2013 第三讲 线程 中国科学技术大学计算机系 陈香兰 Fall 2013.
中央社新聞— <LTTC:台灣學生英語聽說提升 讀寫相對下降>
Presentation 约翰316演示 John 3 : 16
计算机问题求解 – 论题 算法方法 2016年11月28日.
Philosophy of Life.
SpringerLink 新平台介绍.
Remember the five simple rules to be happy 快樂的五個簡單常規
Remember the five simple rules to be happy 快樂的五個簡單常規
冀教版 九年级  Look into Science!.
Remember the five simple rules to be happy 快樂的五個簡單常規
基本句型 1)主 +谓(SV)例如:   I work. 我工作。 2)主 + 谓 + 宾(SVO)例如:   She studies English. 她学英语 3)主 + 系 + 表(SVP)例如:   John is busy. 约翰忙。 。 4)主 + 动 + 宾 + 补(SVOC)例如:   Time.
M; Well, let me check again with Jane
附加疑问句 Tag Question.
TinyDB資料庫 靜宜大學資管系 楊子青.
創造思考的開發與培養.
英语单项解题思路.
P4 STEAM SCIENCE.
多维阅读第13级.
陳情表之外     with 三仁 三樂 歐陽宜璋製於 /10/23.
Presentation transcript:

Python memory management & Impact to memory-hungry application Kilik Kuo

Outline About Photo-Organizing Software What is it ? Who use it ? Problems to developers Brief to Python memory management Thank God, we have Evan Jones & Nice guy Tim Peter Possible solutions ? For those which are handled in Python For those can't be handled in Python

About Photo-Organizing Software

About Photo-Organizing Software

About Photo-Organizing Software

About Photo-Organizing Software Problems that may kill the engineer ! Rapid UI flow experience Full metadata display Fancy UI transformation Handle higher resolution 24 million (6000x4000) pixels 48 million (8000x6000) pixels ... WTF

Metadata - EXIF/ IPTC/ XMP...

Houston, we've got a Problem !!

Hungry for Memory Space !!!

Let's explore the SPACE 8000(W) x 6000(H) x 4(RGBA) x 2(BytePerChannel) = 366 MB Space Start Space End Heap

Does it work on PhD ? Continuous Memory Address is the KEY !! 1 BIG image should be ok, then 2, 3, 4, ... ? On 32-bit OS Limited to 2G user-mode virtual space per process Windows ASLR (Blue part - dll Image) 0x00000000 <-> 0x10000000 0x50000000 <-> 0x80000000 0x00000000 --> You get almost 1G continuous memory space, until Python occupy the RED, and won't return to OS. On 64-bit OS 8TB user-mode by default Physical memory size is what you need to concern. 0x80000000

Python Memory Management Imagine there's a PyBank, and you wanna get mortgage Trick or Treat ...... $240, please ~ sure !

PMM - Allocator 四大天王 /trunk/Objects/obmalloc.c PyObject_New/PyObject_Del

Inside the PyBank - PyObject Allocator 儲櫃 Free 1 255 110 14 26萬2144元 滿抽屜 4096元 262,144 / 4096 = 64 110 110 110 110 4096元 抽屜 14 14 錢袋 14 14 4096元

PyBank Money Management 蟒銀存款 : 21億4748萬3648元 小額放款部 (< 256 元) 大額放款部 (> 256 元) 取款步驟 告知銀行, 大額 or 小額 - 兜幾 ? 小額 檢查(放款儲櫃中的特定抽屜)裡找尋是否有剩餘符合的指定金額錢 袋 有, 從剩餘的指定金額錢袋中拿一袋給你 沒有 從(錢最少但還有滿抽屜的儲櫃)中的滿抽屜裡拿出指定金 額的錢袋 沒有儲櫃 開新的放款儲櫃, 從一個滿抽屜拿一袋指定金額錢袋 給你 大額 拿去拿去拿去 ~ 怎決定的 ?

儲櫃 Arena /抽屜 Pool /錢袋 Block Free 7 110 144 2 43 5 3 13 8 64 1 255 14 23 Free Pool(滿抽屜) Free Pool 26萬2144元 A1 A0 A2 A3 Partially allocated Arena list 1 - 8 Byte 2 - 16 Byte 249 - 256 Byte Pool(非滿抽屜) - A2 Pool(非滿抽屜) - A0 A2 2 1 5 Used Pool list & Free Block list

Release the memory All used Blocks(from Pool A) return to Pool A, and all used Pools(from Arena B) return to Arena B Arena B is freed ! After Python 2.3 (Evan Jones & Nice guy Tim Peter) Something still weird ... int / float / tuple / list / dict doesn't act like above gg...

Back to IMPACT - Solutions ? What can we do ? First, know where we are ! EXE 1. include "Python.h" 2. Load Python2X.dll 3. Py_Initialze() 4. Load script & DeMarshal it as PyCodeobject 5. PyEval_EvalCode(PyCodeObject) Python Interpreter 1. int, float, list, tup, dict 2. PyObjects 3. Others SWIG 1. malloc, new ... 2. Create PyObjects 3. Others

Inside Python World For list/ dict objects gc.collect() is able to release the maintained free list for list/ dict Don't call it in the function where large list/dict items being created. Use subprocess to hold these objects, and retrieve information through IPC. Wrap the creation of these objects down to SWIG.

Outside Python World

Outside Python World (Cont.)

Outside Python World (Cont.) Divide image raw data into chunks Re-write all related functions Decode/ Stretch/ Encode/ Rotate Pixel Operation, eg. BGRA <-> RGB. Color Management Retouch/Edit effect algorithm 族繁不及備載, 臨表涕泣, 不知所云... Performance ? SIMD Multi-threaded (OpenMP) Be aware of thread context switch overhead GPU (OpenCL ?)

Conclusion Be an open-minded engineer !! You're more powerful than you think you are.

Q & A