Python memory management & Impact to memory-hungry application

Presentation on theme: "Python memory management & Impact to memory-hungry application"— Presentation transcript:

1 Python memory management & Impact to memory-hungry application
Kilik Kuo

2 Outline About Photo-Organizing Software
What is it ? Who use it ? Problems to developers Brief to Python memory management Thank God, we have Evan Jones & Nice guy Tim Peter Possible solutions ? For those which are handled in Python For those can't be handled in Python

3 About Photo-Organizing Software

4 About Photo-Organizing Software

5 About Photo-Organizing Software

6 About Photo-Organizing Software
Problems that may kill the engineer ! Rapid UI flow experience Full metadata display Fancy UI transformation Handle higher resolution 24 million (6000x4000) pixels 48 million (8000x6000) pixels ... WTF

7 Metadata - EXIF/ IPTC/ XMP...

8 Houston, we've got a Problem !!

9 Hungry for Memory Space !!!

10 Let's explore the SPACE 8000(W) x 6000(H) x 4(RGBA) x 2(BytePerChannel) = 366 MB Space Start Space End Heap

11 Does it work on PhD ? Continuous Memory Address is the KEY !!
1 BIG image should be ok, then 2, 3, 4, ... ? On 32-bit OS Limited to 2G user-mode virtual space per process Windows ASLR (Blue part - dll Image) 0x <-> 0x 0x <-> 0x 0x > You get almost 1G continuous memory space, until Python occupy the RED, and won't return to OS. On 64-bit OS 8TB user-mode by default Physical memory size is what you need to concern. 0x

12 Python Memory Management
Imagine there's a PyBank, and you wanna get mortgage Trick or Treat ...... $240, please ~ sure !

13 PMM - Allocator 四大天王 /trunk/Objects/obmalloc.c
PyObject_New/PyObject_Del

14 Inside the PyBank - PyObject Allocator
儲櫃 Free 1 255 110 14 26萬2144元 滿抽屜 4096元 262,144 / 4096 = 64 110 110 110 110 4096元 抽屜 14 14 錢袋 14 14 4096元

15 PyBank Money Management
蟒銀存款 : 21億4748萬3648元 小額放款部 (< 256 元) 大額放款部 (> 256 元) 取款步驟 告知銀行, 大額 or 小額 - 兜幾 ? 小額 檢查(放款儲櫃中的特定抽屜)裡找尋是否有剩餘符合的指定金額錢 袋 有, 從剩餘的指定金額錢袋中拿一袋給你 沒有 從(錢最少但還有滿抽屜的儲櫃)中的滿抽屜裡拿出指定金 額的錢袋 沒有儲櫃 開新的放款儲櫃, 從一個滿抽屜拿一袋指定金額錢袋 給你 大額 拿去拿去拿去 ~ 怎決定的 ？

16 儲櫃 Arena /抽屜 Pool /錢袋 Block
Free 7 110 144 2 43 5 3 13 8 64 1 255 14 23 Free Pool(滿抽屜) Free Pool 26萬2144元 A1 A0 A2 A3 Partially allocated Arena list 1 - 8 Byte Byte Byte Pool(非滿抽屜) - A2 Pool(非滿抽屜) - A0 A2 2 1 5 Used Pool list & Free Block list

17 Release the memory All used Blocks(from Pool A) return to Pool A, and all used Pools(from Arena B) return to Arena B Arena B is freed ! After Python 2.3 (Evan Jones & Nice guy Tim Peter) Something still weird ... int / float / tuple / list / dict doesn't act like above gg...

18 Back to IMPACT - Solutions ?
What can we do ? First, know where we are ! EXE 1. include "Python.h" 2. Load Python2X.dll 3. Py_Initialze() 4. Load script & DeMarshal it as PyCodeobject 5. PyEval_EvalCode(PyCodeObject) Python Interpreter 1. int, float, list, tup, dict 2. PyObjects 3. Others SWIG 1. malloc, new ... 2. Create PyObjects 3. Others

19 Inside Python World For list/ dict objects
gc.collect() is able to release the maintained free list for list/ dict Don't call it in the function where large list/dict items being created. Use subprocess to hold these objects, and retrieve information through IPC. Wrap the creation of these objects down to SWIG.

20 Outside Python World

21 Outside Python World (Cont.)

22 Outside Python World (Cont.)
Divide image raw data into chunks Re-write all related functions Decode/ Stretch/ Encode/ Rotate Pixel Operation, eg. BGRA <-> RGB. Color Management Retouch/Edit effect algorithm 族繁不及備載, 臨表涕泣, 不知所云... Performance ? SIMD Multi-threaded (OpenMP) Be aware of thread context switch overhead GPU (OpenCL ?)

23 Conclusion Be an open-minded engineer !!
You're more powerful than you think you are.

24 Q & A