Python memory management & Impact to memory-hungry application Kilik Kuo

Outline About Photo-Organizing Software What is it ? Who use it ? Problems to developers Brief to Python memory management Thank God, we have Evan Jones & Nice guy Tim Peter Possible solutions ? For those which are handled in Python For those can't be handled in Python

About Photo-Organizing Software

About Photo-Organizing Software

About Photo-Organizing Software

About Photo-Organizing Software Problems that may kill the engineer ! Rapid UI flow experience Full metadata display Fancy UI transformation Handle higher resolution 24 million (6000x4000) pixels 48 million (8000x6000) pixels ... WTF

Metadata - EXIF/ IPTC/ XMP...

Houston, we've got a Problem !!

Hungry for Memory Space !!!

Let's explore the SPACE 8000(W) x 6000(H) x 4(RGBA) x 2(BytePerChannel) = 366 MB Space Start Space End Heap

Does it work on PhD ? Continuous Memory Address is the KEY !! 1 BIG image should be ok, then 2, 3, 4, ... ? On 32-bit OS Limited to 2G user-mode virtual space per process Windows ASLR (Blue part - dll Image) 0x00000000 <-> 0x10000000 0x50000000 <-> 0x80000000 0x00000000 --> You get almost 1G continuous memory space, until Python occupy the RED, and won't return to OS. On 64-bit OS 8TB user-mode by default Physical memory size is what you need to concern. 0x80000000

Python Memory Management Imagine there's a PyBank, and you wanna get mortgage Trick or Treat ...... $240, please ~ sure !

PMM - Allocator 四大天王 /trunk/Objects/obmalloc.c PyObject_New/PyObject_Del

Inside the PyBank - PyObject Allocator 儲櫃 Free 1 255 110 14 26萬2144元 滿抽屜 4096元 262,144 / 4096 = 64 110 110 110 110 4096元 抽屜 14 14 錢袋 14 14 4096元

PyBank Money Management 蟒銀存款 : 21億4748萬3648元 小額放款部 (< 256 元) 大額放款部 (> 256 元) 取款步驟 告知銀行, 大額 or 小額 - 兜幾 ? 小額 檢查(放款儲櫃中的特定抽屜)裡找尋是否有剩餘符合的指定金額錢 袋 有, 從剩餘的指定金額錢袋中拿一袋給你 沒有 從(錢最少但還有滿抽屜的儲櫃)中的滿抽屜裡拿出指定金 額的錢袋 沒有儲櫃 開新的放款儲櫃, 從一個滿抽屜拿一袋指定金額錢袋 給你 大額 拿去拿去拿去 ~ 怎決定的 ？

儲櫃 Arena /抽屜 Pool /錢袋 Block Free 7 110 144 2 43 5 3 13 8 64 1 255 14 23 Free Pool(滿抽屜) Free Pool 26萬2144元 A1 A0 A2 A3 Partially allocated Arena list 1 - 8 Byte 2 - 16 Byte 249 - 256 Byte Pool(非滿抽屜) - A2 Pool(非滿抽屜) - A0 A2 2 1 5 Used Pool list & Free Block list

Release the memory All used Blocks(from Pool A) return to Pool A, and all used Pools(from Arena B) return to Arena B Arena B is freed ! After Python 2.3 (Evan Jones & Nice guy Tim Peter) Something still weird ... int / float / tuple / list / dict doesn't act like above gg...

Back to IMPACT - Solutions ? What can we do ? First, know where we are ! EXE 1. include "Python.h" 2. Load Python2X.dll 3. Py_Initialze() 4. Load script & DeMarshal it as PyCodeobject 5. PyEval_EvalCode(PyCodeObject) Python Interpreter 1. int, float, list, tup, dict 2. PyObjects 3. Others SWIG 1. malloc, new ... 2. Create PyObjects 3. Others

Inside Python World For list/ dict objects gc.collect() is able to release the maintained free list for list/ dict Don't call it in the function where large list/dict items being created. Use subprocess to hold these objects, and retrieve information through IPC. Wrap the creation of these objects down to SWIG.

Outside Python World

Outside Python World (Cont.)

Outside Python World (Cont.) Divide image raw data into chunks Re-write all related functions Decode/ Stretch/ Encode/ Rotate Pixel Operation, eg. BGRA <-> RGB. Color Management Retouch/Edit effect algorithm 族繁不及備載, 臨表涕泣, 不知所云... Performance ? SIMD Multi-threaded (OpenMP) Be aware of thread context switch overhead GPU (OpenCL ?)

Conclusion Be an open-minded engineer !! You're more powerful than you think you are.

Q & A