Development of Inter-bank Foreign Exchange Market in - Informs
Inter-process data exchange in Python
-
Upload
yehor-nazarkin -
Category
Technology
-
view
390 -
download
0
Transcript of Inter-process data exchange in Python
![Page 1: Inter-process data exchange in Python](https://reader033.fdocuments.us/reader033/viewer/2022042908/58f23ac81a28abff768b45d7/html5/thumbnails/1.jpg)
COMMUNICATION BETWEEN PROCESSES
LVIV.PY #5
![Page 2: Inter-process data exchange in Python](https://reader033.fdocuments.us/reader033/viewer/2022042908/58f23ac81a28abff768b45d7/html5/thumbnails/2.jpg)
LVIV.PY #5
ABOUT
▸ Myself: Yehor Nazarkin
▸ Occupation: Python Software Engineer at DataRobot LLC
▸ Follow up: @nimnull, github.com/nimnull
![Page 3: Inter-process data exchange in Python](https://reader033.fdocuments.us/reader033/viewer/2022042908/58f23ac81a28abff768b45d7/html5/thumbnails/3.jpg)
LVIV.PY #5
IPC
▸ Client-server
▸ Signals
▸ Message queue
▸ Data Storage
SERVER
CLIENT
CLIENT
BUS
SERVER
![Page 4: Inter-process data exchange in Python](https://reader033.fdocuments.us/reader033/viewer/2022042908/58f23ac81a28abff768b45d7/html5/thumbnails/4.jpg)
LVIV.PY #5
SERIALISATION
▸ CPU/memory aligned
▸ Takes some time
▸ Variadic performance for different data
![Page 5: Inter-process data exchange in Python](https://reader033.fdocuments.us/reader033/viewer/2022042908/58f23ac81a28abff768b45d7/html5/thumbnails/5.jpg)
LVIV.PY #5
SERIALISATION
![Page 6: Inter-process data exchange in Python](https://reader033.fdocuments.us/reader033/viewer/2022042908/58f23ac81a28abff768b45d7/html5/thumbnails/6.jpg)
LVIV.PY #5
SERIALISATION
▸ Tested for json, ujson, msgpack, pickle
▸ first test — huge attributes number, small attribute size
json: 16.32 ms
ujson: 2.89 ms
msgpack: 1.40 ms
pickle: 1.49 ms
![Page 7: Inter-process data exchange in Python](https://reader033.fdocuments.us/reader033/viewer/2022042908/58f23ac81a28abff768b45d7/html5/thumbnails/7.jpg)
LVIV.PY #5
SERIALISATION
![Page 8: Inter-process data exchange in Python](https://reader033.fdocuments.us/reader033/viewer/2022042908/58f23ac81a28abff768b45d7/html5/thumbnails/8.jpg)
LVIV.PY #5
SERIALISATION
▸ Tested for json, ujson, msgpack, pickle
▸ second test — small attributes number, huge attribute size
json: 1.59 ms
ujson: 1.91 ms
msgpack: 1.22 ms
pickle: 1.20 ms
![Page 9: Inter-process data exchange in Python](https://reader033.fdocuments.us/reader033/viewer/2022042908/58f23ac81a28abff768b45d7/html5/thumbnails/9.jpg)
LVIV.PY #5
ZMQ (WHATEVER MQ)
![Page 10: Inter-process data exchange in Python](https://reader033.fdocuments.us/reader033/viewer/2022042908/58f23ac81a28abff768b45d7/html5/thumbnails/10.jpg)
LVIV.PY #5
ZMQ (WHATEVER MQ)
![Page 11: Inter-process data exchange in Python](https://reader033.fdocuments.us/reader033/viewer/2022042908/58f23ac81a28abff768b45d7/html5/thumbnails/11.jpg)
LVIV.PY #5
ZMQ (WHATEVER MQ)
▸ Matrix 1000x1000, float, ~7.68Mb
▸ Mean exchange time ~ 15.61ms
▸ Client-server
▸ Serialisation
▸ Buffer I/O
▸ Netwok latency
![Page 12: Inter-process data exchange in Python](https://reader033.fdocuments.us/reader033/viewer/2022042908/58f23ac81a28abff768b45d7/html5/thumbnails/12.jpg)
LVIV.PY #5
MULTIPROCESSING PIPES
![Page 13: Inter-process data exchange in Python](https://reader033.fdocuments.us/reader033/viewer/2022042908/58f23ac81a28abff768b45d7/html5/thumbnails/13.jpg)
LVIV.PY #5
MULTIPROCESSING PIPES
PID Command USS
# — parent process
20363 python multiproc_test.py 13368 <- keep matrix
# -- child process
20367 python multiproc_test.py 1484 <- no matrix
20367 python multiproc_test.py 10056 <- matrix passed
▸ Pipe — parent connection, child connection
▸ Memory overhead on copying data
![Page 14: Inter-process data exchange in Python](https://reader033.fdocuments.us/reader033/viewer/2022042908/58f23ac81a28abff768b45d7/html5/thumbnails/14.jpg)
LVIV.PY #5
OS.FORK()
![Page 15: Inter-process data exchange in Python](https://reader033.fdocuments.us/reader033/viewer/2022042908/58f23ac81a28abff768b45d7/html5/thumbnails/15.jpg)
LVIV.PY #5
OS.FORK()
PID Command USS PSS RSS
28181 python fork_test.py 4724 15836 29792
28182 python fork_test.py 836 11701 23560 <-- before copy on write
28182 python fork_test.py 8756 15853 23944 <-- after copy on write
▸ Copy-on-write benefits.
▸ One-way channel.
![Page 16: Inter-process data exchange in Python](https://reader033.fdocuments.us/reader033/viewer/2022042908/58f23ac81a28abff768b45d7/html5/thumbnails/16.jpg)
LVIV.PY #5
MEMORY-MAPPED FILES
![Page 17: Inter-process data exchange in Python](https://reader033.fdocuments.us/reader033/viewer/2022042908/58f23ac81a28abff768b45d7/html5/thumbnails/17.jpg)
LVIV.PY #5
MEMORY-MAPPED FILES
PID Command USS PSS RSS
16737 python mmap_test.py 26540 26914 29968 <-- before mmap write
16737 python mmap_test.py 8096 19782 34200 <-- after mmap write
16795 python mmap_test.py 5484 16920 29656 <-- child processing
▸ Limitations — string/byte ndarray type (numpy)
▸ Value data types (no refcounts, etc)
▸ Serialisation
![Page 19: Inter-process data exchange in Python](https://reader033.fdocuments.us/reader033/viewer/2022042908/58f23ac81a28abff768b45d7/html5/thumbnails/19.jpg)
LVIV.PY #5
QUESTIONS?
▸ Source: https://github.com/nimnull/lviv.py-5
▸ Follow me: @nimnull