Robust Distributed Task Allocation for Autonomous Multi-Agent Teams
Distributed Task Processing with Celery - PyZH
-
Upload
cesar-cardenas-desales -
Category
Technology
-
view
86 -
download
1
Transcript of Distributed Task Processing with Celery - PyZH
![Page 1: Distributed Task Processing with Celery - PyZH](https://reader033.fdocuments.us/reader033/viewer/2022051521/5875bc611a28ab33128b4971/html5/thumbnails/1.jpg)
Distributed Task Processing with Celery
Part II César Desales
![Page 2: Distributed Task Processing with Celery - PyZH](https://reader033.fdocuments.us/reader033/viewer/2022051521/5875bc611a28ab33128b4971/html5/thumbnails/2.jpg)
Task granularity
¡ Example:
download and process data from 3 different sources.
¡ Parallellism vs Communication overhead
![Page 3: Distributed Task Processing with Celery - PyZH](https://reader033.fdocuments.us/reader033/viewer/2022051521/5875bc611a28ab33128b4971/html5/thumbnails/3.jpg)
Resource leaks
¡ Why doesn't Python release the memory when I delete a large object? http://bit.ly/1yHUUKt
¡ –maxtasksperchild
¡ CELERYD_MAX_TASKS_PER_CHILD
![Page 4: Distributed Task Processing with Celery - PyZH](https://reader033.fdocuments.us/reader033/viewer/2022051521/5875bc611a28ab33128b4971/html5/thumbnails/4.jpg)
Pooling external resources
¡ Connect to DB every time a task is executed?
¡ Pass non serializable objects to workers?
![Page 5: Distributed Task Processing with Celery - PyZH](https://reader033.fdocuments.us/reader033/viewer/2022051521/5875bc611a28ab33128b4971/html5/thumbnails/5.jpg)
Limits
¡ Task dependencies -> Deadlocks
¡ Stale results
¡ Throttling
¡ Retries
¡ -time-limit or CELERYD_TASK_TIME_LIMIT
¡ –soft-time-limit or CELERYD_TASK_SOFT_TIME_LIMIT
![Page 6: Distributed Task Processing with Celery - PyZH](https://reader033.fdocuments.us/reader033/viewer/2022051521/5875bc611a28ab33128b4971/html5/thumbnails/6.jpg)
Scheduling ¡ Cron syntax
¡ Celery beat: single worker doing the scheduling
![Page 7: Distributed Task Processing with Celery - PyZH](https://reader033.fdocuments.us/reader033/viewer/2022051521/5875bc611a28ab33128b4971/html5/thumbnails/7.jpg)
Queues
¡ celery -A proj worker -Q reporting --concurrency=3
¡ celery -A proj worker -Q operations --concurrency=20
![Page 8: Distributed Task Processing with Celery - PyZH](https://reader033.fdocuments.us/reader033/viewer/2022051521/5875bc611a28ab33128b4971/html5/thumbnails/8.jpg)
Autoscaling
¡ celery --app=myapp worker --autoscale=15,8
¡ How many workers?
¡ How log is a piece of string?
![Page 9: Distributed Task Processing with Celery - PyZH](https://reader033.fdocuments.us/reader033/viewer/2022051521/5875bc611a28ab33128b4971/html5/thumbnails/9.jpg)
Monitoring ¡ celery -A proj status
¡ celery -A proj purge
¡ celery -A proj inspect stats
![Page 10: Distributed Task Processing with Celery - PyZH](https://reader033.fdocuments.us/reader033/viewer/2022051521/5875bc611a28ab33128b4971/html5/thumbnails/10.jpg)
Monitoring (flower) ¡ pip install flower
¡ celery flower --broker=redis://foobar/0 --port=5555
¡ python -m webbrowser -t "http://localhost:5555/"
![Page 11: Distributed Task Processing with Celery - PyZH](https://reader033.fdocuments.us/reader033/viewer/2022051521/5875bc611a28ab33128b4971/html5/thumbnails/11.jpg)
Results ¡ Avoid if possible
¡ CELERY_RESULT_BACKEND='redis://localhost:6379/0’
¡ CELERY_TASK_RESULT_EXPIRES=1800
¡ In synch and asynch mode
![Page 12: Distributed Task Processing with Celery - PyZH](https://reader033.fdocuments.us/reader033/viewer/2022051521/5875bc611a28ab33128b4971/html5/thumbnails/12.jpg)
Thanks!