Post on 13-Apr-2017
Introduction to
Celery
What is Celery?
“Celery is an asynchronous task queue based on distributed message passing.”It is a powerful, production-ready asynchronous job queue, which allows you to run time-consuming Python functions in the background.
Why should I use Celery?
User perspective• Minimize request/response cycle• Smoother user experience• Difference between pleasant and
unpleasant
Developer perspective
• Offload time/cpu intensive processes• Scalability - add workers as needed• Flexibility - many points of customization• Actively developed• Great documentation• Lots of tutorials
OUT OF THE REQUEST/RESPONSE
CYCLE• Example: Sending emails
asynchronously.
TASKS IN THEBACKGROUND.
• Example: Computational heavy jobs.• Example: Interacting with external
APIs.
PERIODIC JOBS
Celery Architecture
PRODUCER• Produces a task for the queue.
BROKER• Stores the task backlog• Answers, what work remains to be
done?• RabbitMQ, Redis, SQLAlchemy,
Django's ORM, MongoDB
WORKER• Execute and consumes tasks.• Distributed.
RESULTS BACKEND• Stores the results from our tasks.
EXAMPLE
INTERGRATING WITHDJANGO.
BEWARE OF DJANGO-CELERY
IMPORTANT SETTINGS• BROKER_URL = 'redis://localhost:6379/0'• CELERY_RESULT_BACKEND = 'redis://localhost:6379/0'• CELERY_TASK_RESULT_EXPIRES = 7*86400 # 7 days• CELERYBEAT_SCHEDULER =
"djcelery.schedulers.DatabaseScheduler"• CELERY_ACCEPT_CONTENT = ['application/json']• CELERY_TASK_SERIALIZER = 'json'• CELERY_RESULT_SERIALIZER = 'json'• BROKER_BACKEND = "redis"• BROKER_CONNECTION_TIMEOUT = 5.0• CELERY_IMPORTS = ('<your-app-name>.tasks',)
Write code in celery.py
Intialize celery in __init__.py
BEST PRACTICES
NEVER PASS OBJECTS ASARGUMENTS.
AVOID LAUNCHING SYNCHRONOUS SUBTASKS
PERIODIC TASKS
THINGS GO WRONG ?
RE-TRY!
Thanks!