Introduction to

Celery

What is Celery?

“Celery is an asynchronous task queue based on distributed message passing.”It is a powerful, production-ready asynchronous job queue, which allows you to run time-consuming Python functions in the background.

Why should I use Celery?

User perspective• Minimize request/response cycle• Smoother user experience• Difference between pleasant and

unpleasant

Developer perspective

• Offload time/cpu intensive processes• Scalability - add workers as needed• Flexibility - many points of customization• Actively developed• Great documentation• Lots of tutorials

OUT OF THE REQUEST/RESPONSE

CYCLE• Example: Sending emails

asynchronously.

TASKS IN THEBACKGROUND.

• Example: Computational heavy jobs.• Example: Interacting with external

APIs.

PERIODIC JOBS

Celery Architecture

PRODUCER• Produces a task for the queue.

BROKER• Stores the task backlog• Answers, what work remains to be

done?• RabbitMQ, Redis, SQLAlchemy,

Django's ORM, MongoDB

WORKER• Execute and consumes tasks.• Distributed.

RESULTS BACKEND• Stores the results from our tasks.

EXAMPLE

INTERGRATING WITHDJANGO.

BEWARE OF DJANGO-CELERY

IMPORTANT SETTINGS• BROKER_URL = 'redis://localhost:6379/0'• CELERY_RESULT_BACKEND = 'redis://localhost:6379/0'• CELERY_TASK_RESULT_EXPIRES = 7*86400 # 7 days• CELERYBEAT_SCHEDULER =

"djcelery.schedulers.DatabaseScheduler"• CELERY_ACCEPT_CONTENT = ['application/json']• CELERY_TASK_SERIALIZER = 'json'• CELERY_RESULT_SERIALIZER = 'json'• BROKER_BACKEND = "redis"• BROKER_CONNECTION_TIMEOUT = 5.0• CELERY_IMPORTS = ('<your-app-name>.tasks',)

Write code in celery.py

Intialize celery in __init__.py

BEST PRACTICES

NEVER PASS OBJECTS ASARGUMENTS.

AVOID LAUNCHING SYNCHRONOUS SUBTASKS

PERIODIC TASKS

THINGS GO WRONG ?

RE-TRY!

Thanks!