Celery for internal API in SOA infrastructure
-
Upload
roman-imankulov -
Category
Documents
-
view
6.934 -
download
0
description
Transcript of Celery for internal API in SOA infrastructure
![Page 1: Celery for internal API in SOA infrastructure](https://reader036.fdocuments.us/reader036/viewer/2022081505/554f5ee2b4c905c8088b488c/html5/thumbnails/1.jpg)
Celery for internal API in SOA infrastructure
Roman ImankulovPyCon Russia, Feb 2013
![Page 2: Celery for internal API in SOA infrastructure](https://reader036.fdocuments.us/reader036/viewer/2022081505/554f5ee2b4c905c8088b488c/html5/thumbnails/2.jpg)
1. Internal API. What's the deal
SOA service architecture example
![Page 3: Celery for internal API in SOA infrastructure](https://reader036.fdocuments.us/reader036/viewer/2022081505/554f5ee2b4c905c8088b488c/html5/thumbnails/3.jpg)
1. Internal API. What's the deal
● DCOM: too Microsoftish
● CORBA: too Enterprisy
● SOAP: XML inside
● RESTful: too sloppy
Celery to the rescue?
![Page 4: Celery for internal API in SOA infrastructure](https://reader036.fdocuments.us/reader036/viewer/2022081505/554f5ee2b4c905c8088b488c/html5/thumbnails/4.jpg)
Agenda
1. Internal API. What's the deal
2. How to use Celery for internal API
3. Organizing Celery-based API. Routing
4. Celery-based API. Benefits, pitfalls and
security notes
![Page 5: Celery for internal API in SOA infrastructure](https://reader036.fdocuments.us/reader036/viewer/2022081505/554f5ee2b4c905c8088b488c/html5/thumbnails/5.jpg)
2. How to use Celery for internal API
Celery Worker
![Page 6: Celery for internal API in SOA infrastructure](https://reader036.fdocuments.us/reader036/viewer/2022081505/554f5ee2b4c905c8088b488c/html5/thumbnails/6.jpg)
2. How to use Celery for internal API
Classic celery client imports task function
![Page 7: Celery for internal API in SOA infrastructure](https://reader036.fdocuments.us/reader036/viewer/2022081505/554f5ee2b4c905c8088b488c/html5/thumbnails/7.jpg)
2. How to use Celery for internal API
Classic celery client imports makes you:● install everything
○ slows down installation
● import everything○ python processes grow in size○ circular dependencies○ tight coupling
● share everything○ ... with outsource developers
![Page 8: Celery for internal API in SOA infrastructure](https://reader036.fdocuments.us/reader036/viewer/2022081505/554f5ee2b4c905c8088b488c/html5/thumbnails/8.jpg)
2. How to use Celery for internal API
"Task implementation is not to be exposed without necessity"
William of Ockham
![Page 9: Celery for internal API in SOA infrastructure](https://reader036.fdocuments.us/reader036/viewer/2022081505/554f5ee2b4c905c8088b488c/html5/thumbnails/9.jpg)
2. How to use Celery for internal API
Low-level interface defines only task name
![Page 10: Celery for internal API in SOA infrastructure](https://reader036.fdocuments.us/reader036/viewer/2022081505/554f5ee2b4c905c8088b488c/html5/thumbnails/10.jpg)
2. How to use Celery for internal API
Convenient API requires some boilerplate code
![Page 11: Celery for internal API in SOA infrastructure](https://reader036.fdocuments.us/reader036/viewer/2022081505/554f5ee2b4c905c8088b488c/html5/thumbnails/11.jpg)
2. How to use Celery for internal API
Make it even more convenient with introspection
![Page 12: Celery for internal API in SOA infrastructure](https://reader036.fdocuments.us/reader036/viewer/2022081505/554f5ee2b4c905c8088b488c/html5/thumbnails/12.jpg)
2. How to use Celery for internal API
Introspection magic with celery-api
http://github.com/imankulov/celery-api
![Page 13: Celery for internal API in SOA infrastructure](https://reader036.fdocuments.us/reader036/viewer/2022081505/554f5ee2b4c905c8088b488c/html5/thumbnails/13.jpg)
3. Celery-based API. Routing
![Page 14: Celery for internal API in SOA infrastructure](https://reader036.fdocuments.us/reader036/viewer/2022081505/554f5ee2b4c905c8088b488c/html5/thumbnails/14.jpg)
3. Celery-based API. Routing
Creating API endpoint
1. Common package for all API endpoints2. Every worker imports celery instance3. Expose tasks for worker with
CELERY_IMPORTS
4. Launch worker withcelery worker -A module_name -Q foo
-n foo
![Page 15: Celery for internal API in SOA infrastructure](https://reader036.fdocuments.us/reader036/viewer/2022081505/554f5ee2b4c905c8088b488c/html5/thumbnails/15.jpg)
3. Celery-based API. Routing
Send tasks with
add.apply_async(queue='foo', args=(1, 1))
![Page 16: Celery for internal API in SOA infrastructure](https://reader036.fdocuments.us/reader036/viewer/2022081505/554f5ee2b4c905c8088b488c/html5/thumbnails/16.jpg)
3. Celery-based API. Routing
Define routing rules
![Page 17: Celery for internal API in SOA infrastructure](https://reader036.fdocuments.us/reader036/viewer/2022081505/554f5ee2b4c905c8088b488c/html5/thumbnails/17.jpg)
3. Celery-based API. Routing
Task in the wire
{ "body": <base64 encoded string with task name and args>, "properties": { "body_encoding": "base64", "delivery_info": { "priority": 0, "routing_key": "foo", "exchange": "foo" }, "delivery_mode": 2, "delivery_tag": <UUID> }, ...}
![Page 18: Celery for internal API in SOA infrastructure](https://reader036.fdocuments.us/reader036/viewer/2022081505/554f5ee2b4c905c8088b488c/html5/thumbnails/18.jpg)
3. Celery-based API. Routing
Task in the wire
"delivery_info": { "priority": 0, "routing_key": "foo", "exchange": "foo" }
![Page 19: Celery for internal API in SOA infrastructure](https://reader036.fdocuments.us/reader036/viewer/2022081505/554f5ee2b4c905c8088b488c/html5/thumbnails/19.jpg)
3. Celery-based API. Routing
AMQP for dummies
![Page 20: Celery for internal API in SOA infrastructure](https://reader036.fdocuments.us/reader036/viewer/2022081505/554f5ee2b4c905c8088b488c/html5/thumbnails/20.jpg)
3. Celery-based API. Routing
Define all queues (API endpoints) you have
![Page 21: Celery for internal API in SOA infrastructure](https://reader036.fdocuments.us/reader036/viewer/2022081505/554f5ee2b4c905c8088b488c/html5/thumbnails/21.jpg)
4. Celery-based API. Benefits
High level protocol.
Takes the responsibility for
data serialization routingexception handling
![Page 22: Celery for internal API in SOA infrastructure](https://reader036.fdocuments.us/reader036/viewer/2022081505/554f5ee2b4c905c8088b488c/html5/thumbnails/22.jpg)
4. Celery-based API. Benefits
Support for wide range of brokers.
You can switch between brokers with no major changes in your code
Brokers:RabbitMQ Redis SQLAlchemy Django MongoDB Amazon SQSCouchDB Beanstalk
![Page 23: Celery for internal API in SOA infrastructure](https://reader036.fdocuments.us/reader036/viewer/2022081505/554f5ee2b4c905c8088b488c/html5/thumbnails/23.jpg)
4. Celery-based API. Benefits
Built-in extra stuff which you were afraid to ever dream of.
asynchronous execution parallel executionasynchronous parallel executiondelayed execution throttlingautomatic retrying of failed tasks limiting the time of executionautoscaling API inspectionpublic key cryptography (message signing)
![Page 24: Celery for internal API in SOA infrastructure](https://reader036.fdocuments.us/reader036/viewer/2022081505/554f5ee2b4c905c8088b488c/html5/thumbnails/24.jpg)
4. Celery-based API. Pitfalls
Python-centric.
Celery is written in Python and meant to be used with Python code exclusively
![Page 25: Celery for internal API in SOA infrastructure](https://reader036.fdocuments.us/reader036/viewer/2022081505/554f5ee2b4c905c8088b488c/html5/thumbnails/25.jpg)
4. Celery-based API. Pitfalls
Increasing complexity.
Celery codebase 46k LOCKombu codebase 19k LOC
CompareFlask codebase 10k LOCWerkzeug codebase 28k LOCJinja2 codebase 16k LOC
![Page 26: Celery for internal API in SOA infrastructure](https://reader036.fdocuments.us/reader036/viewer/2022081505/554f5ee2b4c905c8088b488c/html5/thumbnails/26.jpg)
4. Celery-based API. Security note
Never ever expose Celery broker to the Web!
![Page 27: Celery for internal API in SOA infrastructure](https://reader036.fdocuments.us/reader036/viewer/2022081505/554f5ee2b4c905c8088b488c/html5/thumbnails/27.jpg)
4. Celery-based API. Security note
1. Try >>> redis_url = 'redis://:[email protected]'>>> celery = Celery(broker=redis_url, result=redis_url)
2. Get the contents of /etc/pycon_secret
3. Exchange it for a secret prize ;)