Message Queueing in Python -...
Transcript of Message Queueing in Python -...
![Page 1: Message Queueing in Python - PyConpycon.pk/static/media/uploads/presentations/redis_with_python_-_pu… · Pub/Sub Messaging Pattern Redis Pub/Sub uses a message passing system that](https://reader033.fdocuments.us/reader033/viewer/2022042804/5f5b1f7139bd483ab872ac63/html5/thumbnails/1.jpg)
Redis with Python Pub/Sub and Message Queueing in Python
PYCON MEETUP 2018#PyConPakistan#PythonPakistan
Ali Raza Bhayani
Founder at PYTHON.ORG.PKCTO at BitsWits (Pvt) Ltd.CEO and Founder at DataLysis.ioBlogger at LearningByDoing.io
Open Source Enthusiast, Hacker, Enabler, DoTank, Autodidact,
Yogi and an avid Reader.
Web: www.alirazabhayani.com
Web: www.dataLysis.io
Web: www.LearningByDoing.io
Email: [email protected]
Twitter: @alirazabhayani
Facebook: https://www.facebook.com/alirazabhayani
![Page 2: Message Queueing in Python - PyConpycon.pk/static/media/uploads/presentations/redis_with_python_-_pu… · Pub/Sub Messaging Pattern Redis Pub/Sub uses a message passing system that](https://reader033.fdocuments.us/reader033/viewer/2022042804/5f5b1f7139bd483ab872ac63/html5/thumbnails/2.jpg)
Data Science Case Study posted on LearningByDoing.IO
![Page 3: Message Queueing in Python - PyConpycon.pk/static/media/uploads/presentations/redis_with_python_-_pu… · Pub/Sub Messaging Pattern Redis Pub/Sub uses a message passing system that](https://reader033.fdocuments.us/reader033/viewer/2022042804/5f5b1f7139bd483ab872ac63/html5/thumbnails/3.jpg)
![Page 4: Message Queueing in Python - PyConpycon.pk/static/media/uploads/presentations/redis_with_python_-_pu… · Pub/Sub Messaging Pattern Redis Pub/Sub uses a message passing system that](https://reader033.fdocuments.us/reader033/viewer/2022042804/5f5b1f7139bd483ab872ac63/html5/thumbnails/4.jpg)
What is Redis?
![Page 5: Message Queueing in Python - PyConpycon.pk/static/media/uploads/presentations/redis_with_python_-_pu… · Pub/Sub Messaging Pattern Redis Pub/Sub uses a message passing system that](https://reader033.fdocuments.us/reader033/viewer/2022042804/5f5b1f7139bd483ab872ac63/html5/thumbnails/5.jpg)
Redis is an open source (BSD licensed), inmemory data structure store, used as a database, cache and message broker.
![Page 6: Message Queueing in Python - PyConpycon.pk/static/media/uploads/presentations/redis_with_python_-_pu… · Pub/Sub Messaging Pattern Redis Pub/Sub uses a message passing system that](https://reader033.fdocuments.us/reader033/viewer/2022042804/5f5b1f7139bd483ab872ac63/html5/thumbnails/6.jpg)
How Redis is Different?
![Page 7: Message Queueing in Python - PyConpycon.pk/static/media/uploads/presentations/redis_with_python_-_pu… · Pub/Sub Messaging Pattern Redis Pub/Sub uses a message passing system that](https://reader033.fdocuments.us/reader033/viewer/2022042804/5f5b1f7139bd483ab872ac63/html5/thumbnails/7.jpg)
One of the big differences between Redis and other NoSQL databases is the data structures that Redis provides. Instead of working with a table abstraction, it uses different Data Structures
![Page 8: Message Queueing in Python - PyConpycon.pk/static/media/uploads/presentations/redis_with_python_-_pu… · Pub/Sub Messaging Pattern Redis Pub/Sub uses a message passing system that](https://reader033.fdocuments.us/reader033/viewer/2022042804/5f5b1f7139bd483ab872ac63/html5/thumbnails/8.jpg)
Redis – Data Structures
![Page 9: Message Queueing in Python - PyConpycon.pk/static/media/uploads/presentations/redis_with_python_-_pu… · Pub/Sub Messaging Pattern Redis Pub/Sub uses a message passing system that](https://reader033.fdocuments.us/reader033/viewer/2022042804/5f5b1f7139bd483ab872ac63/html5/thumbnails/9.jpg)
It supports data structures such as strings, hashes, lists, sets, sorted sets with range queries etc.
![Page 10: Message Queueing in Python - PyConpycon.pk/static/media/uploads/presentations/redis_with_python_-_pu… · Pub/Sub Messaging Pattern Redis Pub/Sub uses a message passing system that](https://reader033.fdocuments.us/reader033/viewer/2022042804/5f5b1f7139bd483ab872ac63/html5/thumbnails/10.jpg)
Redis Performance
![Page 11: Message Queueing in Python - PyConpycon.pk/static/media/uploads/presentations/redis_with_python_-_pu… · Pub/Sub Messaging Pattern Redis Pub/Sub uses a message passing system that](https://reader033.fdocuments.us/reader033/viewer/2022042804/5f5b1f7139bd483ab872ac63/html5/thumbnails/11.jpg)
Outstanding performance! Redis works with an inmemory dataset
PERFORMANCE
![Page 12: Message Queueing in Python - PyConpycon.pk/static/media/uploads/presentations/redis_with_python_-_pu… · Pub/Sub Messaging Pattern Redis Pub/Sub uses a message passing system that](https://reader033.fdocuments.us/reader033/viewer/2022042804/5f5b1f7139bd483ab872ac63/html5/thumbnails/12.jpg)
Redis – Data Persistence
![Page 13: Message Queueing in Python - PyConpycon.pk/static/media/uploads/presentations/redis_with_python_-_pu… · Pub/Sub Messaging Pattern Redis Pub/Sub uses a message passing system that](https://reader033.fdocuments.us/reader033/viewer/2022042804/5f5b1f7139bd483ab872ac63/html5/thumbnails/13.jpg)
You can persist it either by dumping the dataset to disk every once in a while, or by appending each command to a log.
PERSISTENCE
![Page 14: Message Queueing in Python - PyConpycon.pk/static/media/uploads/presentations/redis_with_python_-_pu… · Pub/Sub Messaging Pattern Redis Pub/Sub uses a message passing system that](https://reader033.fdocuments.us/reader033/viewer/2022042804/5f5b1f7139bd483ab872ac63/html5/thumbnails/14.jpg)
Companies Using Redis
![Page 15: Message Queueing in Python - PyConpycon.pk/static/media/uploads/presentations/redis_with_python_-_pu… · Pub/Sub Messaging Pattern Redis Pub/Sub uses a message passing system that](https://reader033.fdocuments.us/reader033/viewer/2022042804/5f5b1f7139bd483ab872ac63/html5/thumbnails/15.jpg)
TWITTER Utilizing the list data structure, Twitter stores the 800 most recent incoming tweets for a given user
PINTEREST Stores the user follower graphs in a Redis cluster where data is sharded across hundreds of instances
GITHUB Github uses Redis for their job queueing needs.
![Page 16: Message Queueing in Python - PyConpycon.pk/static/media/uploads/presentations/redis_with_python_-_pu… · Pub/Sub Messaging Pattern Redis Pub/Sub uses a message passing system that](https://reader033.fdocuments.us/reader033/viewer/2022042804/5f5b1f7139bd483ab872ac63/html5/thumbnails/16.jpg)
WHY PYTHON ?
![Page 17: Message Queueing in Python - PyConpycon.pk/static/media/uploads/presentations/redis_with_python_-_pu… · Pub/Sub Messaging Pattern Redis Pub/Sub uses a message passing system that](https://reader033.fdocuments.us/reader033/viewer/2022042804/5f5b1f7139bd483ab872ac63/html5/thumbnails/17.jpg)
![Page 18: Message Queueing in Python - PyConpycon.pk/static/media/uploads/presentations/redis_with_python_-_pu… · Pub/Sub Messaging Pattern Redis Pub/Sub uses a message passing system that](https://reader033.fdocuments.us/reader033/viewer/2022042804/5f5b1f7139bd483ab872ac63/html5/thumbnails/18.jpg)
Python and Redis
![Page 19: Message Queueing in Python - PyConpycon.pk/static/media/uploads/presentations/redis_with_python_-_pu… · Pub/Sub Messaging Pattern Redis Pub/Sub uses a message passing system that](https://reader033.fdocuments.us/reader033/viewer/2022042804/5f5b1f7139bd483ab872ac63/html5/thumbnails/19.jpg)
Redis – Installation
![Page 20: Message Queueing in Python - PyConpycon.pk/static/media/uploads/presentations/redis_with_python_-_pu… · Pub/Sub Messaging Pattern Redis Pub/Sub uses a message passing system that](https://reader033.fdocuments.us/reader033/viewer/2022042804/5f5b1f7139bd483ab872ac63/html5/thumbnails/20.jpg)
wget http://download.redis.io/releases/redisstable.tar.gz
tar xzf redisstable.tar.gz
cd redisstable
![Page 22: Message Queueing in Python - PyConpycon.pk/static/media/uploads/presentations/redis_with_python_-_pu… · Pub/Sub Messaging Pattern Redis Pub/Sub uses a message passing system that](https://reader033.fdocuments.us/reader033/viewer/2022042804/5f5b1f7139bd483ab872ac63/html5/thumbnails/22.jpg)
pip install redis
(redispy requires a running Redis server)
![Page 23: Message Queueing in Python - PyConpycon.pk/static/media/uploads/presentations/redis_with_python_-_pu… · Pub/Sub Messaging Pattern Redis Pub/Sub uses a message passing system that](https://reader033.fdocuments.us/reader033/viewer/2022042804/5f5b1f7139bd483ab872ac63/html5/thumbnails/23.jpg)
Redis – Pub/Sub
![Page 24: Message Queueing in Python - PyConpycon.pk/static/media/uploads/presentations/redis_with_python_-_pu… · Pub/Sub Messaging Pattern Redis Pub/Sub uses a message passing system that](https://reader033.fdocuments.us/reader033/viewer/2022042804/5f5b1f7139bd483ab872ac63/html5/thumbnails/24.jpg)
Pub/Sub Messaging PatternRedis Pub/Sub uses a message passing system that message senders - called publishers - post a message to a channel that the message receivers - called subscribers - can respond to messages without either the publishers or subscribers knowing any details about each other.
publisher
subscriber
subscriber
Redis ChannelMessage published to channel
Subscribers receive messages & can respond to messages or not
![Page 25: Message Queueing in Python - PyConpycon.pk/static/media/uploads/presentations/redis_with_python_-_pu… · Pub/Sub Messaging Pattern Redis Pub/Sub uses a message passing system that](https://reader033.fdocuments.us/reader033/viewer/2022042804/5f5b1f7139bd483ab872ac63/html5/thumbnails/25.jpg)
Redis – Pub/Sub
Pub/Sub, is characterized by listeners subscribing to channels, with publishers sending binary string messages to channels
![Page 26: Message Queueing in Python - PyConpycon.pk/static/media/uploads/presentations/redis_with_python_-_pu… · Pub/Sub Messaging Pattern Redis Pub/Sub uses a message passing system that](https://reader033.fdocuments.us/reader033/viewer/2022042804/5f5b1f7139bd483ab872ac63/html5/thumbnails/26.jpg)
Redis – Pub/Sub PythonHands On Implementation
![Page 27: Message Queueing in Python - PyConpycon.pk/static/media/uploads/presentations/redis_with_python_-_pu… · Pub/Sub Messaging Pattern Redis Pub/Sub uses a message passing system that](https://reader033.fdocuments.us/reader033/viewer/2022042804/5f5b1f7139bd483ab872ac63/html5/thumbnails/27.jpg)
Redis – Queue
![Page 28: Message Queueing in Python - PyConpycon.pk/static/media/uploads/presentations/redis_with_python_-_pu… · Pub/Sub Messaging Pattern Redis Pub/Sub uses a message passing system that](https://reader033.fdocuments.us/reader033/viewer/2022042804/5f5b1f7139bd483ab872ac63/html5/thumbnails/28.jpg)
Redis – Queue
![Page 29: Message Queueing in Python - PyConpycon.pk/static/media/uploads/presentations/redis_with_python_-_pu… · Pub/Sub Messaging Pattern Redis Pub/Sub uses a message passing system that](https://reader033.fdocuments.us/reader033/viewer/2022042804/5f5b1f7139bd483ab872ac63/html5/thumbnails/29.jpg)
RQ (Redis Queue) is a simple Python library for queueing jobs and processing them in the background with workers.
pip install rq
![Page 30: Message Queueing in Python - PyConpycon.pk/static/media/uploads/presentations/redis_with_python_-_pu… · Pub/Sub Messaging Pattern Redis Pub/Sub uses a message passing system that](https://reader033.fdocuments.us/reader033/viewer/2022042804/5f5b1f7139bd483ab872ac63/html5/thumbnails/30.jpg)
Task Queues in Python Hands On Implementation