PyCon India, 2011 (Work in Progress) Sunil Arora
description
Transcript of PyCon India, 2011 (Work in Progress) Sunil Arora
![Page 1: PyCon India, 2011 (Work in Progress) Sunil Arora](https://reader035.fdocuments.us/reader035/viewer/2022081520/568149e9550346895db7135c/html5/thumbnails/1.jpg)
Redis And Python
PyCon India, 2011
(Work in Progress)Sunil Arora
![Page 2: PyCon India, 2011 (Work in Progress) Sunil Arora](https://reader035.fdocuments.us/reader035/viewer/2022081520/568149e9550346895db7135c/html5/thumbnails/2.jpg)
Raising Hands...
How many of you have used Redis before ?
How many of you have got a laptop here ?
![Page 3: PyCon India, 2011 (Work in Progress) Sunil Arora](https://reader035.fdocuments.us/reader035/viewer/2022081520/568149e9550346895db7135c/html5/thumbnails/3.jpg)
Who
Sunil Arora / @_sunil_ Work for ShopSocially (http://shopsocially.com) Interests:
Backend, Frontend, scaling, internet technologies in general, startups ... all kind of shit
![Page 4: PyCon India, 2011 (Work in Progress) Sunil Arora](https://reader035.fdocuments.us/reader035/viewer/2022081520/568149e9550346895db7135c/html5/thumbnails/4.jpg)
Who
Sunil Arora / @_sunil_ Work for ShopSocially (http://shopsocially.com) Interests:
Backend, Frontend, scaling, internet technologies in general, startups...
![Page 5: PyCon India, 2011 (Work in Progress) Sunil Arora](https://reader035.fdocuments.us/reader035/viewer/2022081520/568149e9550346895db7135c/html5/thumbnails/5.jpg)
Today's talk
What is Redis
How it works and what you can do with it
Real life use-cases
![Page 6: PyCon India, 2011 (Work in Progress) Sunil Arora](https://reader035.fdocuments.us/reader035/viewer/2022081520/568149e9550346895db7135c/html5/thumbnails/6.jpg)
What is Redis ?
Its between lot of stuff, so difficult to categorize it precisely
![Page 7: PyCon India, 2011 (Work in Progress) Sunil Arora](https://reader035.fdocuments.us/reader035/viewer/2022081520/568149e9550346895db7135c/html5/thumbnails/7.jpg)
What is Redis ?
I see Redis definitely more as a flexible tool that as a
solution specialized to solve a specific problem: his mixed soul of cache,
store, and messaging server shows this very
well
-Salvatore Sanfilippo
Picture by herzogbrhttp://www.flickr.com/photos/herzogbr/2274372747/sizes/z/in/photostream/
![Page 8: PyCon India, 2011 (Work in Progress) Sunil Arora](https://reader035.fdocuments.us/reader035/viewer/2022081520/568149e9550346895db7135c/html5/thumbnails/8.jpg)
Brief History of Redis
Released in March 2009 by Salvator Sanfilippo (@antirez)
Open source, BSD licensed
VMWare sponsored its development in March, 2010
![Page 9: PyCon India, 2011 (Work in Progress) Sunil Arora](https://reader035.fdocuments.us/reader035/viewer/2022081520/568149e9550346895db7135c/html5/thumbnails/9.jpg)
A few fundamentals
Written in C (25K LOC) Uses memory as main storage Single Threaded Uses disk for persistence Screamingly fast performance 50K read/write operations per seconds 200K read/write ops per second on a regular
EC2 instance
![Page 10: PyCon India, 2011 (Work in Progress) Sunil Arora](https://reader035.fdocuments.us/reader035/viewer/2022081520/568149e9550346895db7135c/html5/thumbnails/10.jpg)
Installation
$ git clone http://github.com/antirez/redis
$ cd redis
$ make
$ ./src/redis-server
..........
..........
![Page 11: PyCon India, 2011 (Work in Progress) Sunil Arora](https://reader035.fdocuments.us/reader035/viewer/2022081520/568149e9550346895db7135c/html5/thumbnails/11.jpg)
redis-py
The most popular python client library Andy McCurdy ([email protected]) Github: http://github.com/andymccurdy/redis-py easy_install redis OR pip install redis Optional: easy_install hiredis or pip install
hiredis
![Page 12: PyCon India, 2011 (Work in Progress) Sunil Arora](https://reader035.fdocuments.us/reader035/viewer/2022081520/568149e9550346895db7135c/html5/thumbnails/12.jpg)
Lets get started...
$ cd redis
$ ./src/redis-server
.........
>>> from redis import Redis
>>> redis_client = Redis()
>>> redis_client.keys()
>>> help(redis_client)
![Page 13: PyCon India, 2011 (Work in Progress) Sunil Arora](https://reader035.fdocuments.us/reader035/viewer/2022081520/568149e9550346895db7135c/html5/thumbnails/13.jpg)
Redis Keys
Not binary safe. Should not contain space or newline character A few rules about keys:
Too long keys are not a good idea Too short keys is also not a good idea “object-type:id:field” can be a nice idea, i.e.
“user:1001:name”
![Page 14: PyCon India, 2011 (Work in Progress) Sunil Arora](https://reader035.fdocuments.us/reader035/viewer/2022081520/568149e9550346895db7135c/html5/thumbnails/14.jpg)
Operations on Keys
KEYS EXISTS DEL EXPIRE OBJECT PERSIST
RANDOMKEY RENAME TYPE TTL EXPIREAT MOVE
![Page 15: PyCon India, 2011 (Work in Progress) Sunil Arora](https://reader035.fdocuments.us/reader035/viewer/2022081520/568149e9550346895db7135c/html5/thumbnails/15.jpg)
Lets play with keys
>>>redis_client.keys()
>>>redis_client.exists('key')
>>>redis_client.delete('key')
>>>redis_client.type('key')
>>>......
>>>......
![Page 16: PyCon India, 2011 (Work in Progress) Sunil Arora](https://reader035.fdocuments.us/reader035/viewer/2022081520/568149e9550346895db7135c/html5/thumbnails/16.jpg)
Data Structures
Strings Lists Sets Sorted Sets Hashes
![Page 17: PyCon India, 2011 (Work in Progress) Sunil Arora](https://reader035.fdocuments.us/reader035/viewer/2022081520/568149e9550346895db7135c/html5/thumbnails/17.jpg)
Strings
SET GET MSET MGET SETEX
INCR INCRBY DECR DECRBY
![Page 18: PyCon India, 2011 (Work in Progress) Sunil Arora](https://reader035.fdocuments.us/reader035/viewer/2022081520/568149e9550346895db7135c/html5/thumbnails/18.jpg)
Strings – with redis client
>>> redis_client.set('key', 'value')
>>> redis_client.get('key')
>>> redis_client.delete('key')
![Page 19: PyCon India, 2011 (Work in Progress) Sunil Arora](https://reader035.fdocuments.us/reader035/viewer/2022081520/568149e9550346895db7135c/html5/thumbnails/19.jpg)
Fetch multiple keys at once
mget/mset redis_client.mset({'key1': 'val1', 'key2': 'val2'}) redis_client.mget('key1', 'key2',......)
![Page 20: PyCon India, 2011 (Work in Progress) Sunil Arora](https://reader035.fdocuments.us/reader035/viewer/2022081520/568149e9550346895db7135c/html5/thumbnails/20.jpg)
Expiration
Set a value with expire
>>>redis_client.setex('key', 'value', 2) #key to expire in 2 secs
>>>redis_client.expire('key', 2)
>>>redis_client.get('key')
>>>None
![Page 21: PyCon India, 2011 (Work in Progress) Sunil Arora](https://reader035.fdocuments.us/reader035/viewer/2022081520/568149e9550346895db7135c/html5/thumbnails/21.jpg)
Uses
To store transient states in your web application
![Page 22: PyCon India, 2011 (Work in Progress) Sunil Arora](https://reader035.fdocuments.us/reader035/viewer/2022081520/568149e9550346895db7135c/html5/thumbnails/22.jpg)
Uses
Who is online?
![Page 23: PyCon India, 2011 (Work in Progress) Sunil Arora](https://reader035.fdocuments.us/reader035/viewer/2022081520/568149e9550346895db7135c/html5/thumbnails/23.jpg)
Uses
Redis as LRU cache (http://antirez.com/post/redis-as-LRU-cache.html)
![Page 24: PyCon India, 2011 (Work in Progress) Sunil Arora](https://reader035.fdocuments.us/reader035/viewer/2022081520/568149e9550346895db7135c/html5/thumbnails/24.jpg)
Atomic Increments
>>>help(redis_client.incr)
>>>help(redis_client.decr)
>>>
>>> redis_client.incr('counter', 1)
>>>1
>>> redis_client.incr('counter')
>>>2
>>> redis_client.incr('counter')
>>>3
![Page 25: PyCon India, 2011 (Work in Progress) Sunil Arora](https://reader035.fdocuments.us/reader035/viewer/2022081520/568149e9550346895db7135c/html5/thumbnails/25.jpg)
Uses
High Speed counters (views/clicks/votes/likes..)
![Page 26: PyCon India, 2011 (Work in Progress) Sunil Arora](https://reader035.fdocuments.us/reader035/viewer/2022081520/568149e9550346895db7135c/html5/thumbnails/26.jpg)
Uses
API Rate Limiting
![Page 27: PyCon India, 2011 (Work in Progress) Sunil Arora](https://reader035.fdocuments.us/reader035/viewer/2022081520/568149e9550346895db7135c/html5/thumbnails/27.jpg)
Uses
Generating unique IDs
![Page 28: PyCon India, 2011 (Work in Progress) Sunil Arora](https://reader035.fdocuments.us/reader035/viewer/2022081520/568149e9550346895db7135c/html5/thumbnails/28.jpg)
Lists
Ordered list of binarysafe strings Doubly linked list Memory footprint optimized for smaller list O(1) insertion/deletion at both ends
![Page 29: PyCon India, 2011 (Work in Progress) Sunil Arora](https://reader035.fdocuments.us/reader035/viewer/2022081520/568149e9550346895db7135c/html5/thumbnails/29.jpg)
Lists - operations
LPUSH RPUSH LSET LRANGE LPOP BLPOP BRPOP BRPOPLPUSH
LINSERT RPOP RPOPLPUSH LPUSHX RPUSHX
![Page 30: PyCon India, 2011 (Work in Progress) Sunil Arora](https://reader035.fdocuments.us/reader035/viewer/2022081520/568149e9550346895db7135c/html5/thumbnails/30.jpg)
Uses
Web apps are full of lists :)
![Page 31: PyCon India, 2011 (Work in Progress) Sunil Arora](https://reader035.fdocuments.us/reader035/viewer/2022081520/568149e9550346895db7135c/html5/thumbnails/31.jpg)
Uses
Capped List
![Page 32: PyCon India, 2011 (Work in Progress) Sunil Arora](https://reader035.fdocuments.us/reader035/viewer/2022081520/568149e9550346895db7135c/html5/thumbnails/32.jpg)
Uses
Normal QueueReal time message Queue
![Page 33: PyCon India, 2011 (Work in Progress) Sunil Arora](https://reader035.fdocuments.us/reader035/viewer/2022081520/568149e9550346895db7135c/html5/thumbnails/33.jpg)
Uses
Social Activity Streams
![Page 34: PyCon India, 2011 (Work in Progress) Sunil Arora](https://reader035.fdocuments.us/reader035/viewer/2022081520/568149e9550346895db7135c/html5/thumbnails/34.jpg)
Sets
An unordered collection of distinct byte strings
Nothing different from the data type in python
![Page 35: PyCon India, 2011 (Work in Progress) Sunil Arora](https://reader035.fdocuments.us/reader035/viewer/2022081520/568149e9550346895db7135c/html5/thumbnails/35.jpg)
Sets - Operations
SADD SCARD SDIFF SDIFFSTORE SINTER SINTERSTORE SISMEMBER SMEMBERS
SMOVE SPOP SRANDMEMBER SREM SUNION SUNIONSTORE
![Page 36: PyCon India, 2011 (Work in Progress) Sunil Arora](https://reader035.fdocuments.us/reader035/viewer/2022081520/568149e9550346895db7135c/html5/thumbnails/36.jpg)
Uses
Picking random items from a set using SRANDMEMBER
![Page 37: PyCon India, 2011 (Work in Progress) Sunil Arora](https://reader035.fdocuments.us/reader035/viewer/2022081520/568149e9550346895db7135c/html5/thumbnails/37.jpg)
Uses
Primitive to construct filtering mechanism on items
![Page 38: PyCon India, 2011 (Work in Progress) Sunil Arora](https://reader035.fdocuments.us/reader035/viewer/2022081520/568149e9550346895db7135c/html5/thumbnails/38.jpg)
Which of my friend's are online right now?
Uses
![Page 39: PyCon India, 2011 (Work in Progress) Sunil Arora](https://reader035.fdocuments.us/reader035/viewer/2022081520/568149e9550346895db7135c/html5/thumbnails/39.jpg)
Sorted Sets
ZADD ZCARD ZCOUNT ZINCRBY ZINTERSTORE ZRANGE ZRANGEBYSCORE ZRANK
ZREM ZREMRANGEBYRA
NK ZREMRANGEBYSC
ORE ZREVRANGE ZREVRANGEBYSCO
RE ZSCORE ZUNIONSTORE
![Page 40: PyCon India, 2011 (Work in Progress) Sunil Arora](https://reader035.fdocuments.us/reader035/viewer/2022081520/568149e9550346895db7135c/html5/thumbnails/40.jpg)
SORT
SORT KEY
SORT key BY pattern (e.g. object-type:*:age)
SORT key LIMIT 0 10
SORT key GET user:*:username
SORT key BY pattern STORE dstkey
![Page 41: PyCon India, 2011 (Work in Progress) Sunil Arora](https://reader035.fdocuments.us/reader035/viewer/2022081520/568149e9550346895db7135c/html5/thumbnails/41.jpg)
Uses
Generating sorted transient views
![Page 42: PyCon India, 2011 (Work in Progress) Sunil Arora](https://reader035.fdocuments.us/reader035/viewer/2022081520/568149e9550346895db7135c/html5/thumbnails/42.jpg)
Publish/Subscribe
A simple and efficient implementation of publish/subscribe messaging paradigm
Client can subscribe/psubscribe to receive messages on channels (key patterns)
![Page 43: PyCon India, 2011 (Work in Progress) Sunil Arora](https://reader035.fdocuments.us/reader035/viewer/2022081520/568149e9550346895db7135c/html5/thumbnails/43.jpg)
Publish/Subscribe
PSUBSCRIBE PUBLISH PUNSUBSCRIBE SUBSCRIBE UNSUBSCRIBE
![Page 44: PyCon India, 2011 (Work in Progress) Sunil Arora](https://reader035.fdocuments.us/reader035/viewer/2022081520/568149e9550346895db7135c/html5/thumbnails/44.jpg)
Uses
Many to Many message passing
![Page 45: PyCon India, 2011 (Work in Progress) Sunil Arora](https://reader035.fdocuments.us/reader035/viewer/2022081520/568149e9550346895db7135c/html5/thumbnails/45.jpg)
Uses
Web Chat
![Page 46: PyCon India, 2011 (Work in Progress) Sunil Arora](https://reader035.fdocuments.us/reader035/viewer/2022081520/568149e9550346895db7135c/html5/thumbnails/46.jpg)
Questions