Redis And Python
PyCon India, 2011Sunil Arora
Raising Hands...
How many of you have used Redis before ?
How many of you have got a laptop ?
About Me
I tweet at Sunil Arora / @_sunil_
I work at ShopSocially
I blog at http://sunilarora.org
Today's talk
What is Redis
How it works and what you can do with it
Real life use-casesReal life use-cases
Some hand-on with Redis
Redis - Brief History
Initially written to improve performance of Web Analytics product LLOOGG out of his startup
Salvatore Sanfilippo
@antirez
http://antirez.com
Redis – Brief History
Released in March 2009 (Open Source, BSD licensed)
VMWare hired Salvatore in March, 2010
Then Pieter Noordhuis (key contributor) was hired
What is Redis ?
Its between lot of stuff, so difficult to categorize it precisely
What is Redis ?
I see Redis definitely more as a flexible tool that as a
solution specialized to solve a specific problem: his mixed soul of cache,
store, and messaging server shows this very
well
-Salvatore Sanfilippo
Picture by herzogbrhttp://www.flickr.com/photos/herzogbr/2274372747/sizes/z/in/photostream/
Redis is...
Remote Data Structure Server
Redis is...
Supports rich data types of computer science
- Strings, Lists, Sets, Sorted Sets, Hashes...
Rich sets of primitives (commands) to manipulate these types
Predictive complexity measurements
A few fundamentals
Written in C (no external dependency) Uses memory as main storage Uses disk for persistence Single Threaded Every command performs atomic execution
Performance
Screamingly fast performance ~50K read/write operations per seconds 100K+ read/write ops per second on a regular
EC2 instance
Installation
$ git clone http://github.com/antirez/redis
$ cd redis
$ make
$ ./src/redis-server
..........
..........
$./src/redis-cli
Redis> PING
PONG
Python Libraries
Redis-py - Python client library
Txredisapi - An asynchronous Python client for the Redis database, based on Twisted.
Redisco - ORM for redis along the lines Ohm for Ruby
redis-py
The most popular python client library Andy McCurdy ([email protected]) Github: http://github.com/andymccurdy/redis-py
$easy_install redis OR
$pip install redis Optional
$easy_install hiredis
$pip install hiredis
Lets get started...
$ cd redis
$ ./src/redis-server
.........
>>> from redis import Redis
>>> redis_client = Redis()
>>> redis_client.keys()
>>> help(redis_client)
Redis Keys
Not binary safe. Should not contain space or newline character A few rules about keys:
Too long keys are not a good idea Too short keys is also not a good idea “object-type:id:field” can be a nice idea, i.e.
“user:1001:name”
Operations on Keys
KEYS EXISTS DEL EXPIRE OBJECT PERSIST
RANDOMKEY RENAME TYPE TTL EXPIREAT MOVE
Lets play with keys
>>>redis_client.keys()
>>>redis_client.exists('key')
>>>redis_client.delete('key')
>>>redis_client.type('key')
>>>......
>>>......
Data Structures
Strings Lists Sets Sorted Sets Hashes
Strings
SET GET MSET MGET SETEX GETSET SETNX
INCR INCRBY DECR DECRBY
Strings – with redis client
>>> redis_client.set('key', 'value')
>>> redis_client.get('key')
>>> redis_client.delete('key')
Fetch multiple keys at once
mget/mset redis_client.mset({'key1': 'val1', 'key2': 'val2'}) redis_client.mget('key1', 'key2',......)
Expiration
Set a value with expire
>>>redis_client.setex('key', 'value', 2) #key to expire in 2 secs
>>>redis_client.expire('key', 2)
>>>redis_client.get('key')
>>>None
Expire Semantics
Key with expiry known as volatile keys
Whenever a volatile key is modified, its expiry is reset
- To avoid inconsistency in following cases• Replication
• Append Only Log
Uses
To store transient states in your web application
Uses
Who is online?
Uses
Redis as LRU cache (http://antirez.com/post/redis-as-LRU-cache.html)
Atomic Increments
>>>help(redis_client.incr)
>>>help(redis_client.decr)
>>>
>>> redis_client.incr('counter', 1)
>>>1
>>> redis_client.incr('counter')
>>>2
>>> redis_client.incr('counter')
>>>3
Uses
High Speed counters (views/clicks/votes/likes..)
Uses
API Rate Limiting
Uses
Generating unique IDs
Lists
Ordered list of binarysafe strings Doubly linked list Memory footprint optimized for smaller list O(1) insertion/deletion at both ends
Lists - operations
LPUSH RPUSH LSET LRANGE LPOP BLPOP BRPOP BRPOPLPUSH
LINSERT RPOP RPOPLPUSH LPUSHX RPUSHX
Uses
Web apps are full of lists :)
Uses
Capped List
Uses
Real time message QueueBackground Worker queues (Resque, Celery)
https://github.com/blog/542-introducing-resque
Uses
Social Activity Streams or notifications
Sets
An unordered collection of distinct byte strings
Nothing different from the data type in python
Sets - Operations
SADD SCARD SREM SISMEMBER SMEMBERS SPOP SRANDMEMBER
Operations between Sets
SMOVE SUNION SDIFF SINTER
Sets - Operations
SDIFFSTORE SINTERSTORE SUNIONSTORE
Sets - Uses
Picking random items from a set using SRANDMEMBER
Ex.
– Picking a random article from daily news
– Pick a random Ad for serving
– Pick a random option for A/B
Note: Time complexity is O(1). Compare it with SQL's “order by Rand()”
Sets - Uses
To model Relations in social graph Ex.
>>>redis_client.sadd('friends:john', 'jenny', 'maria')
>>>redis_client.sadd('friends:ben', 'maria', 'kate')
>>>redis_client.sinter('friends:john', 'friends:ben')
Relations (friend/followers)
Common followlist for A and B
>>>sinter('users:A:follows', 'users:B:follows')
Unique to B compared to C
>>>sdiff('users:B:follows', 'users:C:follows') Mutual relationship (friend as well as follower)
>>>sinter('users:B:followers', 'users:B:friends')
Who does not follow me back ?>>>sdiff('users:B:friends', 'users:B:followers')
Who am I not following back?>>>sdiff('users:B:followers', 'user:B:friends')
Which of my friend's are online right now?
http://www.lukemelia.com/blog/archives/2010/01/17/redis-in-practice-whos-online/
SINTER online_people my_friends
Which of my friends are online right now?
>>>RENAME online:fresh online:stale #every minute
>>>SUNIONSTORE online:fresh online:stale
>>>#friend jack connects
>>>SADD online:fresh 'jack'
>>>SUNIONSTORE online online:fresh online:stale
>>>#who is online ?
>>>SINTER online friends
Sorted Sets
Ordered sets on the basis of score
Sorted Sets
ZADD ZCARD ZCOUNT ZINCRBY ZINTERSTORE ZRANGE ZRANGEBYSCORE ZRANK
ZREM ZREMRANGEBYRA
NK ZREMRANGEBYSC
ORE ZREVRANGE ZREVRANGEBYSCO
RE ZSCORE ZUNIONSTORE
Sorted Sets – Use Case
To build index on your dataset
Uses - Realtime Leaderboards
>>>zincrby('leaderboard', 'john', 2)
>>>zincrby('leaderboard', 'jack', 5)
>>>zincrby('leaderboard', 'kate', 1)
>>>zincrby('leaderboard', 'kate', 10)
....
>>>zrange('leaderboard', 0, -1, withscores = True)
Uses - Realtime Leaderboards
Most download resources
Most popular articles on the website
Weekly popular list
Most downloaded stuff between date1 and date2
.....
.....
AutoComplete with Sored Sets
http://antirez.com/post/autocomplete-with-redis.html
Hashes
Equivalent to Python dictionary or Ruby hash or Java hashmap
Operations– hmset users:1 {'username': 'jim', 'score': 23}
– hgetall users:1
– Hincrby Useful for storing structured data
– hmset user:1:preferences {'flash_shown': 'yes', 'bgcolor': '#fff'}
– Expire user:1:preferences <duration>
– Hmgetall user:1:preferences #returns entire dict
Redis Transactions
Using Multi/Watch
>>>p = redis_client.pipeline()
>>>p.lpush('a', 1)
>>>p.ltrim('a', 0, 100)
>>>p.execute Limitation
– Since commands are queued, Can't read values
– No conditional execution
– If not high write contention scenario, WATCH can be used
Use Redis Scripting to write your own primitive
Designing with Redis
Data layout design on basis of Query
No Query Optimizer
Manually build indexes
Durability
Snapshotting mode
– Binary dump every x secs or y ops Append Only File (AOF)
– Every command is written to a file
– On restart/crash, commands replayed
– Fsync on every new command
– Fsync every second
– OS decide to Replication
Publish/Subscribe
A simple and efficient implementation of publish/subscribe messaging paradigm
Client can subscribe/psubscribe to receive messages on channels (key patterns)
Publish/Subscribe
PSUBSCRIBE PUBLISH PUNSUBSCRIBE SUBSCRIBE UNSUBSCRIBE
Uses
Many to Many message passing
Uses
Web Chat
References
Redis Website (http://redis.io) SimonWillison Tutorial (
http://simonwillison.net/static/2010/redis-tutorial/) Redis from ground up (
http://blog.mjrusso.com/2010/10/17/redis-from-the-ground-up.html#heading_toc_j_19)
Redis under the hood (http://pauladamsmith.com/articles/redis-under-the-hood.html)
Tumbler and Redis (http://engineering.tumblr.com/post/7819252942/staircar-redis-powered-notifications)
References
Redis at disqus (http://bretthoerner.com/2011/2/21/redis-at-disqus/)
Redis at Craiglist http://blog.zawodny.com/2011/02/26/redis-sharding-at-craigslist/
Redis at Bump
http://devblog.bu.mp/how-we-use-redis-at-bump
Redis: AK 47 of postrelational database
http://www.slideshare.net/karmi/redis-the-ak47-of-postrelational-databases
Questions
Thanks
Contact details
Twitter: @_sunil_Email: arora.sunil AT gmail.com
Top Related