Redis Basics

by Arthur ShvetsovCoreValue/MalkosUA

Agenda

● What is Redis?● Data structures overview● Data modeling examples● Keys expiration● Redis transactions● Redis pub/sub● Pipelining

2

What is Redis?● In-memory key-value store, with persistence● Open source, written in C● Extremely fast and lightweight● Binary-safe keys with supported expiration● High level data structures● Support for atomic operations and transactions● Internal scripting with LUA● Master-slave replication● Tons of client libraries for all major languages

3

History

● REmote DIctionary Server● Released in March 2009 by Salvatore

Sanfilippo● Built in order to scale http://lloogg.com/● According to the monthly ranking by DB-

Engines.com, Redis is the most popular key-value store

4

How to install?

● http://redis.io/download● The Redis project does not officially support

Windows.● Microsoft Open Tech group develops and

maintains this Windows port targeting Win64.

5

Redis implementation details

● Redis is a TCP server using the client-server model.● Redis protocol - simple and efficient text-based

protocol called RESP (REdis Serialization Protocol).● Every Redis command is atomic, including the ones

that do multiple things.● Redis is single-threaded, which is how every

command is guaranteed to be atomic. While one command is executing, no other command will run.

6

Data structures overviewKey

● STRING - binary-safe string with max allowed size 512 MBValue

● Primitives○ STRING - Strings, integers, or floating point values

● Containers (of strings)○ LIST - Linked list of strings○ SET - Unordered collection of unique strings○ HASH - Unordered hash table of keys to values○ ZSET - Ordered mapping of string members to floating-

point scores, ordered by score7

Redis STRINGThe Redis String type is the simplest type of value you can associate with a Redis key. Integer and float numbers can be stored as STRING type in Redis.

hello

world

string

Key name Type of value

Value stored

Command What it does

GET Fetches the data stored at the given key

SET Sets the value stored at the given key

DEL Deletes the value stored at the given key (works for all types)

8

Redis STRING examplesredis> SET hello worldOK

redis> GET hello"world"

redis> DEL hello(integer) 1

redis> GET hello(nil)

redis> SET counter 100OK

redis> INCR counter(integer) 101

redis> INCR counter(integer) 102

redis> INCRBY counter 50(integer) 152

9

Redis STRING use cases

● Use Strings to store serialized JSON/XML/etc data● Use Strings as atomic counters using commands in

the INCR family: INCR, DECR, INCRBY.● Append to strings with the APPEND command. Use

Strings as a random access vectors with GETRANGE and SETRANGE.

10

Redis LISTLISTs in Redis is a sequence of order strings, implemented using a Linked List.

list-key

itemitem2item

list

Key name Type of value

List of values, duplicates possible

Command What it does

RPUSH Pushes the value onto the right end of the list

LRANGE Fetches a range of values from the list

LINDEX Fetches an item at a given position in the list

LPOP Pops the value from the left end of the list and returns it

11

Redis LIST implementation details

● The max length of a list is 232 - 1 elements (4294967295, more than 4 billion of elements per list).

● Redis lists are implemented via Linked Lists.● The operation of adding a new element in the head or

in the tail of the list is performed in constant time.● Accessing an element by index is very fast near the

head and tail of the list, but O(N) operation in the middle.

12

Redis LIST examplesredis> RPUSH list-key item(integer) 1

redis> RPUSH list-key item2(integer) 2

redis> RPUSH list-key item(integer) 3

redis> LRANGE list-key 0 -11) "item"2) "item2"3) "item"

redis> LINDEX list-key 1"item2"

redis> LPOP list-key"item"

redis> LRANGE list-key 0 -11) "item2"2) "item"

13

Redis LIST use cases● Model a timeline, for example in a social network, using

LPUSH in order to add new elements in the user time line, and using LRANGE in order to retrieve a few of recently inserted items.

● You can use LPUSH together with LTRIM to create a list that never exceeds a given number of elements, but just remembers the latest N elements.

● Lists can be used as a message passing primitive.

14

Redis SETIn Redis, SETs are similar to LISTs in that they’re a sequence of strings, but unlike LISTs, Redis SETs use a hash table to keep all strings unique.

set-key

item2itemitem3

set

Key name Type of value

Set of distinct values, undefined order

Command What it does

SADD Adds the item to the set

SMEMBERS Returns the entire set of items

SISMEMBER Checks if an item is in the set

SREM Removes the item from the set, if it exists15

Redis SET implementation details

● Redis SET provides O(1) constant time regardless of the number of elements contained inside the set for add, remove and test for member existance operations.

● Supports server-side operations of unions, intersections, differences of sets.

● The max number of elements in set 232 - 1.

16

redis> SADD set-key item(integer) 1

redis> SADD set-key item2(integer) 1

redis> SADD set-key item3(integer) 1

redis> SADD set-key item(integer) 0

redis> SMEMBERS set-key1) "item"2) "item2"3) "item3"

Redis SET examplesredis> SISMEMBER set-key item4(integer) 0

redis> SISMEMBER set-key item(integer) 1

redis> SREM set-key item2(integer) 1

redis> SREM set-key item2(integer) 0

redis> SMEMBERS set-key1) "item"2) "item3"

17

Redis SET use cases

● Use SETs to track unique things, for example unique IP addresses or user ids visiting a web page.

● Sets are good to represent relations between objects. For example 1 to many relation could be represented as a set order:1234:items → [234, 555, 674]

18

Redis HASHRedis HASHes store a mapping of keys to values. The values that can be stored in HASHes are strings and numbers.

hash-key

sub-key-1sub-key-2

hash

Key name Type of value

Distinct keys, undefined order

Command What it does

HSET Stores the value at the key in the hash

HGET Fetches the value at the given hash key

HGETALL Fetches the entire hash

HDEL Removes a key from the hash, if it exists

Values associated with the key

value-1value-2

19

Redis HASH implementation details

● The max number of elements in hash 232 - 1.● A hash with a few fields (where few means

up to one hundred or so) is stored in a way that takes very little space, so you can store millions of objects in a small Redis instance.

20

redis> HSET hash-key sub-key1 value1(integer) 1

redis> HSET hash-key sub-key2 value2(integer) 1

redis> HSET hash-key sub-key1 value1(integer) 0

redis> HGETALL hash-key1) "sub-key1"2) "value1"3) "sub-key2"4) "value2"

Redis HASH examplesredis> HDEL hash-key sub-key2(integer) 1

redis> HDEL hash-key sub-key2(integer) 0

redis> HGET hash-key sub-key1"value1"

redis> HGETALL hash-key1) "sub-key1"2) "value1"

21

● Hashes are used mainly to represent objects. For example, storing users as hashes:

Redis HASH use cases

users:1

{id: 1,name: john,email: john@gmail.com

}

users:2

{id: 1,name: john,email: john@gmail.com

}

22

Shopping cart examplecart

cart_line

Redis model

SET cart:john → [ 1, 3 ]SET cart:james → [ 2 ]

HASH cart:1 { user: “john”, status: “submitted” }HASH cart:2 { user: “james”, status: “in progress” }

HASH cart:1:products [ 28: 1, 372: 2 ]HASH cart:2:products [ 15: 5, 160: 4 ]

CartID User Status

1 john Submitted

2 james In Progress

3 john Submitted

CartID ProductID Quantity

1 28 1

1 372 2

2 15 5

2 160 423

Redis ZSETZSET (Sorted set) is a data type which is similar to a mix between a Set and a Hash. Every element (called member) in a sorted set is associated with a floating point value (called score).

zset-key

member1member2

zset

Key name Type of value

Named members,ordered byassociated

score

Command What it does

ZADD Adds member with the given score to the ZSET

ZRANGE Fetches the items in the ZSET from their positions in sorted order

ZRANGEBYSCORE

Fetches items in the ZSET based on a range of scores

ZREM Removes the item from the ZSET, if it exists

Scores, orderedby numeric value

728982

24

Redis ZSET implementation details● ZSET supports add, remove, or update elements in a time

proportional to the ln(N) (logarithm of the number of elements).

● Since elements are taken in order get ranges by score or by rank (position) could be done in a very fast way.

● Accessing the middle of a sorted set is also very fast, so ZSETs can be used as a smart list of non repeating elements where you can quickly access everything you need: elements in order, fast existence test, fast access to elements in the middle.

25

redis> ZADD zset-key 728 member1(integer) 1

redis> ZADD zset-key 982 member0(integer) 1

redis> ZADD zset-key 982 member0(integer) 0

redis> ZRANGE zset-key 0 -1 withscores1) "member1"2) "728"3) "member0"4) "982"

Redis ZSET examplesredis> ZRANGEBYSCORE zset-key 0 800 withscores1) "member1"2) "728"

redis> ZREM zset-key member1(integer) 1

redis> ZREM zset-key member1(integer) 0

redis> ZRANGE zset-key 0 -1 withscores1) "member0"2) "982"

26

Redis ZSET use cases

● A leader board in an online game, where every time a new score is submitted you update it using ZADD.

● Top users can be easily get using ZRANGE.● Using ZRANK can get a rank by username.

27

Redis key rules/recommendations● Very long/short keys are not a good idea. For example,

key user:1000:followers is more readable than u1000flw.● While short keys will obviously consume a bit less

memory, your job is to find the right balance between long and short keys.

● Stick with a schema. For instance object-type:id is a good idea, as in user:1000. Dots or dashes are often used for multi-word fields, as in comment:1234:reply.to or comment:1234:reply-to.

28

Redis keys anti-pattern

● We are building a blog and we need to store comments added by users with keys look like: posts:post_id:comment_id

● To find all comments added to specific post you will use command: KEYS posts:1234:*

● Is there a better solution?

29

Redis keys anti-pattern (solution)● The better solution is to use a hash with a new key

scheme posts:post_id:comments ● To add new comments to specific post use next

commands:○ HSET posts:1234:comments 1 '{"id":1,

"account": 1233, "subject": "..."}' ○ HSET posts:1234:comments 2 '{"id":2,

"account": 1233, "subject": "..."}'

30

Redis keys anti-pattern (solution)● To get all comment ids use command:

○ HKEYS posts:1234:comments● To get comment by specific id use command:

○ HGET posts:1234:comments 2● To delete a specific comment use command

○ HDEL posts:1234:comments 2● To delete all post comments use command

○ DEL posts:1234:comments

31

Redis keys expiration● Redis allows to set timeout (TTL) on keys by calling

EXPIRE key seconds command● Timeout is cleared only when key is removed or using

DEL or overwritten with SET or GETSET commands● Expiration could be refreshed by calling EXPIRE

command● After timeout has expired, the key will be deleted

automatically

32

How Redis expires keys?● Active way

○ Key is actively expired when some client tries to access it, and the key is found to be timed out.

● Passive way○ Redis does 10 times per second:

■ Test 20 random keys from the set of keys with an associated expire.

■ Delete all the keys found expired. ■ If more than 25% of keys were expired, start again

from step 1.33

Redis keys expiration examplesredis> SET mykey "Hello"OK

redis> EXPIRE mykey 10(integer) 1

redis> TTL mykey(integer) 10

redis> SET mykey "Hello World"OK

redis> TTL mykey(integer) -1

34

Redis Transactions● All commands are serialized and executed sequentially● Transactions are atomic (without another client’s command

being executed halfway through)● Either all commands or no commands will be executed● Redis commands for transactions:

○ WATCH ○ MULTI ○ DISCARD ○ EXEC ○ UNWATCH

35

Redis Transactions exampleredis> MULTIOK

redis> INCR fooQUEUED

redis> INCR booQUEUED

redis> EXEC1) (integer) 12) (integer) 1

redis> WATCH foo, booOKredis> MULTIOKredis> SET foo 2QUEUEDredis> SET boo 3QUEUEDredis> EXEC1) OK2) OK

36

Optimistic locking using CAS

● WATCH is used to provide a check-and-set (CAS) behavior to Redis transactions.

● WATCHed keys are monitored in order to detect changes against them.

● If at least one watched key is modified before the EXEC command, the whole transaction aborts, and EXEC returns a Null reply to notify that the transaction failed.

37

Errors inside transactionsDuring a transaction it is possible to encounter two kind of command errors:● A command may fail to be queued, so there may be an

error before EXEC is called. For instance: syntactically wrong command or another critical condition

● A command may fail after EXEC is called, for instance since we performed an operation against a key with the wrong value (like calling a list operation against a string value).

38

Redis does not support roll backs

● Redis commands can fail only if called with a wrong syntax, or against keys holding the wrong data type.

● Redis is internally simplified and faster because it does not need the ability to roll back.

39

Redis Publish/Subscribe● Redis has first-class support for publishing messages

and subscribing to channels implemented by SUBSCRIBE, PUBLISH and UNSUBSCRIBE commands.

● Messages published into channels, without knowledge of what subscribers there may be.

● Subscribers express interest in one or more channels, and only receive messages that are of interest, without knowledge of what publishers there are.

40

Pattern-matching subscriptions

● The Redis Pub/Sub supports pattern-matching subscriptions

● PSUBSCRIBE tweets.* will receive message from both tweets.john and tweets.sean channels

● PUNSUBSCRIBE tweets.* will unsubscribe the client from this pattern

41

Redis Pub/Sub example (DEMO)Subscriber 1SUBSCRIBE foo

Subscriber 2SUBSCRIBE foo boo

PublisherPUBLISH foo “hello”PUBLISH boo “world”

42

Request/Response protocol and RTT● Redis request is accomplished with the following two

steps:○ The client sends a query to the server, and reads from

the socket, usually in a blocking way, for the server response.

○ The server processes the command and sends the response back to the client.

● RTT (Round Trip Time) - time for the packets to travel from the client to the server, and back from the server to the client to carry the reply.

43

What’s wrong with blocking request● Four commands sequence is something like this:

○ Request: INCR X○ Response: 1○ Request: INCR X○ Response: 2○ Request: INCR X○ Response: 3○ Request: INCR X○ Response: 4

● RTT time is 250 ms, Redis server is able to process 100k request per second, so will be able to process max 4 requests per second

44

Redis Pipelining● Pipelining is a way to send multiple commands to the server

without waiting for the replies at all, and finally read the replies in a single step.

● Pipelined version of four commands sequence:○ Request: INCR X○ Request: INCR X○ Request: INCR X○ Request: INCR X○ Response: 1○ Response: 2○ Response: 3○ Response: 4

45

Redis Pipelining recommendations

● While client sends commands using pipelining, server forced to queue the responses in memory.

● To reduce memory usage when sending a lot of commands better to send them as batches having a reasonable number.

46

Links and Literature

● redis.io/documentation● The Little Redis Book by Karl Seguin● Redis in Action by Josiah L. Carlson● Seven databases in seven weeks (The

Pragmatic Bookshelf, 2012)

47

Thank You!

48