Whats new in Redis v3.2
-
Upload
itamar-haber -
Category
Technology
-
view
1.712 -
download
0
Transcript of Whats new in Redis v3.2
What's new
#RedisTLV Jan 21st 2016
in v3.2
v3.2RC1 - TL;DR
~/src/redis$ git rev-list 3.0..3.2 --count1606~/src/redis$ git diff 3.0..3.2 --shortstat262 files changed, 46931 insertions(+), 28720 deletions(-)
Today's meetup: ~6.379 topics1. Geospatial indices2. Quadtree & redimension3. Internals deep dive4. Effects-based replication5. Security & `protected-mode`6. Redis Lua debugger
About our sponsor- Home of open source Redis- The commercial provider for managed and downloadable Redis solutions- HQ @ Mountain View, R&D @ Italy & Israel- ~70 employees & growing fast - we're hiring :)
The newsletter: Redis Watch
http://bit.ly/RedisWatch
Geospatialindices
OSS crossover: Redis 2.x -> ardb ->
2D spatial index ->Matt Stancliff @mattsta ->
Salvatore Sanfilippo @antirez ->Redis 3.2
Geospatial indices redhistory
The Geohash- A geocoding system, hierarchical spatial grid - The hash value maps to a location (lon, lat)- Is usually base-32 encoded (.e.g sv8y8v66bt0)- By Gustavo Niemeyer, in the public's domain since 28 Feb. 2009
Prim
e Me
ridian
= 0°lon -180° lon 180°
0 1
0 1110
Equator = 0°
lat -85.05112878
lat -85.05112878
0 110
10
111
Geospatial Indices in Redis- Redis' Geohashes are 52-bit integers (~0.6m)- Redis' Sorted Sets' scores are IEEE 754 floats, i.e. 53-bit integer precision…
BINGO!
New! GEO API, part 1Add a point:GEOADD key longitude latitude member [...]
Get longitude & latitude / geohash:GEOPOS|GEOHASH key member [...]
Get the distance between two points:GEODIST key member1 member2 [unit]
GEO API - "Demo"127.0.0.1:6379> GEOADD g 34.84076 32.10942 RL@TLV(integer) 1127.0.0.1:6379> GEOADD g -122.0678325 37.3775256 RL@MV(integer) 1127.0.0.1:6379> GEODIST g RL@TLV RL@MV km"11928.692170353959"127.0.0.1:6379> GEOADD g 34.8380433 32.1098095 Hudson(integer) 1
Size (almost) doesn't matter127.0.0.1:6379> GEOHASH g RL@TLV Hudson1) "sv8y8v66bt0"2) "sv8y8v2m1n0"
- Shorter hashes -> "same" location, bigger area- Close spatial proximity usually means a shared hash prefix
New! GEO API, part 2Search for members in a radial area:GEORADIUS key longitude latitude radius unit ...GEORADIUSBYMEMBER key member radius unit ...
Overthrows ZREVRANGEBYSCORE!!! #RedisTrivia
Delete a point - no GEOREM for you:ZREM key member [...]
GEO Use casesAny ideas?
Quadtree &redimension
Multi-dimensional queriesSELECT idFROM usersWHERE age > 35 ANDsalary BETWEEN 250 AND 350
http://stackoverflow.com/questions/32911604/intersection-of-two-or-more-sorted-sets
The Redis way - "ZQUERY"ZUNIONSTORE t 1 age WEIGHTS 1ZREMRANGEBYSCORE t -inf (25ZREMRANGEBYSCORE t (35 +infZINTERSTORE t 2 t salary WEIGHTS 0 1ZRANGEBYSCORE t 250 350DEL t
Works, but not too efficient.
Would indexing the data help?rqtih.lua: Another Redis Wayhttps://gist.github.com/itamarhaber/c1ffda42d86b314ea701
rqtih.lua is about 32.5 times faster than ZQUERY on 100K users (age & salary)
rqtih.lua?!? A PoC forR - Redis, duhQT - QuadtreeIH - In Hash.LUA - "object oriented", JiT reads, delayed writes
Trillustrationa
b
d
e
f
g
h
i
a
d b f
h g c e i
c
{/ : x, y, w, h, {a}/00/ : x, y, w, h, {d}/01/ : x, y, w, h, {b}
...}
* Node capacity = 1
A new Redis data structure?- Discussions in proximity to Redis Developers Day 2015 (London)- k-d tree: similar principles for k dimensions, but complex complexity- Outcomes: topics/indexes & experimental API that uses existing data types (Zorted & Hash)
Redimension: k-d query API@antirez's idea: interleave the dimensions, store "score"+data in a Zorted Set for lexicographical ranges, maintain a Hash for lookups
redimension.rb - implementation by @antirezredimension.lua - port by @itamarhaber
Redimension "Demo"~/src/lua-redimension$ redis-cli SCRIPT LOAD "$(cat redimension.lua)""4abdad23c459145cbd658c991c0c8ad93d984d91"~/src/lua-redimension$ redis-cli EVALSHA 4abdad23c459145cbd658c991c0c8ad93d984d91 0 1) "KEYS[1] - index sorted set key" 2) "KEYS[2] - index hash key" 3) "ARGV[1] - command. Can be:"
Redimension "Demo", 2 4) " create - create an index with ARGV[2] as dimension and ARGV[3] as precision" 5) " drop - drops an index" 6) " index - index an element ARGV[2] with ARGV[3]..ARGV[3+dimension] values" 7) " unindex - unindex an element ARGV[2] with ARGV[3]..ARGV[3+dimension] values"
Redimension "Demo", 3 9) " update - update an element ARGV[2] with ARGV[3]..ARGV[3+dimension] values"10) " query - query using ranges ARGV[2], ARGV[3]..ARGV[2+dimension-1], ARGV[2+dimension]"11) " fuzzy_test - fuzzily tests the library on ARGV[2] dimension with ARGV[3] items using ARGV[4] queries"
redimension.next()- Currently just an experiment- Many improvements still needed- Planned to become a part of the core project- Need more feedback WRT functionality & API- Any ideas?
Internals deep diveOran Agra @RedisLabs
changes that made it (or didn’t) to OSS redis● merged into 3.0
○ Fix a race condition in processCommand() with freeMemoryIfNeeded()○ diskless replication fixes○ psync fixes○ fixes in LRU eviction (dict random keys during rehasing)
● merged into 3.2○ sds optimizations○ jemalloc size class optimization
● changes not merged yet○ diskless slave replication○ dict.c improvements
● other changes i didn’t get to push yet○
nothing is user facing. only optimizations, and fixes 8-
(
diskless replication● how normal replication works.
master->fork->rdb on disk->main process streams to slaveslave->save to disk while serving clients->flushdb->load rdb
● disadvantages of diskless replication○ slaves must connect together○ slave side flush before RDB was fully received○ on slow network, longer fork duration
● a word about fork() and CoW?
diskless replication benchmark (replication time)
two instances of r3.2xl (60GB ram, with 160GB SSD), 4,000,000 string keys of 1k random data. (consuming 52GB of RAM), 19GB RDB file.
fully disk based: 513 secondsonly master diskless: 365 secondsfully diskless: 231 secondsonly salve is diskless: 360 seconds
● we all know what fragmentation is● history: on the search for the ultimate allocator● how an allocator works (bins) to overcome that● a word about virtual address space vs OS pages
○ RSS = VM pages mapped to physical RAM● what’s internal fragmentation / used_memory
(maxmemory) includes internal frag● RSS = used_memory (+external frag)
○ external frag are unused bins, and pages●
unused
unused
fragmentation
allocators16 byte bins pool
32 byte bins pool
internal fragmentation
22 byte
18 bytes
17 bytes
30 bytes
28 bytes
adding bin 24 bytes pull to jemalloc
used by: dictEntry, listNode, etc
redis-cli debug populate 10000000original code's used_memory: 1,254,709,872with patch used_memory: 1,094,714,048memory optimization: 14%
size classes:8162432404856648096…...
p1 FS cachekernel p1 FS
cache p2p1+p2 p2 p2 p2FS
cacheFS cache
FS cache
physical ram (4k pages)
process 1(virtual address space) process 2(virtual address space)
unmapped
unmapped
unmapped
unmapped
unmapped
4k page
4k page
4k page
4k pagecan be returned to os (won’t be rss anymore)
4k page
ABCDEF\n
char*
used unusedfree
4 bytes 4 bytesold sds header
● grows in place(sometimes no need for realloc)○ although realloc may nop instead of give new pointer and do
memcpy● no need for strlen (search for null terminator)● can be used in normal string functions like printf
struct sdshdr { unsigned int len; unsigned int free; char buf[];};
new sds headerABCDEF\n
char*
used unusedfree
4 bytes 4 bytesold sds header
ABCDEF\nused unusable
5 bits 3 bits
type5bit
ABCDEF\nused unusedallocated
1 byte 1 byte 1 byte
type8bit
ABCDEF\nused unusedallocated
2 bytes 2 bytes 1 byte
type16bit
ABCDEF\nused unusedallocated
4 bytes 4 bytes 1 byte
type32bit
ABCDEF\nused unusedallocated
8 bytes 8 bytes 1 byte
type64bit
struct __attribute__ ((__packed__)) sdshdr5 { unsigned char flags; /* 3 lsb of type, and 5 msb of string length */ char buf[];};
struct __attribute__ ((__packed__)) sdshdr8 { uint8_t len; /* used */ uint8_t alloc; /* excluding the header and null terminator */ unsigned char flags; /* 3 lsb of type, 5 unused bits */ char buf[];};
struct __attribute__ ((__packed__)) sdshdr16 { uint16_t len; /* used */ uint16_t alloc; /* excluding the header and null terminator */ unsigned char flags; /* 3 lsb of type, 5 unused bits */ char buf[];};
struct __attribute__ ((__packed__)) sdshdr32 { uint32_t len; /* used */ uint32_t alloc; /* excluding the header and null terminator */ unsigned char flags; /* 3 lsb of type, 5 unused bits */ char buf[];};
struct __attribute__ ((__packed__)) sdshdr64 { uint64_t len; /* used */ uint64_t alloc; /* excluding the header and null terminator */ unsigned char flags; /* 3 lsb of type, 5 unused bits */ char buf[];};
sds size classesdebug populate 10000000used_memory of original code: 1,254,709,872used_memory with new code: 1,078,723,024memory optimization: 16%
Intermission600 seconds
Effects replication
Script replication before v3.2Lua scripts are pushed down to the slaves for local execution. This reduces wire traffic in cases such as:for i = 1, 1000000 do
redis.call('LPUSH', KEYS[1], i)end
Script replication caveatsCompute-intensive scripts (e.g. ZQUERY) waste CPU time because they are run:- 1+number of slaves times: wasteful- When recovering from AOF: really bad
And then there's also...
Free willvs.
> EVAL "redis.call('SET', KEYS[1], redis.call('TIME')[1])" 1 foo
(error) ... Write commandsnot allowed after nondeterministic commands
Script replication in v3.2- Same defaults- NEW! redis.replicate_commands()
causes the script's effects to be replicated- NEW! redis.set_repl(...)redis.REPL_[ALL|NONE|AOF|SLAVE]
Effect-based replication usesAny ideas?
Security
"A few things about Redis security"
"The Redis security model is: it’s totally insecure to let untrusted clients access the system, please
protect it from the outside world yourself...Let’s crack Redis for fun and no profit…"
HOWTO: http://antirez.com/news/96
The totally unexpected result
Script kiddies, cybercriminals and white hackers
3 critical points about security
Honesty is always the best option. That said:1. Never leave an unprotected server open to the outside world
2. If your server has been compromised, burn it
3. Always read the documentation
NEW! protected-mode directive
By default is enabled -> a breaking upgrade!When (protected-mode && !requirepass && !bind):- Allow only 127.0.0.1, ::1 or socket connections- DENY (with the longest message ever!) others
Protection in Action - "Demo"-DENIED Redis is running in protected mode because protected mode is enabled, no bind address was specified, no authentication password is requested to clients. In this mode connections are only accepted from the loopback interface. If you want to connect from external computers to Redis you may adopt one of the following solutions: 1) Just disable protected mode sending the command 'CONFIG SET protected-mode no' from the loopback interface by connecting to Redis from the same host the server is running, however MAKE SURE Redis is not publicly accessible from internet if you do so. Use CONFIG REWRITE to make this change permanent. 2) Alternatively you can just disable the protected mode by editing the Redis configuration file, and setting the protected mode option to 'no', and then restarting the server. 3) If you started the server manually just for testing, restart it with the '--protected-mode no' option. 4) Setup a bind address or an authentication password. NOTE: You only need to do one of the above things in order for the server to start accepting connections from the outside.
Redis Lua debugger
NEW! Integrated Lua debugger
- Step-by-step journey through history- LDB: SCRIPT DEBUG yes/sync/no- Demo: redis-cli, ZeroBrane Studio IDE pluginhttp://redis.io/topics/ldbhttps://redislabs.com/blog/zerobrane-studio-plugin-for-redis-lua-scripts