Redis Developers Day 2015 - Secondary Indexes and State of Lua
-
Upload
itamar-haber -
Category
Software
-
view
532 -
download
1
Transcript of Redis Developers Day 2015 - Secondary Indexes and State of Lua
Redis Developers Day, Oct 19th 2015, London
Itamar Haber, Chief Developer [email protected]
New Data Structure or New API?
3
A range of queries that’re non-trivial with Redis
• Sorted Sets are awesome for answering single score queries• Composite scores (53 bits space) are also very useful• With ZRANGEBYLEX you can do any range queries• Have you read the new page at http://redis.io/topics/indexes?
• But a common query “category” that’s non-trivial and inefficient to do with Redis is ranges over n-dimensions
4
Example: 2-dimensional query
Given users ages and salaries – find all that are between x1 and x2 years old and earn between y1 and y2
Sorted Sets zquery “Algorithm”:1. ZUNIONSTORE tmp 1 ages2. ZREMBYRANGE ages –inf (x13. ZREMBYRANGE ages (x2 +inf4. ZINTERSTORE tmp 2 tmp salaries WEIGHTS 0 15. reply = ZRANGEBYSCORE tmp [y1 [y26. [DEL tmp]
5
A proof of concept: Quadtree in Redis Hash
https://gist.github.com/itamarhaber/c1ffda42d86b314ea701
Idea: store nodes as hash fields, lazy loading
Benchmark results on 100K points
zquery: 2.41 requests per secondQuadtree query: 73.69 requests per second
k-d trees are similar, but for k dimensions
6
So?
1. Proposed to @antirez to add k-d trees2. Have you read the new page at http://redis.io/topics/indexes? 3. We still need, regardless the data structure, a higher level API to
do effective querying (reminder: there’s no STORE in ZRANGEBYLEX)
Not a real command but:
SELECT myzset WHERE cond1 cond2 cond3…
The State of Lua
8
We need a debugger
You can trace: redis.log, PUBLISH, ECHO…You can simulate: http://www.trikoder.net/blog/redis-and-lua-scripts-new-workflow-89But you can’t debug “in vitro” (https://github.com/RedisLabs/redis-lua-debugger/ - doesn’t work since https://github.com/antirez/redis/commit/30278061cc834b4073b004cb1a2bfb0f195734f7)Two approaches:1. Embedded debugger (https://github.com/slembcke/debugger.lua)2. Remote debugger (ZeroBrane?)
9
Other Lua Stuff
• Calling a script from a script (f_<sha1> hack? Modules?)• Persist Lua scripts and load them from disk, also on startup• Keyspace notifications/PubSub triggered Lua scripts ~= triggers
(mattsta)
10
Lua PRs
• Removes dofile() from Lua by evilpacket: antirez/redis: https://github.com/antirez/redis/pull/732
• Lua-based maxmemory eviction policies by lamby: https://github.com/antirez/redis/pull/2319
• new Lua memory limit by dovkfir: antirez/redis: https://github.com/antirez/redis/pull/2505
• Adds Lua transaction support by josiahcarlson: antirez/redis: https://github.com/antirez/redis/pull/2701
• No PR, but Redis Labs once suggested doing operation replication to avoid long running scripts on slaves
Thank you