@Redis Developers Day 2014

1st talk: day to day challenges

● ±150 new databases every day (>60,000 total)

● A mix of very small (few MB) and extremely large (TB+)

datasets

● PAYG model that can grow to any size by changing

maxmemory

● ±250 node failure events (~10 every month) – w/o

upgrades

● 5-6 complete data-center outages so far

● Supporting/managing DBs with zero knowledge of

application

● Running a modified version of Redis from day 1

2nd talk: Extending Redis to Flash

● Redis #1 hindering factor is the cost of

deployment ➔ RAM costs x10 SSD and x250 HDD

● Advanced flash technologies:➔ 1M IOPS (4KB)

➔ Latency: DIMM – 1-2µsec; Plugable – 100µsec

Why do we modify Redis?

● Dense multi-tenant architecture and still

maintaining SLAs (single-tenant, dedicated

instances supported)

● Efficient large-scale system administration

● Clustering (in production since 06/2012)

How do we manage changes?

● Many custom modifications are of no use for anyone else,

so we don’t bother

● Fixes, minor changes to upstream ASAP

● Periodic batches of more significant features

o Low hit rate, community view vs. enterprise/service

provider view

● Our “playground” fork

https://github.com/redislabs/redis

Skin-of-our-teeth replication

● DB is growing, sync to a bigger instance

● Lots of writes, output buffers and COW bloat

● Redis has a point of no return

⇓

Off-load some of this to the slave

Flush output buffer to slave, Mux RDB

Improved Replication Performance

Master Memory

Master Disk Network Slave Disk Slave Memory

● Eliminate disk impact on replication performance (pain point)● Current Redis scheme can still be better sometimes

Lua Replication

● Old debate: SCRIPT vs Commandso In some cases performance is dramatic

o We think both options are needed (user-controlled)

External RDB processing

● We do it often, and not only us:

https://github.com/sripathikrishnan/redis-rdb-tools

● Python is too slow for serious work

● We want to avoid code duplication

● Built-in rdb.c was too hard-coded

● Alternative SAX-like (callback-based) parser

● Maybe even librdb.a?

Resource Management

● maxmemory can be confusing (think eviction/OOM

because of output buffers)

● How big can we expect the actual process? Master

switches are missing (e.g. max allocated network buffers)

● Graceful handling of output buffer limits

● Fragmentation issues

o jemalloc and the 12% magic

o active defrag and bigger ideas for bigger problems

More Resource Management

● Automatic process oom_score_adj settings to

guard data in low memory events

● Automatically reject BGSAVE and

BGREWRITEAOF or terminate child on low

memory

Cluster features

● Lua key ranges / patterns

● Global keys

Extending Redis to Flash - goals

● More (cheap) memory not better persistence

● 100% compatibility with Redis

● Pluggable storage back-ends / non-volatile DIMM

● Utilize storage concurrency

● Use case: limited set of hot objects over large

keyspace

● Different from other approaches: Ledis, ARDB,

SSDB

Extending Redis to Flash - blocks

Main thread

Clients

I/O jobqueues

I/O Threads

K/V storage backend API

Storage backend

Extending Redis to Flash - moar

● Still being prepared for prime time

● Planned to be open sourced once ready

Thank You