Large Scale NoSql DB Migration Under Fire - Ido Barkan - DevOpsDays Tel Aviv 2017
-
Upload
devopsdays-tel-aviv -
Category
Technology
-
view
98 -
download
1
Transcript of Large Scale NoSql DB Migration Under Fire - Ido Barkan - DevOpsDays Tel Aviv 2017
Large Scale NoSQL DB Migration
- Under Fire -
Ido Barkan
- Proprietary & Confidential -
Agenda
● What is Appsflyer?● Motivation and limitations● War plan● What have we learned?
What is Appsflyer?~25B events per day
Installs DB- Zoom in
app installs writer
2K write IOPS
app events enrichments 180k read
IOPS
We need to replace CouchBase
● Scale out is out of control(45 r3.4xl instances)
● XDCR can’t keep up● Daily backups take more than a day● Paid support is not good enough
chosen solution: Aerospike
Rules of the game- starting point
● # of records: ~ 2,000,000,000 (repl. factor 3)
● Read intensive: 180K read + 2K write IOPS
● # of machines: 45 r3.4xlarge● Cost: ~ 30,000 $ per month● Time frame: a few months.
Mission critical
Why is data migration harderthan in a relational DB?
Relational DB NoSql DB
Data types conventional blob, JSON, specialized bins
Vendors a few 100+
Query Language
SQL N1QL, SQL, JavaScript, HTTP/REST-- creating views/ MapReduce style
Schema Yes Maybe
Migration tools
Yes DIY
Rules of the game
No DowntimeNo data loss
War plan- easy part
● Write new installs/updates to BOTH● Continue reading from the OLD DBApplication
Breathing (testing) time
test test test● Sizing RAM/Disk● Expected load● Metrics● Alerting● Logs● Backups
Easy part (Cont.)
● Write new installs/updates to NEW DB● Read from BOTH DBs
● A fraction of records (not VIP)
● Do NOT delete yet!● Have metrics on
misses
Breathing timeTest Test Test● Sizing (again)● Expected load (with reads)● Metrics (again)
Is that it?
If your data is short lived- you are mostly done.
Otherwise...
`Manual` partMigrating the long tail
Dumperview (query)dump keys/records
Kafka as a buffer
Loader-> write(do not overwrite)
values
Detour: DB views
function (doc, meta) {emit(meta.expiration, doc.id);}
key (sorting) value
Map function (JS)
● Record itself is NOT a part of the view
`Manual` partMigrating the long tail
Dumperview (query)dump keys/records
Kafka as a buffer
Loader-> write(do not overwrite)
values
New Cluster
● # of records: ~ 2,000,000,000 ● 2AZ- repl. factor 2 in each one● IOPS, same and growing● # of machines: 2 x 5 i3.4xlarge● Cost: ~ 10,000 $ per month (⅓)
What have we learned?
1. Plan, Plan, Plan.2. Go easy: test a sample before you start
(1 app / API / Customer).3. Breath (read: test) between stages.4. Do NOT delete until you have to.5. slower -> safer (-> $$$).
Thank you!