Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
-
Upload
theo-hultberg -
Category
Documents
-
view
222 -
download
0
Transcript of Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
![Page 1: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/1.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 1/80
SHORTCUTS AROUND THEMISTAKES I’VEMADE SCALINGMONGODB
Theo, Chief
Architect
at
onsdag 21 september 11
![Page 2: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/2.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 2/80
What we doWe want to revolutionize the digital advertising
industry by showing that there is more to adanalytics than click through rates.
onsdag 21 september 11
![Page 3: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/3.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 3/80
Ads
onsdag 21 september 11
![Page 4: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/4.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 4/80
Data
onsdag 21 september 11
![Page 5: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/5.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 5/80
Assembling sessionsexposure
ping
ping
ping ping
ping
event
event
ping
session➔ ➔
onsdag 21 september 11
![Page 6: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/6.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 6/80
Crunching
session
session
session
session
sessionsession
sessi on sessi on
session
session
session
session
session
➔ ➔ 42
onsdag 21 september 11
![Page 7: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/7.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 7/80
Reports
onsdag 21 september 11
![Page 8: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/8.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 8/80
What we do Track ads, make pretty reports.
onsdag 21 september 11
![Page 9: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/9.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 9/80
That doesn’t sound so hard
onsdag 21 september 11
![Page 10: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/10.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 10/80
That doesn’t sound so hardWe don’t know when sessions end
onsdag 21 september 11
![Page 11: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/11.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 11/80
That doesn’t sound so hardWe don’t know when sessions end
There’s a lot of data
onsdag 21 september 11
![Page 12: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/12.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 12/80
That doesn’t sound so hardWe don’t know when sessions end
There’s a lot of data
It’s all done in (close to) real time
onsdag 21 september 11
![Page 13: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/13.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 13/80
Numbers
onsdag 21 september 11
![Page 14: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/14.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 14/80
Numbers40 Gb data
onsdag 21 september 11
![Page 15: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/15.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 15/80
Numbers40 Gb data
50 million documents
onsdag 21 september 11
![Page 16: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/16.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 16/80
Numbers40 Gb data
50 million documents
per day
onsdag 21 september 11
![Page 17: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/17.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 17/80
How we use MongoDB
onsdag 21 september 11
![Page 18: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/18.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 18/80
How we use MongoDB“Virtual memory” to offload data while we wait
for sessions to finish
onsdag 21 september 11
![Page 19: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/19.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 19/80
How we use MongoDB“Virtual memory” to offload data while we wait
for sessions to finishShort time storage (<48 hours) for batch jobs
onsdag 21 september 11
![Page 20: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/20.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 20/80
How we use MongoDB“Virtual memory” to offload data while we wait
for sessions to finishShort time storage (<48 hours) for batch jobs
Metrics storage
onsdag 21 september 11
![Page 21: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/21.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 21/80
Why we use MongoDB
onsdag 21 september 11
![Page 22: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/22.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 22/80
Why we use MongoDBSchemalessness makes things so much easier,
the data we collect changes as we come upwith new ideas
onsdag 21 september 11
![Page 23: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/23.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 23/80
Why we use MongoDBSchemalessness makes things so much easier,
the data we collect changes as we come upwith new ideas
Sharding makes it possible to scale writes
onsdag 21 september 11
![Page 24: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/24.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 24/80
Why we use MongoDBSchemalessness makes things so much easier,
the data we collect changes as we come upwith new ideas
Sharding makes it possible to scale writes
Secondary indexes and rich query language aregreat features (for the metrics store)
onsdag 21 september 11
![Page 25: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/25.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 25/80
Why we use MongoDBSchemalessness makes things so much easier,
the data we collect changes as we come upwith new ideas
Sharding makes it possible to scale writes
Secondary indexes and rich query language aregreat features (for the metrics store)
It’s just… nice
onsdag 21 september 11
![Page 26: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/26.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 26/80
Btw.
onsdag 21 september 11
![Page 27: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/27.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 27/80
Btw.We use JRuby, it’s awesome
onsdag 21 september 11
![Page 28: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/28.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 28/80
A story in 7 iterations
onsdag 21 september 11
![Page 29: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/29.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 29/80
secondary indexes and updates1st iteration
onsdag 21 september 11
![Page 30: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/30.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 30/80
secondary indexes and updates1st iterationOne document per session, update as new
data comes alongOutcome: 1000% write lock
onsdag 21 september 11
![Page 31: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/31.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 31/80
#1Everything is aboutworking around theGLOBALWRITE
LOCK
onsdag 21 september 11
![Page 32: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/32.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 32/80
MongoDB 2.0.0
db.coll.update({_id: "xyz"}, {$inc: {x: 1}}, true)
db.coll.update({_id: "abc"}, {$push: {x: “...”}}, true)
onsdag 21 september 11
![Page 33: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/33.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 33/80
MongoDB 1.8.1
db.coll.update({_id: "xyz"}, {$inc: {x: 1}}, true)
db.coll.update({_id: "abc"}, {$push: {x: “...”}}, true)
onsdag 21 september 11
![Page 34: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/34.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 34/80
using scans for two step assembling2nd iterationInstead of updating, save each fragment, then
scan over _id to assemble sessions
onsdag 21 september 11
![Page 35: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/35.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 35/80
using scans for two step assembling2nd iterationOutcome: not as much lock, but still not great
performance. We also realised we couldn’tremove data fast enough
onsdag 21 september 11
![Page 36: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/36.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 36/80
#2Everything is aboutworking around theGLOBALWRITE
LOCK
onsdag 21 september 11
![Page 37: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/37.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 37/80
#3Give a lot of
thought to yourPRIMARY KEY
onsdag 21 september 11
![Page 38: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/38.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 38/80
partitioning
3rd iteration
onsdag 21 september 11
![Page 39: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/39.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 39/80
partitioning
3rd iterationWe came up with the idea of partitioning the
data by writing to a new collection every hour
onsdag 21 september 11
![Page 40: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/40.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 40/80
partitioning
3rd iterationWe came up with the idea of partitioning the
data by writing to a new collection every hourOutcome: lots of complicated code, lots of
bugs, but we didn’t have to care about
removing data
onsdag 21 september 11
![Page 41: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/41.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 41/80
#4Make sure you can
REMOVE
OLD DATA
onsdag 21 september 11
![Page 42: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/42.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 42/80
sharding
4th iteration
onsdag 21 september 11
![Page 43: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/43.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 43/80
sharding
4th iteration To get around the global write lock and get
higher write performance we moved to a
sharded cluster.
onsdag 21 september 11
![Page 44: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/44.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 44/80
sharding
4th iteration To get around the global write lock and get
higher write performance we moved to a
sharded cluster.
Outcome: higher write performance, lots of
problems, lots of ops time spent debugging
onsdag 21 september 11
![Page 45: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/45.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 45/80
#5Everything is aboutworking around theGLOBALWRITE
LOCK
onsdag 21 september 11
![Page 46: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/46.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 46/80
#6
SHARDINGIS NOT A
SILVER BULLETand it’s buggy,
if you can, avoid it
onsdag 21 september 11
![Page 47: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/47.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 47/80
onsdag 21 september 11
![Page 48: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/48.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 48/80
#7
IT WILL FAILdesign for it
onsdag 21 september 11
![Page 49: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/49.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 49/80
onsdag 21 september 11
![Page 50: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/50.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 50/80
onsdag 21 september 11
![Page 51: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/51.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 51/80
moving things to separate clusters5th iteration
onsdag 21 september 11
![Page 52: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/52.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 52/80
moving things to separate clusters5th iterationWe saw very different loads on the shards and
realised we had databases with very different
usage patterns, some that made autosharding
not work. We moved these off the cluster.
onsdag 21 september 11
![Page 53: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/53.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 53/80
moving things to separate clusters5th iterationWe saw very different loads on the shards and
realised we had databases with very different
usage patterns, some that made autosharding
not work. We moved these off the cluster.
Outcome: a more balanced and stable cluster
onsdag 21 september 11
![Page 54: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/54.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 54/80
#8Everything is aboutworking around theGLOBALWRITELOCK
onsdag 21 september 11
![Page 55: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/55.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 55/80
#9
ONE DATABASEwith one usage patternPER CLUSTER
onsdag 21 september 11
![Page 56: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/56.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 56/80
#10
MONITOR
EVERYTHINGlook at your healthgraphs daily
onsdag 21 september 11
![Page 57: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/57.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 57/80
monster machines6th iteration
onsdag 21 september 11
![Page 58: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/58.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 58/80
monster machines6th iterationWe got new problems removing data and
needed some room to breathe and think
onsdag 21 september 11
![Page 59: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/59.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 59/80
monster machines6th iterationWe got new problems removing data and
needed some room to breathe and think
Solution: upgraded the servers to High-
Memory Quadruple Extra Large (with cheese).
onsdag 21 september 11
![Page 60: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/60.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 60/80
monster machines6th iterationWe got new problems removing data and
needed some room to breathe and think
Solution: upgraded the servers to High-
Memory Quadruple Extra Large (with cheese).
♥I
onsdag 21 september 11
![Page 61: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/61.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 61/80
#11Don’t try to scale upSCALE OUT
onsdag 21 september 11
![Page 62: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/62.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 62/80
#12When you’re out of ideas
CALL THE EXPERTS
onsdag 21 september 11
![Page 63: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/63.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 63/80
partitioning (again) and pre-chunking7th iteration
onsdag 21 september 11
![Page 64: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/64.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 64/80
partitioning (again) and pre-chunking7th iterationWe rewrote the database layer to write to a
new database each day, and we created all
chunks in advance. We also decreased the size
of our documents by a lot.
onsdag 21 september 11
![Page 65: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/65.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 65/80
partitioning (again) and pre-chunking7th iterationWe rewrote the database layer to write to a
new database each day, and we created all
chunks in advance. We also decreased the size
of our documents by a lot.
Outcome: no more problems removing data.
onsdag 21 september 11
![Page 66: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/66.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 66/80
#13Smaller objects means a smaller database, and a smaller database meansLESS RAM NEEDED
onsdag 21 september 11
![Page 67: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/67.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 67/80
#14Give a lot of
thought to yourPRIMARY KEY
onsdag 21 september 11
![Page 68: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/68.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 68/80
#15Everything is aboutworking around theGLOBALWRITELOCK
onsdag 21 september 11
![Page 69: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/69.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 69/80
#16Everything is aboutworking around theGLOBALWRITELOCK
onsdag 21 september 11
![Page 70: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/70.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 70/80
KTHXBAI
@iconara
architecturalatrocities.comburtcorp.com
onsdag 21 september 11
![Page 71: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/71.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 71/80
Since we got time…
onsdag 21 september 11
![Page 72: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/72.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 72/80
Safe modeTips
onsdag 21 september 11
![Page 73: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/73.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 73/80
Safe modeTips
Run every Nth insert in safe mode
onsdag 21 september 11
![Page 74: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/74.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 74/80
Safe modeTips
Run every Nth insert in safe mode
This will give you warnings when bad thingshappen; like failovers
onsdag 21 september 11
![Page 75: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/75.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 75/80
Avoid bulk insertsTips
onsdag 21 september 11
![Page 76: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/76.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 76/80
Avoid bulk insertsTips
Very dangerous if there’s a possibility of
duplicate key errors
onsdag 21 september 11
![Page 77: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/77.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 77/80
EC2
Tips
onsdag 21 september 11
![Page 78: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/78.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 78/80
EC2
Tips
You have three copies of your data, do you
really need EBS?
onsdag 21 september 11
![Page 79: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/79.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 79/80
EC2
Tips
You have three copies of your data, do you
really need EBS?
Instance store disks are included in the price
and they have predictable performance.
onsdag 21 september 11
![Page 80: Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB](https://reader031.fdocuments.us/reader031/viewer/2022021213/577d25fc1a28ab4e1e9ffd26/html5/thumbnails/80.jpg)
8/4/2019 Shortcuts Around the Mistakes I've Made Trying to Scale MongoDB
http://slidepdf.com/reader/full/shortcuts-around-the-mistakes-ive-made-trying-to-scale-mongodb 80/80
EC2
Tips
You have three copies of your data, do you
really need EBS?
Instance store disks are included in the price
and they have predictable performance.
m1.xlarge comes with 1.7 TB of storage.