Scaling with mongo db - SF Mongo User Group 7-19-2011

download Scaling with mongo db - SF Mongo User Group 7-19-2011

of 36

  • date post

    15-Jan-2015
  • Category

    Technology

  • view

    3.222
  • download

    0

Embed Size (px)

description

My talk from SF Mongo User Group on 7-19-2011

Transcript of Scaling with mongo db - SF Mongo User Group 7-19-2011

  • 1. Scaling with MongoDBJared Rosoff (jsr@10gen.com) - @forjared

2. 3. How do we do it today?
We use a relational database but
We dont use joins
We dont use transactions
We add read-only slaves
We added a caching layer
We de-normalized our data
We implemented custom sharding
We buy bigger servers
4. Hows that working out for you?
5. Costs go up
6. Productivity goes down
7. By engineers, for engineers
8. The landscape
Memcached
Key / Value
Scalability & Performance
RDBMS
Depth of functionality
9. Scaling your app
Use documents
Indexes make me happy
Knowing your working set
Disks are the bottleneck
Replication makes reading fun
Sharding for profit
10. Scaling your data model
11. Documents
{
author : "roger",
date : "Sat Jul 24 2010 19:47:11 GMT-0700 (PDT)",
text : "Spirited Away",
tags : [ "Tezuka", "Manga" ],
comments : [
{
author : "Fred",
date : "Sat Jul 24 2010 20:51:03 GMT-0700 (PDT)",
text : "Best Movie Ever
}
]
}
12. Disk Seeks & Data Locality
Read = really really fast
Seek = 5+ ms
13. Disk Seeks & Data Locality
Post
Comment
Author
14. Disk Seeks & Data Locality
Post
Author
Comment
Comment
Comment
Comment
Comment
15. Optimized indexes
16. Table scans
Find where x equals 7
1
2
3
4
5
6
7
Looked at 7 objects
17. Tree Lookup
Find where x equals 7
4
6
2
7
5
3
1
Looked at 3 objects
18. Random Index
Entire index must fit in RAM
19. Right Aligned
Only small portion in RAM
20. Working set size
21. Working Set
Active Documents + Used Indexes
RAM
Disk
22. Page Fault
App requests document
Document not in memory
Evict a page from memory
Read block from disk
Return document from memory
App
1
5
2
RAM
3
4
Disk
23. Figuring out working Set
> db.foo.stats()
{
"ns" : "test.foo",
"count" : 1338330,
"size" : 46915928,
"avgObjSize" : 35.05557523181876,
"storageSize" : 86092032,
"numExtents" : 12,
"nindexes" : 2,
"lastExtentSize" : 20872960,
"paddingFactor" : 1,
"flags" : 0,
"totalIndexSize" : 99860480,
"indexSizes" : {
"_id_" : 55877632,
"x_1" : 43982848
},
"ok" : 1
}
Size of data
Average document size
Size on disk (and in memory!)
Size of all indexes
Size of each index
24. Disk configurations
25. Single Disk
~200 seeks / second
26. RAID0
~200 seeks / second
~200 seeks / second
~200 seeks / second
27. RAID10
~400 seeks / second
~400 seeks / second
~400 seeks / second
28. replication
29. Replica Sets
Read / Write
Secondary
Read
Primary
Read
Secondary
30. Replica Sets
Read / Write
Read
Secondary
Secondary
Read
Primary
Read
Secondary
Secondary
Read
31. Sharding
32. Secondary
Secondary
Secondary
Secondary
MongoS
MongoS
Shard 1
0..10
Shard 2
10..20
Shard 3
20..30
Shard 4
30..40
Primary
Primary
Primary
Primary
Secondary
Secondary
Secondary
Secondary
33. 400GB Index?
34. 400GB Index?
Shard 1
0..10
Shard 2
10..20
Shard 3
20..30
Shard 4
30..40
100GB
Index!
100GB
Index!
100GB
Index!
100GB
Index!
35. Summary
36. Summary
Use documents to your advantage!
Optimize your indexes
Understand your working set
Use a sane disk configuratino
Use replicas to scale reads
Use sharding to scale writes & working RAM