2
• Customer Stories
• Sharding for Performance/Scale– When to shard?– How many shards do I need?
• Types of Sharding
• How to Pick a Shard Key
• Sharding for Other Reasons
Agenda
Customer Stories
4
5
• 50M users.
• 6B check-ins to date (6M per day growth).
• 55M points of interest / venues.
• 1.7M merchants using the platform for marketing
• Operations Per Second: 300,000
• Documents: 5.5B
Foursquare
6
• 11 MongoDB clusters– 8 are sharded
• Largest cluster has 15 shards (check ins)– Sharded on user id
Foursquare clusters
7
• Large data set
CarFax
8
• 13 billion+ documents– 1.5 billion documents added every year
• 1 vehicle history report is > 200 documents
• 12 Shards
• 9-node replica sets
• Replicas distributed across 3 data centers
CarFax Shards
9
What is Sharding?
12
Sharding Overview
Primary
Secondary
Secondary
Shard 1
Primary
Secondary
Secondary
Shard 2
Primary
Secondary
Secondary
Shard 3
Primary
Secondary
Secondary
Shard N
…
Query Router
Query Router
Query Router
……
Driver
Application
14
Scaling: Sharding
mongod
Read/Write Scalability
Key Range0..100
15
Scaling: Sharding
Read/Write Scalability
mongod mongod
Key Range0..50
Key Range51..100
16
Scaling: Sharding
mongod mongod mongod mongod
Key Range0..25
Key Range26..50
Key Range51..75
Key Range76.. 100
Read/Write Scalability
How do I know I need to shard?
18
Does one server/replica…
• Have enough disk space to store all my data?
• Handle my query throughput (operations per second)?
• Respond to queries fast enough (latency)?
19
• Have enough disk space to store all my data?
• Handle my query throughput (operations per second)?
• Respond to queries fast enough (latency)?
Does one server/replica set…
Server Specs
Disk Capacity
Disk IOPSRAMNetwork
Disk IOPSRAMNetwork
How many shards do I need?
21
• Sum of disk space across shards > greater than required storage size
Disk Space: How Many Shards Do I Need?
22
• Sum of disk space across shards > greater than required storage size
Disk Space: How Many Shards Do I Need?
Example
Storage size = 3 TBServer disk capacity = 2 TB
2 Shards Required
23
• Working set should fit in RAM– Sum of RAM across shards > Working Set
• WorkSet = Indexes plus the set of documents accessed frequently
• WorkSet in RAM – Shorter latency– Higher Throughput
RAM: How Many Shards Do I Need?
24
• Measuring Index Size and Working Setdb.stats() – index size of each collection
db.serverStatus({ workingSet: 1}) – working set size estimate
RAM: How Many Shards Do I Need?
25
• Measuring Index Size and Working Setdb.stats() – index size of each collection
db.serverStatus({ workingSet: 1}) – working set size estimate
RAM: How Many Shards Do I Need?
Example
Working Set = 428 GBServer RAM = 128 GB
428/128 = 3.34
4 Shards Required
26
• Sum of IOPS across shards > greater than required IOPS
• IOPS are difficult to estimate– Update doc– Update indexes– Append to journal– Log entry?
• Best approach – build a prototype and measure
Disk Throughput: How Many Shards Do I Need
27
• Sum of IOPS across shards > greater than required IOPS
• IOPS are difficult to estimate– Update doc– Update indexes– Append to journal– Log entry?
• Best approach – build a prototype and measure
Disk Throughput: How Many Shards Do I Need
Example
Required IOPS = 11000Server disk IOPS = 5000
3 Shards Required
Types of Sharding
32
• Range
• Tag-Aware
• Hashed
Sharding Types
33
Range Sharding
mongod mongod mongod mongod
Key Range0..25
Key Range26..50
Key Range51..75
Key Range76.. 100
Read/Write Scalability
34
Tag-Aware Sharding
mongod mongod mongod mongod
Shard Tags
Shard Tag Start End
Winter 23 Dec 21 Mar
Spring 22 Mar 21 Jun
Summer 21 Jun 23 Sep
Fall 24 Sep 22 Dec
Tag Ranges
Winter Spring Summer Fall
35
Hash-Sharding
mongod mongod mongod mongod
Hash Range0000..4444
Hash Range4445..8000
Hash Rangei8001..aaaa
Hash Rangeaaab..ffff
36
Hashed shard key
• Pros:– Evenly distributed writes
• Cons:– Random data (and index) updates can be IO
intensive– Range-based queries turn into scatter gather
Shard 1
mongos
Shard 2 Shard 3 Shard N
37
Range sharding document distribution
38
Hashed sharding document distribution
How do I Pick A Shard Key
40
Shard Key characteristics
• A good shard key has:– sufficient cardinality– distributed writes– targeted reads ("query isolation")
• Shard key should be in every query if possible– scatter gather otherwise
• Choosing a good shard key is important!– affects performance and scalability– changing it later is expensive
41
Low cardinality shard key
• Induces "jumbo chunks"
• Examples: boolean field
Shard 1
mongos
Shard 2 Shard 3 Shard N
[ a, b )
42
Ascending shard key
• Monotonically increasing shard key values cause "hot spots" on inserts
• Examples: timestamps, _id
Shard 1
mongos
Shard 2 Shard 3 Shard N
[ ISODate(…), $maxKey )
Reasons to Shard
44
• Scale– Data volume– Query volume
• Global deployment with local writes– Geography aware sharding
• Tiered Storage
• Fast backup restore
Reasons to shard
45
Global Deployment/Local Writes
Primary:NYC
Secondary:NYC
Primary:LON
Primary:SYD
Secondary:LON
Secondary:NYC
Secondary:SYD
Secondary:LON
Secondary:SYD
46
• Save hardware costs
• Put frequently accessed documents on fast servers– Infrequently accessed documents on less capable
servers
• Use Tag aware sharding
Tiered Storage
mongod mongod mongod mongod
Current Current Archive Archive
SSD SSD HDD HDD
47
• 40 TB Database
• 2 shards of 20 TB each
• Challenge– Cannot meet restore SLA after data loss
Fast Restore
mongod mongod
20 TB 20 TB
48
• 40 TB Database
• 4 shards of 10 TB each
• Solution– Reduce the restore time by 50%
Fast Restore
mongod mongod
10 TB 10 TB
mongod mongod
10 TB 10 TB
Summary
50
• To determine required # of shards determine– Storage requirements– Latency requirements– Throughput requirements
• Derive total– Disk capacity– Disk throughput– RAM
• Calculate # of shards based upon individual server specs
Determining the # of shards
51
• Scalability
• Geo-aware clusters
• Tiered Storage
• Reduce backup restore times
Leverage Sharding For
52
• MongoDB Manual: http://docs.mongodb.org/manual/sharding/
• Other Webinars:– How to Achieve Scale With MongoDB
• White Papers– MongoDB Performance Best Practices– MongoDB Architecture Guide
Sharding: Where to go from here…
Thank You
Top Related