MongoAsia - Scaling

8/7/2019 MongoAsia - Scaling

1/44

Scaling with MongoDBAlvin Richards

[email protected]


2/44

Agenda

Vertical Scaling Horizontal Scaling with MongoDB

Schema & Index design Auto Sharding Replication


3/44

Scaling

Operations/sec go up Storage needs go up

Capacity IOPs

Complexity goes up Caching


4/44

Optimization & Tuning Schema & Index Design O/S tuning Hardware conguration

Vertical scaling Hardware is expensive Hard to scale in cloud

How do you scale now?

$$$

throughput


5/44

MongoDB Scaling - Single Node

write

read

node_a1


6/44

Read scaling - add Replicas

write

read

node_b1

node_a1


7/44

Read scaling - add Replicas

write

read

node_c1

node_b1

node_a1


8/44

Write scaling - Sharding

write

read

shard1

node_c1

node_b1

node_a1


9/44

Write scaling - add Shards

write

read

shard1

node_c1

node_b1

node_a1

shard2

node_c2

node_b2

node_a2


10/44

Write scaling - add Shards

write

read

shard1

node_c1

node_b1

node_a1

shard2

node_c2

node_b2

node_a2

shard3

node_c3

node_b3

node_a3


11/44

Scaling with MongoDB

Schema & Index Design Sharding Replication


12/44

Schema

Data model e f ects performance Embedding versus Linking

Roundtrips to database Disk seek time Size if data to read & write

Partial versus full document writes

Performance problems can be solved by changingschema

not schema less - dynamic schemaschema is just as important, or more important than relationalunderstand write vs read tradeo f s


13/44

Indexes

Index common queries Do not over index

(A) and (A,B) are equivalent, choose one Right-balanced indexes keep working set small

most common performance problemwhy _id index can be ignored


14/44

Random Index Access

Entire indexin ram


15/44

Right-Balanced Index Access

Small portionin ram


16/44

What is Sharding

Ad-hoc partitioning

Consistent hashing Amazon Dynamo

Range based partitioning Google BigTable Yahoo! PNUTS MongoDB

sharding isnt new


17/44

MongoDB Sharding

Automatic partitioning and management

Range based

Convert to sharded system with no downtime

Almost no functionality lost over single master

Fully consistent


18/44

How MongoDB Sharding works

> db.runCommand( { addshard : "shard1" } );

- +

Range keys from - to +

Ranges are stored as chunks


19/44


> db.posts.save( {age:40} )

- +

- 40 41 +

Data in inserted

Ranges are split into more chunks


20/44


> db.posts.save( {age:40} )> db.posts.save( {age:50} )

- +

- 40 41 +

41 50 51 +

More Data in inserted

Ranges are split into morechunks


21/44


> db.posts.save( {age:40} )> db.posts.save( {age:50} )> db.posts.save( {age:60} )

- +

- 40 41 +

41 50 51 +

61 + 51 60


22/44

- +

41 +

51 +


> db.posts.save( {age:40} )> db.posts.save( {age:50} )> db.posts.save( {age:60} )

- 40

41 50

61 + 51 60


23/44


- 40

41 50

61 +

51 60

shard1


24/44



- 40

41 50

61 +

51 60


25/44



- 40

41 50

61 +

51 60

shard1


26/44



- 40

41 50

61 +

51 60

shard1 shard2


27/44



- 40

41 50

61 +

51 60

shard1 shard2


shard3


28/44

Sharding Key Examples

Good : {server:1} All data for one server is in a single chunk Chunk cannot be split any smaller

Better : {server:1,time:1} Chunk can be split by millisecond

{server : "ny153.example.com" ,application : "apache" ,time : "2011-01-02T21:21:56.249Z" ,level : "ERROR" ,msg : "something is broken"

}


29/44

Sharding Key Examples

Good : {time : 1} Time is an increasing number All data will be rst written to a single shard Data balanced to other shards later

Better : {server:1,application:1,time:1}

More key values to enable writes to all shards

{server : "ny153.example.com" ,application : "apache" ,time : "2011-01-02T21:21:56.249Z" ,level : "ERROR" ,msg : "something is broken"

}


30/44

Sharding Features

Shard data without no downtime Automatic balancing as data is written Commands routed (switched) to correct node

Inserts - must have the Shard Key Updates - must have the Shard Key Queries

With Shard Key - routed to nodes Without Shard Key - scatter gather

Indexed Queries With Shard Key - routed in order

Without Shard Key - distributed sort merge


31/44

MongoDB Replication

MongoDB replication like MySQL replicationAsynchronous master/slave

Variations:Master / slaveReplica Sets


32/44

A cluster of N servers Any (one) node can be primary Consensus election of primary Automatic failover Automatic recovery All writes to primary Reads can be to primary (default) or a secondary

Replica Set features


33/44

How MongoDB Replication works

Member 1

Member 2

Member 3

Set is made up of 2 or more nodes


34/44


Member 1

Member 2PRIMARY

Member 3

Election establishes the PRIMARY

Data replication from PRIMARY to

SECONDARY


35/44


Member 1

Member 2DOWN

Member 3

negotiatenew master

PRIMARY may fail

Automatic election of new PRIMARY


36/44


Member 1

Member 2DOWN

Member 3PRIMARY

New PRIMARY elected

Replication Set re-established


37/44


Member 1

Member 2RECOVERING

Member 3PRIMARY

Automatic recovery


38/44


Member 1

Member 2

Member 3PRIMARY

Replication Set re-established


39/44

Using Replicas

slaveOk()

- driver will send read requests to Secondaries- driver will always send writes to Primary

Java examples- DB.slaveOk()- Collection.slaveOk()- find(q).addOption(Bytes.QUERYOPTION_SLAVEOK);


40/44

Creating a Replica Set

> cfg = {... _id : "acme_a",... members : [... { _id : 0, host : "sf1.acme.com" },... { _id : 1, host : "sf2.acme.com" },... { _id : 2, host : "sf3.acme.com" } ] }> use admin> db.runCommand({replSetInitiate:cfg})


41/44

Replica Set Member Types

Normal {priority:1}

Passive {priority:0} Cannot be elected as PRIMARY

Arbiters Can vote in an election Do not hold any data


42/44

Replication features

Reads from Primary are always consistent

Reads from Secondaries are eventually consistent

Automatic failover if a Primary fails

Automatic recovery when a node joins the set


43/44

Summary

Schema & Index design Simplest way to scale

Sharding Automatically scale writes

Replication Automatically scale reads


44/44

@mongodb

conferences, appearances, and meetupshttp://www.10gen.com/events

http://bit.ly/mongoJFacebook | Twitter | LinkedIn

http://linkd.in/joinmongo

download at mongodb.org

MongoAsia - Scaling

Documents

Transcript of MongoAsia - Scaling