NoSQL Infrastructure - Late 2013
-
Upload
server-density -
Category
Technology
-
view
788 -
download
1
description
Transcript of NoSQL Infrastructure - Late 2013
![Page 1: NoSQL Infrastructure - Late 2013](https://reader033.fdocuments.us/reader033/viewer/2022061210/548c9651b47959eb548b48ab/html5/thumbnails/1.jpg)
NoSQL Infrastructure
![Page 2: NoSQL Infrastructure - Late 2013](https://reader033.fdocuments.us/reader033/viewer/2022061210/548c9651b47959eb548b48ab/html5/thumbnails/2.jpg)
~3333 write ops/s 0.07 - 0.05 ms response
![Page 3: NoSQL Infrastructure - Late 2013](https://reader033.fdocuments.us/reader033/viewer/2022061210/548c9651b47959eb548b48ab/html5/thumbnails/3.jpg)
David Mytton
Woop Japan!
![Page 4: NoSQL Infrastructure - Late 2013](https://reader033.fdocuments.us/reader033/viewer/2022061210/548c9651b47959eb548b48ab/html5/thumbnails/4.jpg)
![Page 5: NoSQL Infrastructure - Late 2013](https://reader033.fdocuments.us/reader033/viewer/2022061210/548c9651b47959eb548b48ab/html5/thumbnails/5.jpg)
MongoDB at Server Density
![Page 6: NoSQL Infrastructure - Late 2013](https://reader033.fdocuments.us/reader033/viewer/2022061210/548c9651b47959eb548b48ab/html5/thumbnails/6.jpg)
•27 nodes
MongoDB at Server Density
![Page 7: NoSQL Infrastructure - Late 2013](https://reader033.fdocuments.us/reader033/viewer/2022061210/548c9651b47959eb548b48ab/html5/thumbnails/7.jpg)
• June 2009 - 4yrs
•27 nodes
MongoDB at Server Density
![Page 8: NoSQL Infrastructure - Late 2013](https://reader033.fdocuments.us/reader033/viewer/2022061210/548c9651b47959eb548b48ab/html5/thumbnails/8.jpg)
•MySQL -> MongoDB
•27 nodes
MongoDB at Server Density
• June 2009 - 4yrs
![Page 9: NoSQL Infrastructure - Late 2013](https://reader033.fdocuments.us/reader033/viewer/2022061210/548c9651b47959eb548b48ab/html5/thumbnails/9.jpg)
•MySQL -> MongoDB
•25TB data per month
•27 nodes
MongoDB at Server Density
• June 2009 - 4yrs
![Page 10: NoSQL Infrastructure - Late 2013](https://reader033.fdocuments.us/reader033/viewer/2022061210/548c9651b47959eb548b48ab/html5/thumbnails/10.jpg)
Why?
![Page 11: NoSQL Infrastructure - Late 2013](https://reader033.fdocuments.us/reader033/viewer/2022061210/548c9651b47959eb548b48ab/html5/thumbnails/11.jpg)
Why?
• Replication
![Page 12: NoSQL Infrastructure - Late 2013](https://reader033.fdocuments.us/reader033/viewer/2022061210/548c9651b47959eb548b48ab/html5/thumbnails/12.jpg)
Why?
• Replication
• Official drivers
![Page 13: NoSQL Infrastructure - Late 2013](https://reader033.fdocuments.us/reader033/viewer/2022061210/548c9651b47959eb548b48ab/html5/thumbnails/13.jpg)
Why?
• Replication
• Official drivers
• Easy deployment
![Page 14: NoSQL Infrastructure - Late 2013](https://reader033.fdocuments.us/reader033/viewer/2022061210/548c9651b47959eb548b48ab/html5/thumbnails/14.jpg)
Why?
• Replication
• Official drivers
• Easy deployment
• Fast out of the box
![Page 15: NoSQL Infrastructure - Late 2013](https://reader033.fdocuments.us/reader033/viewer/2022061210/548c9651b47959eb548b48ab/html5/thumbnails/15.jpg)
It’s a little different.
![Page 16: NoSQL Infrastructure - Late 2013](https://reader033.fdocuments.us/reader033/viewer/2022061210/548c9651b47959eb548b48ab/html5/thumbnails/16.jpg)
Picture is unrelated! Mmm, ice cream.
• Fast network
Performance
![Page 17: NoSQL Infrastructure - Late 2013](https://reader033.fdocuments.us/reader033/viewer/2022061210/548c9651b47959eb548b48ab/html5/thumbnails/17.jpg)
• Fast network
Performance
EC2 10 Gigabit Ethernet- Cluster Compute- High Memory Cluster- Cluster GPU- High I/O- High Storage
![Page 18: NoSQL Infrastructure - Late 2013](https://reader033.fdocuments.us/reader033/viewer/2022061210/548c9651b47959eb548b48ab/html5/thumbnails/18.jpg)
• Fast network
Performance
Workload: Read/Write?
Result set size
What is being stored?
![Page 19: NoSQL Infrastructure - Late 2013](https://reader033.fdocuments.us/reader033/viewer/2022061210/548c9651b47959eb548b48ab/html5/thumbnails/19.jpg)
• Fast network
Performance
Use Network Throughput
Normal 0-100Mb/s
Replication (Initial Sync) Burst +100Mb/s
Replication (Oplog) 0-100Mb/s
Backup Initial Sync + Oplog
![Page 20: NoSQL Infrastructure - Late 2013](https://reader033.fdocuments.us/reader033/viewer/2022061210/548c9651b47959eb548b48ab/html5/thumbnails/20.jpg)
• Fast network
Performance
Inter-DC LAN
![Page 21: NoSQL Infrastructure - Late 2013](https://reader033.fdocuments.us/reader033/viewer/2022061210/548c9651b47959eb548b48ab/html5/thumbnails/21.jpg)
• Fast network
Performance
Inter-DC LAN
Cross USA Washington, DC - San Jose, CA
![Page 22: NoSQL Infrastructure - Late 2013](https://reader033.fdocuments.us/reader033/viewer/2022061210/548c9651b47959eb548b48ab/html5/thumbnails/22.jpg)
• Fast network
Performance
Location Ping RTT Latency
Within USA 40-80ms
Trans-Atlantic 100ms
Trans-Pacific 150ms
Europe - Japan 300ms
![Page 23: NoSQL Infrastructure - Late 2013](https://reader033.fdocuments.us/reader033/viewer/2022061210/548c9651b47959eb548b48ab/html5/thumbnails/23.jpg)
Failover
•Replica sets
![Page 24: NoSQL Infrastructure - Late 2013](https://reader033.fdocuments.us/reader033/viewer/2022061210/548c9651b47959eb548b48ab/html5/thumbnails/24.jpg)
Failover
•Replica sets
•Master/slave
![Page 25: NoSQL Infrastructure - Late 2013](https://reader033.fdocuments.us/reader033/viewer/2022061210/548c9651b47959eb548b48ab/html5/thumbnails/25.jpg)
Failover
•Replica sets
•Min 3 nodes
•Master/slave
![Page 26: NoSQL Infrastructure - Late 2013](https://reader033.fdocuments.us/reader033/viewer/2022061210/548c9651b47959eb548b48ab/html5/thumbnails/26.jpg)
Failover
•Replica sets
•Min 3 nodes
•Master/slave
•Automatic failover
![Page 27: NoSQL Infrastructure - Late 2013](https://reader033.fdocuments.us/reader033/viewer/2022061210/548c9651b47959eb548b48ab/html5/thumbnails/27.jpg)
•Replication lag
Performance
Location Ping RTT Latency
Within USA 40-80ms
Trans-Atlantic 100ms
Trans-Pacific 150ms
Europe - Japan 300ms
![Page 28: NoSQL Infrastructure - Late 2013](https://reader033.fdocuments.us/reader033/viewer/2022061210/548c9651b47959eb548b48ab/html5/thumbnails/28.jpg)
Replication Lag
1. Reads: eventual consistency
![Page 29: NoSQL Infrastructure - Late 2013](https://reader033.fdocuments.us/reader033/viewer/2022061210/548c9651b47959eb548b48ab/html5/thumbnails/29.jpg)
Replication Lag
1. Reads: eventual consistency
2. Failover: slave behind
![Page 30: NoSQL Infrastructure - Late 2013](https://reader033.fdocuments.us/reader033/viewer/2022061210/548c9651b47959eb548b48ab/html5/thumbnails/30.jpg)
Eventual Consistency
Stale data
![Page 31: NoSQL Infrastructure - Late 2013](https://reader033.fdocuments.us/reader033/viewer/2022061210/548c9651b47959eb548b48ab/html5/thumbnails/31.jpg)
Eventual Consistency
Stale data
Inconsistent data
![Page 32: NoSQL Infrastructure - Late 2013](https://reader033.fdocuments.us/reader033/viewer/2022061210/548c9651b47959eb548b48ab/html5/thumbnails/32.jpg)
Eventual Consistency
Stale data
Inconsistent data
Changing data
![Page 33: NoSQL Infrastructure - Late 2013](https://reader033.fdocuments.us/reader033/viewer/2022061210/548c9651b47959eb548b48ab/html5/thumbnails/33.jpg)
Eventual Consistency
Use Case Needs consistency?
Graphs No
User profile Yes
Statistics Depends
Alert config Yes
![Page 34: NoSQL Infrastructure - Late 2013](https://reader033.fdocuments.us/reader033/viewer/2022061210/548c9651b47959eb548b48ab/html5/thumbnails/34.jpg)
Replication Lag
1. Reads: eventual consistency
2. Failover: slave behind
![Page 35: NoSQL Infrastructure - Late 2013](https://reader033.fdocuments.us/reader033/viewer/2022061210/548c9651b47959eb548b48ab/html5/thumbnails/35.jpg)
Slave behind
Failover: out of date master
![Page 36: NoSQL Infrastructure - Late 2013](https://reader033.fdocuments.us/reader033/viewer/2022061210/548c9651b47959eb548b48ab/html5/thumbnails/36.jpg)
• Safe by default
>>> from pymongo import MongoClient>>> connection = MongoClient(w=int/str)
Value Meaning
0 Unsafe
1 Primary
2 Primary + x1 secondary
3 Primary + x2 secondaries
MongoDB WriteConcern
![Page 37: NoSQL Infrastructure - Late 2013](https://reader033.fdocuments.us/reader033/viewer/2022061210/548c9651b47959eb548b48ab/html5/thumbnails/37.jpg)
![Page 38: NoSQL Infrastructure - Late 2013](https://reader033.fdocuments.us/reader033/viewer/2022061210/548c9651b47959eb548b48ab/html5/thumbnails/38.jpg)
Tags
{ _id : "someSet", members : [ {_id : 0, host : "A", tags : {"dc": "ny"}}, {_id : 1, host : "B", tags : {"dc": "ny"}}, {_id : 2, host : "C", tags : {"dc": "sf"}}, {_id : 3, host : "D", tags : {"dc": "sf"}}, {_id : 4, host : "E", tags : {"dc": "cloud"}} ] settings : { getLastErrorModes : { veryImportant : {"dc" : 3}, sortOfImportant : {"dc" : 2} } }}> db.foo.insert({x:1})> db.runCommand({getLastError : 1, w : "veryImportant"})
![Page 39: NoSQL Infrastructure - Late 2013](https://reader033.fdocuments.us/reader033/viewer/2022061210/548c9651b47959eb548b48ab/html5/thumbnails/39.jpg)
{ _id : "someSet", members : [ {_id : 0, host : "A", tags : {"dc": "ny"}}, {_id : 1, host : "B", tags : {"dc": "ny"}}, {_id : 2, host : "C", tags : {"dc": "sf"}}, {_id : 3, host : "D", tags : {"dc": "sf"}}, {_id : 4, host : "E", tags : {"dc": "cloud"}} ] settings : { getLastErrorModes : { veryImportant : {"dc" : 3}, sortOfImportant : {"dc" : 2} } }}> db.foo.insert({x:1})> db.runCommand({getLastError : 1, w : "veryImportant"})
(A or B) + (C or D) + E
Tags
![Page 40: NoSQL Infrastructure - Late 2013](https://reader033.fdocuments.us/reader033/viewer/2022061210/548c9651b47959eb548b48ab/html5/thumbnails/40.jpg)
{ _id : "someSet", members : [ {_id : 0, host : "A", tags : {"dc": "ny"}}, {_id : 1, host : "B", tags : {"dc": "ny"}}, {_id : 2, host : "C", tags : {"dc": "sf"}}, {_id : 3, host : "D", tags : {"dc": "sf"}}, {_id : 4, host : "E", tags : {"dc": "cloud"}} ] settings : { getLastErrorModes : { veryImportant : {"dc" : 3}, sortOfImportant : {"dc" : 2} } }}> db.foo.insert({x:1})> db.runCommand({getLastError : 1, w : "sortOfImportant"})
(A + C) or (D + E) ...
Tags
![Page 41: NoSQL Infrastructure - Late 2013](https://reader033.fdocuments.us/reader033/viewer/2022061210/548c9651b47959eb548b48ab/html5/thumbnails/41.jpg)
Picture is unrelated! Mmm, ice cream.
• Fast network
•More RAM
Performance
![Page 42: NoSQL Infrastructure - Late 2013](https://reader033.fdocuments.us/reader033/viewer/2022061210/548c9651b47959eb548b48ab/html5/thumbnails/42.jpg)
http://www.slideshare.net/jrosoff/mongodb-on-ec2-and-ebs
![Page 43: NoSQL Infrastructure - Late 2013](https://reader033.fdocuments.us/reader033/viewer/2022061210/548c9651b47959eb548b48ab/html5/thumbnails/43.jpg)
http://blog.pythonisito.com/2011/12/mongodbs-write-lock.html
![Page 44: NoSQL Infrastructure - Late 2013](https://reader033.fdocuments.us/reader033/viewer/2022061210/548c9651b47959eb548b48ab/html5/thumbnails/44.jpg)
http://blog.pythonisito.com/2011/12/mongodbs-write-lock.html
![Page 45: NoSQL Infrastructure - Late 2013](https://reader033.fdocuments.us/reader033/viewer/2022061210/548c9651b47959eb548b48ab/html5/thumbnails/45.jpg)
More RAM = expensive Performance
x2 4GB RAM 12 month Prices
![Page 46: NoSQL Infrastructure - Late 2013](https://reader033.fdocuments.us/reader033/viewer/2022061210/548c9651b47959eb548b48ab/html5/thumbnails/46.jpg)
RAM
SSDs
Spinning disk
Cost Speed
![Page 47: NoSQL Infrastructure - Late 2013](https://reader033.fdocuments.us/reader033/viewer/2022061210/548c9651b47959eb548b48ab/html5/thumbnails/47.jpg)
Softlayer disk pricing Performance
![Page 48: NoSQL Infrastructure - Late 2013](https://reader033.fdocuments.us/reader033/viewer/2022061210/548c9651b47959eb548b48ab/html5/thumbnails/48.jpg)
EC2 disk/RAM pricing Performance
$2232/m
$2520/m
$43/m
$295/m
![Page 49: NoSQL Infrastructure - Late 2013](https://reader033.fdocuments.us/reader033/viewer/2022061210/548c9651b47959eb548b48ab/html5/thumbnails/49.jpg)
SSD vs Spinning Performance
![Page 50: NoSQL Infrastructure - Late 2013](https://reader033.fdocuments.us/reader033/viewer/2022061210/548c9651b47959eb548b48ab/html5/thumbnails/50.jpg)
SSD vs Spinning Performance
![Page 51: NoSQL Infrastructure - Late 2013](https://reader033.fdocuments.us/reader033/viewer/2022061210/548c9651b47959eb548b48ab/html5/thumbnails/51.jpg)
SSD vs Spinning Performance
![Page 52: NoSQL Infrastructure - Late 2013](https://reader033.fdocuments.us/reader033/viewer/2022061210/548c9651b47959eb548b48ab/html5/thumbnails/52.jpg)
Tips: rand()
•Field names
![Page 53: NoSQL Infrastructure - Late 2013](https://reader033.fdocuments.us/reader033/viewer/2022061210/548c9651b47959eb548b48ab/html5/thumbnails/53.jpg)
Tips: rand()
•Field names
•Covered indexes
![Page 54: NoSQL Infrastructure - Late 2013](https://reader033.fdocuments.us/reader033/viewer/2022061210/548c9651b47959eb548b48ab/html5/thumbnails/54.jpg)
Tips: rand()
•Field names
•Covered indexes
•Collections / databases
![Page 55: NoSQL Infrastructure - Late 2013](https://reader033.fdocuments.us/reader033/viewer/2022061210/548c9651b47959eb548b48ab/html5/thumbnails/55.jpg)
Backups
What is the use case?
![Page 56: NoSQL Infrastructure - Late 2013](https://reader033.fdocuments.us/reader033/viewer/2022061210/548c9651b47959eb548b48ab/html5/thumbnails/56.jpg)
Backups
What is the use case?
Fixing user errors?
Point in time restore?
Disaster recovery?
![Page 57: NoSQL Infrastructure - Late 2013](https://reader033.fdocuments.us/reader033/viewer/2022061210/548c9651b47959eb548b48ab/html5/thumbnails/57.jpg)
Backups
•Disaster recovery
Offsite
![Page 58: NoSQL Infrastructure - Late 2013](https://reader033.fdocuments.us/reader033/viewer/2022061210/548c9651b47959eb548b48ab/html5/thumbnails/58.jpg)
Backups
•Disaster recovery
Age
Offsite
![Page 59: NoSQL Infrastructure - Late 2013](https://reader033.fdocuments.us/reader033/viewer/2022061210/548c9651b47959eb548b48ab/html5/thumbnails/59.jpg)
Backups
•Disaster recovery
Age
Offsite
Restore time
![Page 60: NoSQL Infrastructure - Late 2013](https://reader033.fdocuments.us/reader033/viewer/2022061210/548c9651b47959eb548b48ab/html5/thumbnails/60.jpg)
Backups
Frequency
Consistency
Verification
![Page 61: NoSQL Infrastructure - Late 2013](https://reader033.fdocuments.us/reader033/viewer/2022061210/548c9651b47959eb548b48ab/html5/thumbnails/61.jpg)
www.flickr.com/photos/daddo83/3406962115/
Monitoring
•System
Disk i/o
Disk use
![Page 62: NoSQL Infrastructure - Late 2013](https://reader033.fdocuments.us/reader033/viewer/2022061210/548c9651b47959eb548b48ab/html5/thumbnails/62.jpg)
www.flickr.com/photos/daddo83/3406962115/
Monitoring
Disk i/o
Disk use
•System
Swap
![Page 63: NoSQL Infrastructure - Late 2013](https://reader033.fdocuments.us/reader033/viewer/2022061210/548c9651b47959eb548b48ab/html5/thumbnails/63.jpg)
www.flickr.com/photos/daddo83/3406962115/
Monitoring
Optime
State
•Replication
![Page 64: NoSQL Infrastructure - Late 2013](https://reader033.fdocuments.us/reader033/viewer/2022061210/548c9651b47959eb548b48ab/html5/thumbnails/64.jpg)
mongostat
![Page 65: NoSQL Infrastructure - Late 2013](https://reader033.fdocuments.us/reader033/viewer/2022061210/548c9651b47959eb548b48ab/html5/thumbnails/65.jpg)
Monitoring tools
Run yourself
Ganglia
![Page 66: NoSQL Infrastructure - Late 2013](https://reader033.fdocuments.us/reader033/viewer/2022061210/548c9651b47959eb548b48ab/html5/thumbnails/66.jpg)
Monitoring tools
www.serverdensity.com