How to Keep Your Data Safe in MongoDB
description
Transcript of How to Keep Your Data Safe in MongoDB
![Page 1: How to Keep Your Data Safe in MongoDB](https://reader033.fdocuments.us/reader033/viewer/2022061209/548da382b479590d2b8b4abe/html5/thumbnails/1.jpg)
CTO, 10gen
Eliot Horowitz
#MongoDBDays
How to keep your data safe in MongoDB
![Page 2: How to Keep Your Data Safe in MongoDB](https://reader033.fdocuments.us/reader033/viewer/2022061209/548da382b479590d2b8b4abe/html5/thumbnails/2.jpg)
What can go wrong?
• Network breaks in transit
• Server crashes while processing
• Server blows up after processing a write before replication
• Server processes, crashes, and then a conflicting write happens elsewhere
• All copies burn in a fire
• 20 years later, no one remembers how to read it
![Page 3: How to Keep Your Data Safe in MongoDB](https://reader033.fdocuments.us/reader033/viewer/2022061209/548da382b479590d2b8b4abe/html5/thumbnails/3.jpg)
![Page 4: How to Keep Your Data Safe in MongoDB](https://reader033.fdocuments.us/reader033/viewer/2022061209/548da382b479590d2b8b4abe/html5/thumbnails/4.jpg)
What is data safety?
![Page 5: How to Keep Your Data Safe in MongoDB](https://reader033.fdocuments.us/reader033/viewer/2022061209/548da382b479590d2b8b4abe/html5/thumbnails/5.jpg)
Version 1
Probability that a given write or piece of data is accessible given human intervention and infinite time.
![Page 6: How to Keep Your Data Safe in MongoDB](https://reader033.fdocuments.us/reader033/viewer/2022061209/548da382b479590d2b8b4abe/html5/thumbnails/6.jpg)
Version 2
Probability that a given write or piece of data is visible in a query.
![Page 7: How to Keep Your Data Safe in MongoDB](https://reader033.fdocuments.us/reader033/viewer/2022061209/548da382b479590d2b8b4abe/html5/thumbnails/7.jpg)
Single Server – How a Write Works• Client sends a write operation to server
• Received by server’s tcp stack
• MongoDB process queues write
• Write happens in memory
• Depending on what Write Concern asks for
– Respond immediately– Wait for data to be journaled, then respond
![Page 8: How to Keep Your Data Safe in MongoDB](https://reader033.fdocuments.us/reader033/viewer/2022061209/548da382b479590d2b8b4abe/html5/thumbnails/8.jpg)
Single Server – What can go wrong• Network can go down once message hits other side
• Client doesn’t know what happens without going back and checking
• Write could fail for logical reason (unique key exception)
• Server could crash before journaled
• Write is lost journaled • Server could crash after journaled
• When server is recovered, write is replayed and safe• Hard drive can crash irrecoverably
• Data center could lose power for large period of time
![Page 9: How to Keep Your Data Safe in MongoDB](https://reader033.fdocuments.us/reader033/viewer/2022061209/548da382b479590d2b8b4abe/html5/thumbnails/9.jpg)
Any single server will failReplica Sets
![Page 10: How to Keep Your Data Safe in MongoDB](https://reader033.fdocuments.us/reader033/viewer/2022061209/548da382b479590d2b8b4abe/html5/thumbnails/10.jpg)
Replica Set - Reminders
• N nodes
• Each node has a fully copy of the data
• Replication is asynchronous
![Page 11: How to Keep Your Data Safe in MongoDB](https://reader033.fdocuments.us/reader033/viewer/2022061209/548da382b479590d2b8b4abe/html5/thumbnails/11.jpg)
Replica Set - Acknowledgements• “w” : how many servers must apply write
before acknowledged
• w=2 : do not acknowledge until write is on two servers– If primary fails, election guarantees new primary has all
writes acknowledge w=2
• w=majority : do not acknowledge until writes is on a majority of nodes in a replica set– If any primary is elected automatically, all writes
acknowledged with w=majority will be on primary.
![Page 12: How to Keep Your Data Safe in MongoDB](https://reader033.fdocuments.us/reader033/viewer/2022061209/548da382b479590d2b8b4abe/html5/thumbnails/12.jpg)
Good, but not enough…What if I lose an entire data center?
![Page 13: How to Keep Your Data Safe in MongoDB](https://reader033.fdocuments.us/reader033/viewer/2022061209/548da382b479590d2b8b4abe/html5/thumbnails/13.jpg)
Replica Set - tags
• A node can have a set of tags– region=us-east– color=blue
• Operator configures write level– Critical– has to be in 3 regions– Important – has to be in 2 regions
• w=critical– Do not acknowledge write until its in 3 data
centers– Losing an entire data center causes no data loss
![Page 14: How to Keep Your Data Safe in MongoDB](https://reader033.fdocuments.us/reader033/viewer/2022061209/548da382b479590d2b8b4abe/html5/thumbnails/14.jpg)
What about sharding?
• Same rules apply
• Given a series of writes, they may go to different shards– A w=majority at the end means all writes on that
socket are acknowledge by a majority of the relevant replica set
• Config servers have no impact on fault tolerance/durability, only on admin uptime (or real uptime in a disaster)
![Page 15: How to Keep Your Data Safe in MongoDB](https://reader033.fdocuments.us/reader033/viewer/2022061209/548da382b479590d2b8b4abe/html5/thumbnails/15.jpg)
Examples
![Page 16: How to Keep Your Data Safe in MongoDB](https://reader033.fdocuments.us/reader033/viewer/2022061209/548da382b479590d2b8b4abe/html5/thumbnails/16.jpg)
Personal Blog
• Single server
• No replication
• Hourly backups
• If server crashes– Down until back up– All acknowledge writes safe
• If server is destroyed– Have to recover from backup– Lose up to 1 hour of writes
![Page 17: How to Keep Your Data Safe in MongoDB](https://reader033.fdocuments.us/reader033/viewer/2022061209/548da382b479590d2b8b4abe/html5/thumbnails/17.jpg)
Departmental App
• Single replica set
• 3 nodes in a single server
• If any single node goes down– System is still readable/writeable– Writes done with w=2 are safe
• If 2 nodes go down at the same time– Only writes with w=3 are safe (bad idea)– No primary, last node is read-only
![Page 18: How to Keep Your Data Safe in MongoDB](https://reader033.fdocuments.us/reader033/viewer/2022061209/548da382b479590d2b8b4abe/html5/thumbnails/18.jpg)
Core User Database
• Single replica set
• 3 data centers– Primary data center: 3 node (p=2)– 2 alternates with 2 nodes each (p=1)
• Different types of operations– Password change (w=majority)– Adds a “like” (w=2)– Login count (w=1)
![Page 19: How to Keep Your Data Safe in MongoDB](https://reader033.fdocuments.us/reader033/viewer/2022061209/548da382b479590d2b8b4abe/html5/thumbnails/19.jpg)
Core User Database – cont’d• Lose any single server
– Can only lose a login count
• Lose any 2 servers– Could lose a “like” if you are unlucky
• Lose a data center– Still have a majority– All password changes are safe
![Page 20: How to Keep Your Data Safe in MongoDB](https://reader033.fdocuments.us/reader033/viewer/2022061209/548da382b479590d2b8b4abe/html5/thumbnails/20.jpg)
Choice is a double edged sword
![Page 21: How to Keep Your Data Safe in MongoDB](https://reader033.fdocuments.us/reader033/viewer/2022061209/548da382b479590d2b8b4abe/html5/thumbnails/21.jpg)
When to give a choice?
• Give choice over semantics– Developers and Operators know their needs
• Tuning parameters are dangerous– System should be smart enough to avoid
thousands of knobs
• Defaults should be– Intuitive and sensible– Changing is hard– Always changing a little
![Page 22: How to Keep Your Data Safe in MongoDB](https://reader033.fdocuments.us/reader033/viewer/2022061209/548da382b479590d2b8b4abe/html5/thumbnails/22.jpg)
Semantics Knobs – too many?
![Page 23: How to Keep Your Data Safe in MongoDB](https://reader033.fdocuments.us/reader033/viewer/2022061209/548da382b479590d2b8b4abe/html5/thumbnails/23.jpg)
Already have them in different architecture components
• Caching
• Worker queues
• Asynchronous replication
• Synchronous replication
• Two-phase commit
![Page 24: How to Keep Your Data Safe in MongoDB](https://reader033.fdocuments.us/reader033/viewer/2022061209/548da382b479590d2b8b4abe/html5/thumbnails/24.jpg)
MongoDB gives you the choice of durability semantics from many systems in one.
• Control per write
• One source of truth in architecture
![Page 25: How to Keep Your Data Safe in MongoDB](https://reader033.fdocuments.us/reader033/viewer/2022061209/548da382b479590d2b8b4abe/html5/thumbnails/25.jpg)
What should you do?
• Pick a default write level for your app
• Only deviate with good reason
• Test disaster scenario so you know what’s going to happen
![Page 26: How to Keep Your Data Safe in MongoDB](https://reader033.fdocuments.us/reader033/viewer/2022061209/548da382b479590d2b8b4abe/html5/thumbnails/26.jpg)
CTO, 10gen
Eliot Horowitz
#MongoDBDays
Thank You