Post on 16-May-2015
description
Aaron Silverman
Mongo or Die!How MongoDB powers Doodle or Die
(@Zugwalt)
Doodle or Die
@DoodleOrDie
What is Doodle or Die?
Telephone
I'd like some beer!
I'd like some deer
I see no deer
I’ve no idea!
Phrase
Phrase
Phrase
Phrase
Doodle or Die
Shining Apple
Doodle
Phrase
Phrase Eat your fruit or DIE!
Doodle
What Powers Doodle or Die?
Started very, very small
4 Cores128MB RAM
Node Server
MongoDB Server
Small - 2GB
Got serious about our servers
MongoDB Server8 Cores256MB RAM
Node Server
Large - 5GB
Called in some Reinforcements
MongoDB Server12 Cores1GB RAM
Node Server
In the last 30 Days:
• 2,500,000 page views
• 100,000 uniques
• 35,000 active player accounts
• 2,000,000 new doodles and descriptions
“Small Data”Mongo DB4 GB data size<1 GB index size~10 queries/sec
Amazon170 GB data size8 GB in/month200 GB out/month
MongoHQ: $67• Amazon S3: $70
• Player Info• Chain Info (excluding
doodles)• Group Info• Game State• Logs
• Doodles• Static Content• Compressed
Database Backups
MongoDB - $65 / month
Daily backups to Amazon S3: $16/mo
Amazon S3 - $70 / month
Node - $62/ month
Total Cost To Host: $197/monthMongoDB $65Amazon S3 $70Node $62
Total $197
PAAS Provides Easy Upgrade Path
General Principles
Custom _idPartially random string generated using ShortId node module
ObjectIdObjectId("4fd02d5d78315a502d15cdde")ObjectId("4fd02d5a78315a502d15cddd")ObjectId("4fd02d5878315a502d15cddc")
ShortId"8rOIwh2VD""1qyY61Lu1""5GQnbx-1"
• Shorter, less cumbersome in code and queries
db.players.findOne({_id: ‘58mwYlTKV’});
db.chains.update({_id: ‘58mwYlTKV’}, {$set: activePlayer_id: ‘88ueYaL6V’});
http://doodleordie.com/c/5ONtvvSGH
• Randomness could help with sharding; more importantly makes it harder to cheat
<span class="doodle" data-jsonp="http://doodles.s3.amazonaws.com/d2/Eh8-Po2R5/1Em5kj3LY.js">
Question Oriented Subdocuments
What chain is this player working on right now?
What are this player’s stats?
Which players are not eligible to be assigned this chain?
Goal is for most “Questions” to be able to be answered in one query from one sub document
Related “Questions” will share common ancestors
db.players.findOne({_id: ‘58mwYlTKV’}, {‘game.recentSkips’: 1});
db.players.findOne({_id: ‘58mwYlTKV’}, {game: 1});
Indexes are designed to make answering questions easy!
Which player is working on this chain?
db.players.ensureIndex({‘game.activeChain_id’: 1});
What chains are recently awaiting a new doodle?
db.chains.ensureIndex({inUse: -1, activeState: 1 lastModified" : -1});
Doodle or Die Collections
Disclaimer: Much of the original Doodle or Die code and schema were created during a weekend long hackathon!
players
chains
groups
sessions
log
Primary
Support
Players
players
account
info
login
chainHistory
game
stats QueryFrequency
Often
Rarely
players
account
info
login
chainHistory
stats
game • activeState• activeChain_id• activeStepIndex• recentSkips
Answers question: “What is this player working on right now?”db.players.findOne({_id: ‘58mwYlTKV’}, {game: 1});
players
account
info
login
stats
game
chainHistory
Chain1• datePlayed• dateViewed
ChainN
Answers the question: “What has the player worked on?”db.players.findOne({_id: ‘58mwYlTKV’}, {chainHistory: 1});
players
account
info
login
stats
game
chainHistory
Chain1• datePlayed• dateViewed
ChainN
YUCK!Plan to refactor out along with a refactor of how chains store steps
players
account
info
login
game
chainHistory
Answers the question: “How active/good is this player?”
stats
• totalSteps• drawSteps• phraseSteps• numSkips• numLikes
db.players.findOne({_id: ‘58mwYlTKV’}, {stats: 1});
Chains
chains
• activePlayer_id• activeState• numSteps• lastModified
ineligiblePlayer_idssteps
chains are assigned (not as many questions)db.chains.findOne({inUse: false, activeState : player.game.activeState, ineligiblePlayer_ids: {$ne: player._id}, lastModified: { $gte: timeRange}});
chains
• activePlayer_id• activeState• numSteps• lastModified
*
steps
ineligiblePlayer_ids
[player_id1 player_id2, player_id3 … player_idN]
Note: $addToSet and $pull work great in maintaining this array
chains
• activePlayer_id• activeState• numSteps• lastModified
*ineligiblePlayer_ids
steps
[
]
Step1• player_id• state• date
StepN
content
• phrase
Description Step Content:
Doodle Step Content:• url (points to S3)• time• numStrokes• numDistinctColors
• phrase
Description Step Content:
Doodle Step Content:• url (points to S3)• time• numStrokes• numDistinctColors
doodles
• player_id• date• url (points to S3)• time• numStrokes• numDistinctColors
descriptions
• player_id• date• text
We plan to stop embedding steps in chains, and link to “doodles” and “descriptions” collections
How Does it all Work?
players queried for users who are associated with the authenticated twitter account
chain history used to load thumbnails, previous chain is loaded
db.players.find({‘login.twitter.uid’: ‘XXXXXXX’}, {game: 1, chainHistory: 1});
db.chains.find({_id: {$in: [player.game.activeChain_id, player.game.lastChain_id]}});
chains collection searched and atomically updated for unclaimed eligible chain to be given to player
db.chains.findAndModify( {inUse: false, activeState : player.game.activeState, ineligiblePlayer_ids: {$ne: player._id}, lastModified: { $gte: timeRange}}, {$set: {inUse: true, activePlayer_id: player._id, $addToSet: {ineligiblePlayer_ids: player._id}});
Content saved to chain
db.chains.update({_id: chain._id, activePlayer_id: player._id}}, {$inc: {numSteps: 1}, $set: inUse: false, lastModified: datePlayed} $unset: {activePlayer_id: 1}, $push: {steps: { player_id: player._id, state: chain.activeState, content: content, date: datePlayed}}});
Doodle strokes will be saved to S3(but url to S3 and metadata saved to chain)
{"color":"#000000","size":15,"path":[307,66,308,66,308,68,308,69,308,70,308,71,308,72,308,73,306,76,305,79,305,81,302,83,302,84,302,85,302,86,301,87,300,89,300,90,300,91,300,92,300,95,300,97,300,102,300,103,300,104,300,108,300,109,300,110,301,114,303,116,304,116,305,119,305,121,307,124,309,127,310,130,310,131,313,132,314,133,316,137,317,138,320,140,321,140,323,143,326,143,328,143,333,144,337,144,341,144,343,144,347,143,352,143,354,141,357,140,358,140,359,139,362,138,363,138,365,137,368,135,369,132,370,131,371,130,374,128,375,128,376,125,378,125,379,124,380,123,381,123,382,120,385,119,386,118,388,115,391,112,394,109,394,107,394,106,397,104,397,103,397,102,397,101,397,100,397,98,397,96,397,94,397,93,397,91,397,90,397,88,397,87,397,86,397,83,395,80,395,79,395,77,394,74,393,72,393,70,393,67,392,65,391,63,389,62,388,59,386,57,383,54,383,52,381,49,379,48,378,47,376,46,375,44,374,43,372,43,370,42,369,42,368,42,364,41,362,41,359,41,355,41,351,42,349,42,347,42,346,44,343,44,342,45,339,46,337,47,336,48,333,49,332,51,330,51,328,51,327,52,326,53,323,53,322,53,321,54,320,55,317,55,316,56,314,58,309,59,306,61,306,62,305,62,304,63]}
Player’s chainHistory retrieved
db.chains.find({$in: chain_idArr});
db.players.find({‘login.urlSafe’: urlSafeUid}, {chainHistory: 1});
Retrieved chains ordered (on server) based on previously sorted chainHistory
chainHistory filtered and sorted on server, Applicable chains/steps retrieved
Stats are loaded from the counting log which is essentially a bunch of counters incremented
db.log.find({_id: {$in: [‘2012-06-20’, ‘2012-06-20’, ‘2012-06-20’,
‘2012-06-20’]}});
Extremely simple implementation using nested subdocuments for organization
Some Additional Doodle or Die Tricks
Build assumptions into queries
Eve tries to join despite being banned!
groups: { _id: ‘8rOIwh2VD’, members: [ {name: ‘Alice’, dateJoined: ‘2012-05-24’}, {name: ‘Bob’, dateJoined: ‘2012-05-25’}
] banned: [ {name: ‘Eve’, dateBanned: ‘2012-05-25’}
]}
Alice and Bob are in an awesome group
groups.update( {_id: ‘8rOIwh2VD’}, {$push: {members: {name: ‘Eve’, dateJoined: new Date()}
}, }, function(err) {
if (err) throw err;callback();
});
Bugs in our code fail to detect Eve’s trickery and the update query is run!
groups: { _id: ‘8rOIwh2VD’, members: [ {name: ‘Alice’, dateJoined: ‘2012-05-24’}, {name: ‘Bob’, dateJoined: ‘2012-05-25’}, {name: ‘Eve’, dateJoined: ‘2012-05-26’}
] banned: [ {name: ‘Eve’, dateRequested: ‘2012-05-25’}
]}
Blast! Eve got through! How can we help prevent this?
Bake in our assumptions!
groups.update( {_id: ‘8rOIwh2VD’, ‘members.name’: {$ne: ‘Eve’}, ‘banned.name’: {$ne: ‘Eve’}}, {$push: {members: {name: ‘Eve’, dateJoined: new Date()}
}, }, function(err, updateCount) { if (err) throw err; if (updateCount !== 1) throw new Error(‘bugs!’); callback(updateCount === 1); });
Always Specify Fields
To assign a player a new chain, we only need their _id and game information
players
account
info
login
chainHistory
stats
gamegame
To load a player’s profile content, we just need their history, stats, and profile info
players
account
info
login
chainHistory
stats
gamechainHistory
stats
info
We need to assign Bob a new chain, lets figure out what state he needs next
db.players.find({name: ‘Bob’});
This will fetch and send back the ENTIRE player object!
Lets specify that we only want our “game” subdocument
db.players.find({name: ‘Bob’}, {fields: [‘game’]});
Hooray! Much less to retrieve and send over the series of tubes.
Store Everything!
If you have information available, save it even if you don’t plan on using it!
db.chains.update({_id: ‘2VmVO18hs’}, {$push: {player_id: ‘1qyY61Lu1’, state: ‘draw’, date: new Date(), content: {time: 106896,
count: 27,
width: 520,
height: 390,
step_id: ‘1i7RlFbgU’ } });
Some Pain Points
Less obvious database structure
mysql> DESC players;+----------------+---------+------+-----+---------+-------+| Field | Type | Null | Key | Default | Extra |+----------------+---------+------+-----+---------+-------+| id | int(11) | NO | PRI | NULL | || activeStateId | int(11) | YES | | NULL | || activeStepId | int(11) | YES | | NULL | |+----------------+---------+------+-----+---------+-------+
Data integrity up to application
Mysql> ALTER TABLE players ADD CONSTRAINT fk_players_chains FOREIGN KEY (activeChainId) REFERENCES chains(id);
/* Oh no! This is going to mess things up! */
players.update({_id: ‘c58D4’}, {$set: {‘game.activeChain_id’: ‘bad_id’}});
No mature GUI query development tools
If we could start over would we still use MongoDB?
Absolutely!
• NoSQL in general is great for rapid prototyping
• Fantastic performance
• Intuitive language keeps it easy to run one-off queries / refactor schema
• Excellent (and now officially supported) Node driver
Questions?
@DoodleOrDie@Zugwalt