MongoDB With Style

116
with style gabriele lana [email protected] twitter: @gabrielelana http://joind.in/2943

description

I tried to put two years of experience with MongoDB in this few slides, hope you like a few tricks who saved me some headache

Transcript of MongoDB With Style

Page 1: MongoDB With Style

withstyle

gabriele [email protected]

twitter: @gabrielelanahttp://joind.in/2943

Page 2: MongoDB With Style

Styl

e

query

&d

esig

n

scale

Page 3: MongoDB With Style

mongo console

$  ~/Work/opt/mongodb-­‐1.6.5/bin/mongod  \          -­‐-­‐dbpath=~/Work/src/nosqlday/db/mongodb.01  \          -­‐-­‐logpath=~/Work/src/nosqlday/log/mongodb.01  \          -­‐-­‐fork  -­‐-­‐port  30001

$  ~/Work/opt/mongodb-­‐1.6.5/bin/mongo  localhost:30001MongoDB  shell  version:  1.6.5connecting  to:  localhost:30001/test

>  use  nosqldayswitched  to  db  nosqlday

>  db.getCollectionNames()[  "system.indexes",  "users"  ]

>  db.users.find({  "name":  "Gabriele"  }){  "_id"  :  ObjectId("4d8706767bb037a8a8f98db2"),  "name"  :  "Gabriele",  "surname"  :  "Lana",  "job"  :  "softwarecraftsman"  }

>  exitbye

Page 4: MongoDB With Style

ruby driver

require "mongo"

db = Mongo::Connection.new("localhost", 30001).db("nosqlday")

puts "Collections:"db.collections.each do |collection| puts "\t#{collection.name}"end

puts "Gabriele:"db["users"].find(:name => "Gabriele").each do |user| puts "\t#{user["_id"]}"end

db.connection.close

Page 5: MongoDB With Style

require "mongo"

db = Mongo::Connection.new("localhost", 30001).db("nosqlday")

puts "Collections:"db.collections.each do |collection| puts "\t#{collection.name}"end

puts "Gabriele:"db["users"].find(:name => "Gabriele").each do |user| puts "\t#{user["_id"]}"end

db.connection.close

ruby driver

$  ruby  src/connect.rb  Collections:   users   system.indexesGabriele:   4d8706767bb037a8a8f98db2

Page 6: MongoDB With Style

Styl

e

query

&d

esig

n

scale

Page 7: MongoDB With Style

Styl

eStyl

e

know yourdriver

Page 8: MongoDB With Style

document object mapper

smart driver

mongo

Page 9: MongoDB With Style

puts "Gabriele:"db["users"].find(:name => "Gabriele").each do |user| puts "\t#{user["_id"]}"end

puts "Gabriele:"db["users"].select{|user| user["name"] == "Gabriele"}.each do |user| puts "\t#{user["_id"]}"end

smart driver

mongo

Page 10: MongoDB With Style

puts "Gabriele:"db["users"].find(:name => "Gabriele").each do |user| puts "\t#{user["_id"]}"end

puts "Gabriele:"db["users"].select{|user| user["name"] == "Gabriele"}.each do |user| puts "\t#{user["_id"]}"end

smart driver

mongo

$  ruby  src/find_vs_select.rb  Gabriele:   4d8706767bb037a8a8f98db2Gabriele:   4d8706767bb037a8a8f98db2

Page 11: MongoDB With Style

puts "Gabriele:"db["users"].find(:name => "Gabriele").each do |user| puts "\t#{user["_id"]}"end

puts "Gabriele:"db["users"].select{|user| user["name"] == "Gabriele"}.each do |user| puts "\t#{user["_id"]}"end

smart driver

mongo

Page 12: MongoDB With Style

Styl

eStyl

e incremental design

based onapplicationbehavior

Page 13: MongoDB With Style

the best design is the one where needed data can be easily extracted

the way you need to query your data should influence your design

Page 14: MongoDB With Style

Styl

eStyl

e incremental design

based onapplicationmonitoring

Page 15: MongoDB With Style

monitoring and adapting is better than doing it right the first time

...actually the first timeis the worst time :-)

Page 16: MongoDB With Style

monitoring & adapting

>  db.setProfilingLevel(1,  5)                                                                                                                                                                                                                                                                                            {  "was"  :  1,  "slowms"  :  100,  "ok"  :  1  }

//  after  product  usage  find  problematic  queries

>  db.system.profile.find().sort({millis:-­‐1})                                                                                                          {  "ts":  "Mon  Mar  21  2011  14:30:56  GMT+0100  (CET)",    "info":  "        query  pomodorist.pomodori            reslen:202            nscanned:26950            query:                  {  $query:  {  task_id:  ObjectId('4d6f1d3931f2386e9c089796')  }}            nreturned:1      ",      "millis":17}

Page 17: MongoDB With Style

monitoring & adapting

>  db.pomodori.find({        $query:  {  task_id:  ObjectId('4d6f1d3931f2386e9c089796')  },        $explain:  true})                                                      {  "cursor":  "BasicCursor",    "nscanned":  26950,    "nscannedObjects":  26950,    "n":  1,    "millis":  17,    "indexBounds":  {  },    "allPlans":  [        {  "cursor"  :  "BasicCursor",  "indexBounds"  :  {  }  }      ]}

Page 18: MongoDB With Style

monitoring & adapting

>  db.pomodori.ensureIndex({"task_id":  1})                                                                                                              >  db.pomodori.find({        $query:  {  task_id:  ObjectId('4d6f1d3931f2386e9c089796')  },        $explain:  true})

{  "cursor":  "BtreeCursor  task_id_1",    "nscanned":  1,    "nscannedObjects":  1,    "n":  1,    "millis":  0,    "indexBounds":  {        "task_id":  [      [                ObjectId("4d6f1d3931f2386e9c089796"),                ObjectId("4d6f1d3931f2386e9c089796")          ]    ]},  "allPlans":  [...]}

Page 19: MongoDB With Style

Styl

e

query

&d

esig

n

scale

Page 20: MongoDB With Style

use $inoperatorfor batch

query

query

&d

esig

n

Page 21: MongoDB With Style

retrieve all objects with $in

users = [{:name => "Gabriele", :surname => "Lana", :job => "softwarecraftsman"},{:name => "Federico", :surname => "Galassi", :job => "softwarecraftsman"},{:name => "Giordano", :surname => "Scalzo", :job => "softwarecraftsman"}]

ids = users.map{|user| db["users"].insert(user)}

puts ids.map{|id| db["users"].find_one(:_id => id)}

Page 22: MongoDB With Style

retrieve all objects with $in

users = [{:name => "Gabriele", :surname => "Lana", :job => "softwarecraftsman"},{:name => "Federico", :surname => "Galassi", :job => "softwarecraftsman"},{:name => "Giordano", :surname => "Scalzo", :job => "softwarecraftsman"}]

ids = users.map{|user| db["users"].insert(user)}

puts ids.map{|id| db["users"].find_one(:_id => id)}

$  ruby  src/find_by_all_ids.rb  {"_id"=>BSON::ObjectId('4d87605731f23824a0000001'),  ...}{"_id"=>BSON::ObjectId('4d87605731f23824a0000002'),  ...}{"_id"=>BSON::ObjectId('4d87605731f23824a0000003'),  ...}

Page 23: MongoDB With Style

retrieve all objects with $in

users = [{:name => "Gabriele", :surname => "Lana", :job => "softwarecraftsman"},{:name => "Federico", :surname => "Galassi", :job => "softwarecraftsman"},{:name => "Giordano", :surname => "Scalzo", :job => "softwarecraftsman"}]

ids = users.map{|user| db["users"].insert(user)}

puts ids.map{|id| db["users"].find_one(:_id => id)}

Page 24: MongoDB With Style

retrieve all objects with $in

users = [{:name => "Gabriele", :surname => "Lana", :job => "softwarecraftsman"},{:name => "Federico", :surname => "Galassi", :job => "softwarecraftsman"},{:name => "Giordano", :surname => "Scalzo", :job => "softwarecraftsman"}]

ids = users.map{|user| db["users"].insert(user)}ids = db["users"].insert(users)

puts ids.map{|id| db["users"].find_one(:_id => id)}puts db["users"].find(:_id => {:$in => ids}).all

Page 25: MongoDB With Style

retrieve all objects with $in

users = [{:name => "Gabriele", :surname => "Lana", :job => "softwarecraftsman"},{:name => "Federico", :surname => "Galassi", :job => "softwarecraftsman"},{:name => "Giordano", :surname => "Scalzo", :job => "softwarecraftsman"}]

ids = users.map{|user| db["users"].insert(user)}ids = db["users"].insert(users)

puts ids.map{|id| db["users"].find_one(:_id => id)}puts db["users"].find(:_id => {:$in => ids}).all

$  ruby  src/find_by_all_ids.rb  {"_id"=>BSON::ObjectId('4d87605731f23824a0000001'),  ...}{"_id"=>BSON::ObjectId('4d87605731f23824a0000002'),  ...}{"_id"=>BSON::ObjectId('4d87605731f23824a0000003'),  ...}

Page 26: MongoDB With Style

use conventions to

build smartobject

identifiers

query

&d

esig

n

Page 27: MongoDB With Style

conventions are fun to play with

>  db.user_scores.find({},  {"_id":  1})

{  "_id"  :  "4d873ce631f238241d00000d-­‐day-­‐20091106"  }{  "_id"  :  "4d873ce631f238241d00000d-­‐week-­‐200944"  }{  "_id"  :  "4d873ce631f238241d00000d-­‐month-­‐200911"  }{  "_id"  :  "4d873ce631f238241d00000d-­‐year-­‐2009"  }{  "_id"  :  "4d873ce631f238241d00000d-­‐user"  }{  "_id"  :  "4d873ce631f238241d00000d-­‐day-­‐20091106-­‐advertising"  }{  "_id"  :  "4d873ce631f238241d00000d-­‐week-­‐200944-­‐advertising"  }{  "_id"  :  "4d873ce631f238241d00000d-­‐day-­‐20091106-­‐art"  }{  "_id"  :  "4d873ce631f238241d00000d-­‐week-­‐200944-­‐art"  }{  "_id"  :  "4d873ce631f238241d00000d-­‐day-­‐20091106-­‐artist"  }{  "_id"  :  "4d873ce631f238241d00000d-­‐week-­‐200944-­‐artist"  }{  "_id"  :  "4d873ce631f238241d00000d-­‐day-­‐20091106-­‐information"  }

Page 28: MongoDB With Style

conventions are fun to play with

>  db.user_scores.findOne(        {"_id":  "4d873ce631f238241d00000d-­‐day-­‐20091106"}    )  

{   "_id"  :  "4d873ce631f238241d00000d-­‐day-­‐20091106",   "pomodori"  :  15,   "pomodori_squashed"  :  3,   "breaks"  :  7,   "tasks_created"  :  8,   "tasks_done"  :  6,   "estimation_accuracy"  :  0,   "seconds_of_focused_time"  :  22500,   "seconds_of_wasted_time"  :  1999,   "seconds_of_breaks"  :  8820}

Page 29: MongoDB With Style

conventions are fun to play with(user scores in day per tag)

>  db.user_scores.find(        {"_id":  /^4d873ce631f238241d00000d-­‐day-­‐20091106-­‐/},  {"_id":  1}    )                {  "_id"  :  "4d873ce631f238241d00000d-­‐day-­‐20091106-­‐advertising"  }{  "_id"  :  "4d873ce631f238241d00000d-­‐day-­‐20091106-­‐art"  }{  "_id"  :  "4d873ce631f238241d00000d-­‐day-­‐20091106-­‐artist"  }{  "_id"  :  "4d873ce631f238241d00000d-­‐day-­‐20091106-­‐blogging"  }{  "_id"  :  "4d873ce631f238241d00000d-­‐day-­‐20091106-­‐culture"  }{  "_id"  :  "4d873ce631f238241d00000d-­‐day-­‐20091106-­‐html"  }{  "_id"  :  "4d873ce631f238241d00000d-­‐day-­‐20091106-­‐illustration"  }{  "_id"  :  "4d873ce631f238241d00000d-­‐day-­‐20091106-­‐information"  }{  "_id"  :  "4d873ce631f238241d00000d-­‐day-­‐20091106-­‐inspiration"  }{  "_id"  :  "4d873ce631f238241d00000d-­‐day-­‐20091106-­‐marketing"  }{  "_id"  :  "4d873ce631f238241d00000d-­‐day-­‐20091106-­‐movies"  }{  "_id"  :  "4d873ce631f238241d00000d-­‐day-­‐20091106-­‐resources"  }{  "_id"  :  "4d873ce631f238241d00000d-­‐day-­‐20091106-­‐technology"  }{  "_id"  :  "4d873ce631f238241d00000d-­‐day-­‐20091106-­‐tool"  }{  "_id"  :  "4d873ce631f238241d00000d-­‐day-­‐20091106-­‐tutorials"  }

Page 30: MongoDB With Style

conventions are fun to play with(list of tags per day)

>  db.user_scores.find(        {"_id":  /^4d873ce631f238241d00000d-­‐day-­‐20091106-­‐/},  {"_id":  1}

   ).map(function(document)  {        return  document._id.replace(            "4d873ce631f238241d00000d-­‐day-­‐20091106-­‐",  ""        )    })

[   "advertising",   "art",   "artist",   "blogging",   "culture",   "html",   "illustration",   "information",   ...]

Page 31: MongoDB With Style

conventions are fun to play with(anchored regexp uses indexes)

>  db.user_scores.find(        {"_id":  /^4d873ce631f238241d00000d-­‐day-­‐20091106-­‐/},  {"_id":  1}    ).explain()                                                                                                                                                                            {   "cursor"  :  "BtreeCursor  _id_  multi",   "nscanned"  :  15,   "nscannedObjects"  :  15,   "n"  :  15,   "millis"  :  0,   "indexBounds"  :  {     "_id"  :  [       [         "4d873ce631f238241d00000d-­‐day-­‐20091106-­‐",         "4d873ce631f238241d00000d-­‐day-­‐20091106."       ],       [         /^4d873ce631f238241d00000d-­‐day-­‐20091106-­‐/,         /^4d873ce631f238241d00000d-­‐day-­‐20091106-­‐/       ]     ]

Page 32: MongoDB With Style

conventions are fun to play with(anchored regexp uses indexes)

>  db.user_scores.find(        {"_id":  /4d873ce631f238241d00000d-­‐day-­‐20091106-­‐/},  {"_id":  1}    ).explain()

{   "cursor"  :  "BtreeCursor  _id_  multi",   "nscanned"  :  109349,   "nscannedObjects"  :  15,   "n"  :  15,   "millis"  :  217,   "indexBounds"  :  {     "_id"  :  [       ...     ]   }}

Page 33: MongoDB With Style

use “group”method todo small

computationswithoutfetchingrelated

documentsquery

&d

esig

n

Page 34: MongoDB With Style

group to compute data in mongo(inject client side)

days = [ 20091110, 20091111, 20091112 ]scores_id = %r{^4d87d00931f2380c7700000d-day-(#{days.join("|")})$}

scores = db["user_scores"].find(:_id => scores_id)

pomodori = scores.inject(0) do |pomodori, scores| pomodori + scores["pomodori"]end

puts "Pomodori in days #{days.join(",")}: #{pomodori}"

Page 35: MongoDB With Style

group to compute data in mongo(inject client side)

days = [ 20091110, 20091111, 20091112 ]scores_id = %r{^4d87d00931f2380c7700000d-day-(#{days.join("|")})$}

scores = db["user_scores"].find(:_id => scores_id)

pomodori = scores.inject(0) do |pomodori, scores| pomodori + scores["pomodori"]end

puts "Pomodori in days #{days.join(",")}: #{pomodori}"

$  ruby  src/inject_for_reduce.rb  Pomodori  in  days  20091110,20091111,20091112:  36

Page 36: MongoDB With Style

group to compute data in mongo(group server side)

days = [ 20091110, 20091111, 20091112 ]scores_id = %r{^4d87d00931f2380c7700000d-day-(#{days.join("|")})$}

result = db["user_scores"].group( :cond => { :_id => scores_id }, :initial => { :pomodori => 0 }, :reduce => <<-EOF function(document, result) { result.pomodori += document.pomodori } EOF)

puts "Pomodori in days #{days.join(",")}: #{result.first["pomodori"]}"

Page 37: MongoDB With Style

group to compute data in mongo(group server side)

days = [ 20091110, 20091111, 20091112 ]scores_id = %r{^4d87d00931f2380c7700000d-day-(#{days.join("|")})$}

result = db["user_scores"].group( :cond => { :_id => scores_id }, :initial => { :pomodori => 0 }, :reduce => <<-EOF function(document, result) { result.pomodori += document.pomodori } EOF)

puts "Pomodori in days #{days.join(",")}: #{result.first["pomodori"]}"

$  ruby  src/group_for_reduce.rb  Pomodori  in  days  20091110,20091111,20091112:  36

Page 38: MongoDB With Style

group to compute data in mongo(ex. sum pomodori by tag “ruby”)

result = db["user_scores"].group( :cond => { :_id => /^4d87d00931f2380c7700000d-day-\d{8}-ruby$/ }, :initial => { :pomodori => 0, :days => 0 }, :reduce => <<-EOF function(document, result) { result.days += 1 result.pomodori += document.pomodori } EOF).first

puts "In #{result["days"]} days, #{result["pomodori"]} done for ruby"

Page 39: MongoDB With Style

group to compute data in mongo(ex. sum pomodori by tag “ruby”)

result = db["user_scores"].group( :cond => { :_id => /^4d87d00931f2380c7700000d-day-\d{8}-ruby$/ }, :initial => { :pomodori => 0, :days => 0 }, :reduce => <<-EOF function(document, result) { result.days += 1 result.pomodori += document.pomodori } EOF).first

puts "In #{result["days"]} days, #{result["pomodori"]} pomodori"

$  ruby  src/group_for_ruby_tag.rb  In  43  days,  45  pomodori

Page 40: MongoDB With Style

group to compute data in mongo(ex. sum pomodori by tag “ruby”)

>  db.user_scores.find({        "_id":  /^4d87d00931f2380c7700000d-­‐day-­‐\d{8}-­‐ruby$/    }).explain()

{   "cursor"  :  "BtreeCursor  _id_  multi",   "nscanned"  :  43,   "nscannedObjects"  :  43,   "n"  :  43,   "millis"  :  3,   "indexBounds"  :  {     "_id"  :  [...]   }}

Page 41: MongoDB With Style

create indexes on arrays to create local

reverse indexes in documents

query

&d

esig

n

Page 42: MongoDB With Style

reverse index in place(an array could be indexed)

>  db.tasks.find({  "tags":  {  $in:  [  "nosqlday"  ]  }  })                                                                            {  "_id"  :  ObjectId("4d7de446175ca8243d000004"),      "tags"  :  [  "nosqlday"  ],      "description"  :  "#nosqlday  keynote",      "is_recurrent"  :  false,    "estimated"  :  0,      "worked_in"  :  [   "Mon  Mar  14  2011  00:00:00  GMT+0100  (CET)",   "Tue  Mar  15  2011  00:00:00  GMT+0100  (CET)"    ],    "done_at"  :  "Tue  Mar  15  2011  13:05:03  GMT+0100  (CET)",    "todo_at"  :  null,    "created_at"  :  "Mon  Mar  14  2011  10:47:50  GMT+0100  (CET)",    "updated_at"  :  "Tue  Mar  15  2011  13:05:03  GMT+0100  (CET)",    "keywords":  [  "nosqldai",  "keynot"  ],    "user_id":  ObjectId("4d53996c137ce423ff000001"),    "annotations"  :  [  ]}

Page 43: MongoDB With Style

reverse index in place(an array could be indexed)

>  db.tasks.getIndexes()[   {     "name"  :  "_id_",     "ns"  :  "app435386.tasks",     "key"  :  {       "_id"  :  1     }   },   {     "name"  :  "tags_1",     "ns"  :  "app435386.tasks",     "key"  :  {       "tags"  :  1     },     "unique"  :  false   },      ...]

Page 44: MongoDB With Style

reverse index in place(container for deduced data, array)

db["orders"].insert({ :placed_at => [ now.strftime("%Y"), # year: "2011" now.strftime("%Y%m"), # month: "201103" now.strftime("%Yw%U"), # week: "2011w11" now.strftime("%Y%m%d") # day: "20110316" ], :user_id => user, :items => items_in_order.map{|item| item[:id]}, :total => items_in_order.inject(0){|total,item| total += item[:price]}})

# ...

db["orders"].ensure_index([["placed_at", Mongo::DESCENDING]])

Page 45: MongoDB With Style

>  db.orders.findOne()

{  "_id"  :  ObjectId("4d88bf1f31f23812de0003fd"),      "placed_at"  :  [  "2011",  "201103",  "2011w11",  "20110316"  ],    "user_id"  :  ObjectId("4d88bf1f31f23812de0003e9"),    "items"  :  [        ObjectId("4d88bf1f31f23812de0003da"),        ObjectId("4d88bf1f31f23812de000047"),        ObjectId("4d88bf1f31f23812de000078"),        ObjectId("4d88bf1f31f23812de000068"),      ObjectId("4d88bf1f31f23812de000288")    ],    "total"  :  3502}

reverse index in place(container for deduced data, array)

Page 46: MongoDB With Style

>  db.orders.find({  "placed_at":  "20110310"  }).count()77

>  db.orders.find({  "placed_at":  "20110310"  }).explain(){   "cursor"  :  "BtreeCursor  placed_at_-­‐1",   "nscanned"  :  77,   "nscannedObjects"  :  77,   "n"  :  77,   "millis"  :  0,   "indexBounds"  :  {     "placed_at"  :  [       [         "20110310",         "20110310"       ]     ]   }}

reverse index in place(container for deduced data, array)

Page 47: MongoDB With Style

reverse index in place(container for deduced data, hash)

db["orders"].insert({ :placed_at => [ { :year => now.strftime("%Y") }, { :month => now.strftime("%Y%m") }, { :week => now.strftime("%Y%U") }, { :day => now.strftime("%Y%m%d") } ], :user_id => user, :items => items_in_order.map{|item| item[:id]}, :total => items_in_order.inject(0){|total,item| total += item[:price]} })

# ...

db["orders"].ensure_index([["placed_at", Mongo::DESCENDING]])

Page 48: MongoDB With Style

>  db.orders.findOne()                                                            {  "_id"  :  ObjectId("4d88c31531f23812fe0003ea"),    "placed_at"  :  [        {  "year"  :  "2009"  },        {  "month"  :  "200911"  },        {  "week"  :  "200945"  },        {  "day"  :  "20091109"  }    ],    "user_id"  :  ObjectId("4d88c31531f23812fe0003e9"),    "items"  :  [        ObjectId("4d88c31531f23812fe00013f"),        ObjectId("4d88c31531f23812fe000176"),        ObjectId("4d88c31531f23812fe0003e2"),        ObjectId("4d88c31531f23812fe0003d1"),        ObjectId("4d88c31531f23812fe0001c1"),        ObjectId("4d88c31531f23812fe000118"),        ObjectId("4d88c31531f23812fe00031d")    ],    "total"  :  10149}

reverse index in place(container for deduced data, hash)

Page 49: MongoDB With Style

>  db.orders.find({  "placed_at.week":  "201101"  }).count()                331

>  db.orders.find({  "placed_at.week":  "201101"  }).explain()            {   "cursor"  :  "BasicCursor",   "nscanned"  :  22374,   "nscannedObjects"  :  22374,   "n"  :  331,   "millis"  :  23,   "indexBounds"  :  {       }}

reverse index in place(container for deduced data, hash)

Page 50: MongoDB With Style

>  db.orders.find({  "placed_at":  {  "week":  "201101"  }}).count()    331

>  db.orders.find({  "placed_at":  {  "week":  "201101"  }}).explain(){   "cursor"  :  "BtreeCursor  placed_at_-­‐1",   "nscanned"  :  331,   "nscannedObjects"  :  331,   "n"  :  331,   "millis"  :  0,   "indexBounds"  :  {     "placed_at"  :  [       [         {  "week"  :  "2011w01"  },         {  "week"  :  "2011w01"  }       ]     ]   }}

reverse index in place(container for deduced data, hash)

Page 51: MongoDB With Style

use dates butbe aware of

some pitfalls

query

&d

esig

n

Page 52: MongoDB With Style

db["orders"].insert({ :placed_at => now, :user_id => user, :items => items_in_order.map{|item| item[:id]}, :total => items_in_order.inject(0){|total,item| total += item[:price]} })

# ...

db["orders"].ensure_index([["placed_at", Mongo::DESCENDING]])

plain dates are good too

Page 53: MongoDB With Style

>  db.orders.findOne()                                                                                                                                              {   "_id"  :  ObjectId("4d88d1f931f23813a10003ea"),   "placed_at"  :  "Mon  Nov  09  2009  08:00:00  GMT+0100  (CET)",   "user_id"  :  ObjectId("4d88d1f931f23813a10003e9"),   "items"  :  [     ObjectId("4d88d1f931f23813a100016d"),     ObjectId("4d88d1f931f23813a1000346"),     ObjectId("4d88d1f931f23813a10001e7"),     ObjectId("4d88d1f931f23813a10000db"),     ObjectId("4d88d1f931f23813a1000091"),     ObjectId("4d88d1f931f23813a10001c1"),     ObjectId("4d88d1f931f23813a10001d3"),     ObjectId("4d88d1f931f23813a100031b"),     ObjectId("4d88d1f931f23813a1000130")   ],   "total"  :  5871}

plain dates are good too

Page 54: MongoDB With Style

>  db.orders.find({        "placed_at":  {              $gte:  new  Date(2011,2,10),            $lt:  new  Date(2011,2,11)        }    }).explain()

{   "cursor"  :  "BtreeCursor  placed_at_-­‐1",   "nscanned"  :  53,   "nscannedObjects"  :  53,   "n"  :  53,   "millis"  :  0,   "indexBounds"  :  {     "placed_at"  :  [       [         "Fri  Mar  11  2011  00:00:00  GMT+0100  (CET)",         "Thu  Mar  10  2011  00:00:00  GMT+0100  (CET)"       ]     ]   }

plain dates are good too

Page 55: MongoDB With Style

# find all mondays of the yearnow = Time.now.beginning_of_year

now += 1.day until now.monday?mondays = [ now ]mondays << now += 7.days while now.year == Time.now.year

# find all orders placed on mondaysquery = { :$or => mondays.map do |day| { :placed_at => { :$gte => day.beginning_of_day, :$lte => day.end_of_day } } end}

puts query

plain dates are good too, but...(total sold on this year’s mondays)

Page 56: MongoDB With Style

# find all mondays of the yearnow = Time.now.beginning_of_year

now += 1.day until now.monday?mondays = [ now ]mondays << now += 7.days while now.year == Time.now.year

# find all orders placed on mondaysquery = { :$or => mondays.map do |day| { :placed_at => { :$gte => day.beginning_of_day, :$lte => day.end_of_day } } end}

puts query

$  ruby  src/orders_on_mondays.rb  

{:$or=>[    {:placed_at=>{        :$gte=>2011-­‐01-­‐03  00:00:00  +0100,        :$lte=>2011-­‐01-­‐03  23:59:59  +0100    }},    {:placed_at=>{        :$gte=>2011-­‐01-­‐10  00:00:00  +0100,        :$lte=>2011-­‐01-­‐10  23:59:59  +0100    }},    {:placed_at=>{        :$gte=>2011-­‐01-­‐17  00:00:00  +0100,        :$lte=>2011-­‐01-­‐17  23:59:59  +0100    }},    ...]}

plain dates are good too, but...(total sold on this year’s mondays)

Page 57: MongoDB With Style

db["orders"].find({ :$or => mondays.map do |day| { :placed_at => { :$gte => day.beginning_of_day, :$lte => day.end_of_day } } end})

plain dates are good too, but...(it works but it’s too slooow)

Page 58: MongoDB With Style

>  db.orders.find({        $or:  [            "placed_at":{  $gte:  new  Date(2011,2,3),  $lt:  new  Date(2011,2,4)  },            "placed_at":{  $gte:  new  Date(2011,2,10),  $lt:  new  Date(2011,2,11)  }        ]    }).explain()

{    "clauses"  :  [{            "cursor"  :  "BtreeCursor  placed_at_-­‐1",            "indexBounds"  :  {                "placed_at"  :  [[                    "Tue  Mar  3  2011  00:00:00  GMT+0100  (CET)",                    "Wed  Mar  4  2011  00:00:00  GMT+0100  (CET)"                ]]}    },  {        "cursor"  :  "BtreeCursor  placed_at_-­‐1",            "indexBounds"  :  {                "placed_at"  :  [[                    "Tue  Mar  10  2011  00:00:00  GMT+0100  (CET)",                    "Wed  Mar  11  2011  00:00:00  GMT+0100  (CET)"          

plain dates are good too, but...(why it’s too slow)

Page 59: MongoDB With Style

>  db.orders.findOne()

{  "_id"  :  ObjectId("4d88bf1f31f23812de0003fd"),      "placed_at"  :  [  "2011",  "201103",  "2011w11",  "20110316"  ],    "user_id"  :  ObjectId("4d88bf1f31f23812de0003e9"),    "items"  :  [        ObjectId("4d88bf1f31f23812de0003da"),        ObjectId("4d88bf1f31f23812de000047"),        ObjectId("4d88bf1f31f23812de000078"),        ObjectId("4d88bf1f31f23812de000068"),        ObjectId("4d88bf1f31f23812de000288")    ],    "total"  :  3502}

with destructured dates(total sold on mondays this year)

Page 60: MongoDB With Style

now = Time.now.beginning_of_year

now += 1.day until now.monday?mondays = [ now ]mondays << now += 7.days while now.year == Time.now.year

orders = db["orders"].find({ :placed_at => { :$in => mondays.map {|day| day.strftime("%Y%m%d")} }})

puts orders.explain

with destructured dates(total sold on mondays this year)

Page 61: MongoDB With Style

with destructured dates(total sold on mondays this year)

now = Time.now.beginning_of_year

now += 1.day until now.monday?mondays = [ now ]mondays << now += 7.days while now.year == Time.now.year

orders = db["orders"].find({ :placed_at => { :$in => mondays.map {|day| day.strftime("%Y%m%d")} }})

puts orders.explain

$  ruby  src/orders_on_mondays.rb  

{  "cursor"=>"BtreeCursor  placed_at_-­‐1  multi",    "nscanned"=>744,    "nscannedObjects"=>744,    "n"=>744,    "millis"=>1,    "indexBounds"=>{        "placed_at"=>[            ["20120102",  "20120102"],  ["20111226",  "20111226"],            ["20111219",  "20111219"],  ["20111212",  "20111212"],              ["20111205",  "20111205"],  ["20111128",  "20111128"],              ["20111121",  "20111121"],  ...        ]    }}

Page 62: MongoDB With Style

full query power with

$whereoperator

query

&d

esig

n

Page 63: MongoDB With Style

pomodori(find who is ticking)

>  db.pomodori.findOne(){   "_id"  :  ObjectId("4d8916ed31f2381480000021"),   "duration"  :  1500,   "interruptions"  :  0,   "after_break_of"  :  0,   "started_at"  :  "Mon  Mar  14  2011  08:05:00  GMT+0100  (CET)",   "squashed_at"  :  "Mon  Mar  14  2011  08:07:31  GMT+0100  (CET)",   "in_day"  :  {     "position"  :  1,     "is_last"  :  false   },   "task_id"  :  ObjectId("4d8916ec31f2381480000014"),   "user_id"  :  ObjectId("4d8916ec31f2381480000010"),   "annotations"  :  [  ]}

Page 64: MongoDB With Style

now = Time.now.yesterday.beginning_of_day + 10.hourstimestamp_of_now = now.to_i

ticking = db["pomodori"].find( :$where => <<-EOF var startedAt = this.started_at.getTime()/1000 return ((startedAt + this.duration) > #{timestamp_of_now}) && (startedAt < #{timestamp_of_now}) EOF)

puts ticking.map{|pomodoro| pomodoro["_id"]}

pomodori(find who is ticking)

Page 65: MongoDB With Style

pomodori(find who is ticking)

now = Time.now.yesterday.beginning_of_day + 10.hourstimestamp_of_now = now.to_i

ticking = db["pomodori"].find( :$where => <<-EOF var startedAt = this.started_at.getTime()/1000 return ((startedAt + this.duration) > #{timestamp_of_now}) && (startedAt < #{timestamp_of_now}) EOF)

puts ticking.map{|pomodoro| pomodoro["_id"]}

$  ruby  src/find_who_is_ticking.rb  4d8916ef31f238148000011d4d8916f231f23814800002714d8916f931f23814800004dd4d8916f931f23814800004e0

Page 66: MongoDB With Style

now = Time.now.yesterday.beginning_of_day + 10.hourstimestamp_of_now = now.to_iuser_id = BSON::ObjectId.from_string("4d8916ec31f2381480000010")

ticking = db["pomodori"].find( :user_id => user_id, :$where => <<-EOF var startedAt = this.started_at.getTime()/1000 return ((startedAt + this.duration) > #{timestamp_of_now}) && (startedAt < #{timestamp_of_now}) EOF)

puts ticking.map{|pomodoro| pomodoro["_id"]}

pomodori(find who is ticking for an user)

Page 67: MongoDB With Style

now = Time.now.yesterday.beginning_of_day + 10.hourstimestamp_of_now = now.to_iuser_id = BSON::ObjectId.from_string("4d8916ec31f2381480000010")

ticking = db["pomodori"].find( :user_id => user_id, :$where => <<-EOF var startedAt = this.started_at.getTime()/1000 return ((startedAt + this.duration) > #{timestamp_of_now}) && (startedAt < #{timestamp_of_now}) EOF)

puts ticking.map{|pomodoro| pomodoro["_id"]}

pomodori(find who is ticking for an user)

$  ruby  src/find_who_is_ticking_for_an_user.rb  4d8916ef31f238148000011d

Page 68: MongoDB With Style

related_to_maps = db["pomodori"].find( :$where => <<-EOF db.tasks.findOne({ "_id": this.task_id }).tags.indexOf("maps") >= 0 EOF)

puts related_to_maps.map{|pomodoro| pomodoro["_id"]}

pomodori(related to tasks tagged with “maps”)

Page 69: MongoDB With Style

related_to_maps = db["pomodori"].find( :$where => <<-EOF db.tasks.findOne({ "_id": this.task_id }).tags.indexOf("maps") >= 0 EOF)

puts related_to_maps.map{|pomodoro| pomodoro["_id"]}

pomodori(related to tasks tagged with “maps”)

$  ruby  src/related_to_maps.rb  4d8916fa31f23814800005794d8916fa31f238148000057b4d8916fa31f238148000057d4d8916fa31f2381480000580

Page 70: MongoDB With Style

related_to_maps = db["pomodori"].find( :$where => <<-EOF db.tasks.findOne({ "_id": this.task_id }).tags.indexOf("maps") >= 0 EOF)

puts related_to_maps.explain

pomodori(don’t be carried away :-))

$  ruby  src/related_to_maps.rb  {  "cursor"=>"BasicCursor",      "nscanned"=>461,      "nscannedObjects"=>461,    "n"=>4,    "millis"=>52,      "indexBounds"=>{},      "allPlans"=>[...]}

Page 71: MongoDB With Style

related_to_maps = db["pomodori"].find(:task_id => { :$in => db["tasks"].find( {:tags => "maps"}, :fields => {:_id => 1} ).map{|task| task["_id"]}}) puts related_to_maps.map{|pomodoro| pomodoro["_id"]}

pomodori(related to... a better solution)

$  ruby  src/related_to_maps.rb  4d8916fa31f23814800005794d8916fa31f238148000057b4d8916fa31f238148000057d4d8916fa31f2381480000580

Page 72: MongoDB With Style

related_to_maps = db["pomodori"].find(:task_id => { :$in => db["tasks"].find( {:tags => "maps"}, :fields => {:_id => 1} ).map{|task| task["_id"]}}) puts related_to_maps.map{|pomodoro| pomodoro["_id"]}

$  ruby  src/related_to_maps.rb  {  "cursor"=>"BtreeCursor  tags_1",    "nscanned"=>3,    "nscannedObjects"=>3,    "n"=>3,    "millis"=>0,    ...}

{  "cursor"=>"BtreeCursor  task_id_1  multi",    "nscanned"=>4,    "nscannedObjects"=>4,    "n"=>4,    "millis"=>0,    ...}

pomodori(related to... a better solution)

Page 73: MongoDB With Style

real time analytics with increments

query

&d

esig

n

Page 74: MongoDB With Style

result = db["visits"].update( { :_id => Digest::MD5.hexdigest(url) }, { :$inc => { :hits => 1 } }, :upsert => true, :safe => true)

puts "Update: #{result.inspect}"

puts db["visits"].find_one(:_id => Digest::MD5.hexdigest(url))

keep track of url’s visits(upsert with custom id)

Page 75: MongoDB With Style

keep track of url’s visits(upsert with custom id)

result = db["visits"].update( { :_id => Digest::MD5.hexdigest(url) }, { :$inc => { :hits => 1 } }, :upsert => true, :safe => true)

puts "Update: #{result.inspect}"

puts db["visits"].find_one(:_id => Digest::MD5.hexdigest(url))

$  ruby  src/realtime_analytics.rb  Update:  {    "err"=>nil,    "updatedExisting"=>false,    "n"=>1,    "ok"=>1.0}{"_id"=>"2d86a774beffe90e715a8028c7bd177b",  "hits"=>1}

$  ruby  src/realtime_analytics.rb  Update:  {    "err"=>nil,    "updatedExisting"=>true,    "n"=>1,    "ok"=>1.0}{"_id"=>"2d86a774beffe90e715a8028c7bd177b",  "hits"=>2}

Page 76: MongoDB With Style

url_digest = Digest::MD5.hexdigest(url)ids = [ [ url_digest, Time.now.strftime("%Y%m%d") ].join("-"), [ url_digest, Time.now.strftime("%Y%m") ].join("-"), [ url_digest, Time.now.strftime("%Y") ].join("-"), [ url_digest, user_id ].join("-")]puts "Expect to upsert: \n#{ids}"

result = db["visits"].update( { :_id => { :$in => ids } }, { :$inc => { :hits => 1 } }, :multi => true, :upsert => true, :safe => true)puts result.inspectputs db["visits"].all

url’s visits aggregated by time(upsert with multiple documents)

Page 77: MongoDB With Style

url_digest = Digest::MD5.hexdigest(url)ids = [ [ url_digest, Time.now.strftime("%Y%m%d") ].join("-"), [ url_digest, Time.now.strftime("%Y%m") ].join("-"), [ url_digest, Time.now.strftime("%Y") ].join("-"), [ url_digest, user_id ].join("-")]puts "Expect to upsert: \n#{ids}"

result = db["visits"].update( { :_id => { :$in => ids } }, { :$inc => { :hits => 1 } }, :multi => true, :upsert => true, :safe => true)puts result.inspectputs db["visits"].all

$  ruby  src/realtime_analytics_with_aggregation.rb  Expect  to  upsert:[    "2d86a774beffe90e715a8028c7bd177b-­‐20110323",    "2d86a774beffe90e715a8028c7bd177b-­‐201103",    "2d86a774beffe90e715a8028c7bd177b-­‐2011",      "2d86a774beffe90e715a8028c7bd177b-­‐4d899fab31f238165c000001"]

{  "err"=>nil,    "updatedExisting"=>false,    "upserted"=>BSON::ObjectId('4d899fabe23bd37e768ae76d'),      "n"=>1,    "ok"=>1.0}

{"_id"=>BSON::ObjectId('4d899fabe23bd37e768ae76d'),  "hits"=>1}

url’s visits aggregated by time(upsert with multiple documents)

Page 78: MongoDB With Style

url_digest = Digest::MD5.hexdigest(url)ids = [ [ url_digest, Time.now.strftime("%Y%m%d") ].join("-"), [ url_digest, Time.now.strftime("%Y%m") ].join("-"), [ url_digest, Time.now.strftime("%Y") ].join("-"), [ url_digest, user_id ].join("-")]puts "Expect to upsert: \n#{ids}"

result = db["visits"].update( { :_id => { :$in => ids } }, { :$inc => { :hits => 1 } }, :multi => true, :upsert => true, :safe => true)puts result.inspectputs db["visits"].all

$  ruby  src/realtime_analytics_with_aggregation.rb  Expect  to  upsert:[    "2d86a774beffe90e715a8028c7bd177b-­‐20110323",    "2d86a774beffe90e715a8028c7bd177b-­‐201103",    "2d86a774beffe90e715a8028c7bd177b-­‐2011",      "2d86a774beffe90e715a8028c7bd177b-­‐4d899fab31f238165c000001"]

{  "err"=>nil,    "updatedExisting"=>false,    "upserted"=>BSON::ObjectId('4d899fabe23bd37e768ae76e'),      "n"=>1,    "ok"=>1.0}

{"_id"=>BSON::ObjectId('4d899fabe23bd37e768ae76d'),  "hits"=>1}{"_id"=>BSON::ObjectId('4d899fabe23bd37e768ae76e'),  "hits"=>1}

url’s visits aggregated by time(upsert with multiple documents)

Page 79: MongoDB With Style

result = db["visits"].update( { :_id => { :$in => ids } }, { :$inc => { :hits => 1 } }, :multi => true, :upsert => true, :safe => true)

if result["n"] != ids.size updated_ids = db["visits"].find( { :_id => { :$in => ids } }, :fields => { :_id => true } ).map{|document| document["_id"]}

db["visits"].insert((ids - updated_ids).map do |id| { :_id => id, :hits => 1 } end)

db["visits"].remove(:_id => result["upserted"]) if result["upserted"]end

url’s visits aggregated by time(look before you leap)

Page 80: MongoDB With Style

result = db["visits"].update( { :_id => { :$in => ids } }, { :$inc => { :hits => 1 } }, :multi => true, :upsert => true, :safe => true)

if result["n"] != ids.size updated_ids = db["visits"].find( { :_id => { :$in => ids } }, :fields => { :_id => true } ).map{|document| document["_id"]}

db["visits"].insert((ids - updated_ids).map do |id| { :_id => id, :hits => 1 } end)

db["visits"].remove(:_id => result["upserted"]) if result["upserted"]end

$  ruby  src/realtime_analytics_with_aggregation.rb{  "err"=>nil,      "updatedExisting"=>false,    "upserted"=>BSON::ObjectId('4d89a5ebe23bd37e768ae76f'),      "n"=>1,    "ok"=>1.0

}

{"_id"=>"<url_digest>-­‐20110323",  "hits"=>1}{"_id"=>"<url_digest>-­‐201103",  "hits"=>1}{"_id"=>"<url_digest>-­‐2011",  "hits"=>1}{"_id"=>"<url_digest>-­‐4d89a43b31f238167a000001",  "hits"=>1}

url’s visits aggregated by time(look before you leap)

Page 81: MongoDB With Style

result = db["visits"].update( { :_id => { :$in => ids } }, { :$inc => { :hits => 1 } }, :multi => true, :upsert => true, :safe => true)

if result["n"] != ids.size updated_ids = db["visits"].find( { :_id => { :$in => ids } }, :fields => { :_id => true } ).map{|document| document["_id"]}

db["visits"].insert((ids - updated_ids).map do |id| { :_id => id, :hits => 1 } end)

db["visits"].remove(:_id => result["upserted"]) if result["upserted"]end

$  ruby  src/realtime_analytics_with_aggregation.rb{  "err"=>nil,    "updatedExisting"=>true,    "n"=>3,    "ok"=>1.0}

{"_id"=>"<url_digest>-­‐20110323",  "hits"=>2}{"_id"=>"<url_digest>-­‐201103",  "hits"=>2}{"_id"=>"<url_digest>-­‐2011",  "hits"=>2}{"_id"=>"<url_digest>-­‐4d89a43b31f238167a000001",  "hits"=>1}{"_id"=>"<url_digest>-­‐4d89a44231f238167e000001",  "hits"=>1}

url’s visits aggregated by time(look before you leap)

Page 82: MongoDB With Style

incrementalmap/reduce

query

&d

esig

n

Page 83: MongoDB With Style

map/reduce hits per day(we have raw events)

>  db.visit_events.findOne(){   "_id"  :  ObjectId("4d89fc6531f2381d2c00000b"),   "url"  :  "8aa8b68e0b849f70df6dbb3031c6182b",   "user_id"  :  ObjectId("4d89fc6531f2381d2c000005"),   "at"  :  "Thu  Jan  13  2011  08:00:06  GMT+0100  (CET)"}

Page 84: MongoDB With Style

def generate_events(visits, db, now) visits.times do |time| now += BETWEEN_VISITS.sample.seconds db["visit_events"].insert( :url => Digest::MD5.hexdigest(URLS.sample), :user_id => USERS.sample[:id], :at => now ) endend

generate_events(10_000, db, now)

map/reduce hits per day(generate data WITH something like)

Page 85: MongoDB With Style

MAP = <<-EOF function() { emit([ this.url, this.at.format("Ymd") ].join("-"), { "hits": 1 }) }EOF

REDUCE = <<-EOF function(key, values) { var hits = 0 for(var index in values) hits += values[index]["hits"] return { "hits": hits } }EOF

result = db["visit_events"].map_reduce( MAP, REDUCE, :out => "visits", :raw => true, :verbose => true)

puts result.inspect

map/reduce hits per day(simple map/reduce)

Page 86: MongoDB With Style

MAP = <<-EOF function() { emit([ this.url, this.at.format("Ymd") ].join("-"), { "hits": 1 }) }EOF

REDUCE = <<-EOF function(key, values) { var hits = 0 for(var index in values) hits += values[index]["hits"] return { "hits": hits } }EOF

result = db["visit_events"].map_reduce( MAP, REDUCE, :out => "visits", :raw => true, :verbose => true)

puts result.inspect

map/reduce hits per day(date.prototype.format don’t exists)

Page 87: MongoDB With Style

MAP = <<-EOF function() { Date.prototype.format = function(format) { ... } emit([ this.url, this.at.format("Ymd") ].join("-"), { "hits": 1 }) }EOF

REDUCE = <<-EOF function(key, values) { var hits = 0 for(var index in values) hits += values[index]["hits"] return { "hits": hits } }EOF

map/reduce hits per day(implement format in place)

Page 88: MongoDB With Style

MAP = <<-EOF function() { if (!Date.prototype.format) { Date.prototype.format = function(format) { ... } } emit([ this.url, this.at.format("Ymd") ].join("-"), { "hits": 1 }) }EOF

REDUCE = <<-EOF function(key, values) { var hits = 0 for(var index in values) hits += values[index]["hits"] return { "hits": hits } }EOF

map/reduce hits per day(implement format only if needed)

Page 89: MongoDB With Style

db[Mongo::DB::SYSTEM_JS_COLLECTION].save( :_id => "formatDate", :value => BSON::Code.new( <<-EOF function(date, format) { if (!Date.prototype.format) { Date.prototype.format = function(format) { ... } } return date.format(format) } EOF ))

MAP = <<-EOF function() { emit([ this.url, formatDate(this.at, "Ymd") ].join("-"), {"hits":1}) }EOF

map/reduce hits per day(implement format once and for all)

Page 90: MongoDB With Style

db[Mongo::DB::SYSTEM_JS_COLLECTION].save( :_id => "load", :value => BSON::Code.new( <<-EOF function(module) { if ((module === "date") && !Date.prototype.format) { Date.prototype.format = function(format) { ... } } return true } EOF ))

MAP = <<-EOF function() { load("date") && emit( [ this.url, this.at.format("Ymd") ].join("-"), { "hits": 1 } ) }EOF

map/reduce hits per day(implement format once and for all)

Page 91: MongoDB With Style

MAP = <<-EOF function() { emit([ this.url, this.at.format("Ymd") ].join("-"), { "hits": 1 }) }EOF

REDUCE = <<-EOF function(key, values) { var hits = 0 for(var index in values) hits += values[index]["hits"] return { "hits": hits } }EOF

result = db["visit_events"].map_reduce( MAP, REDUCE, :out => "visits", :raw => true, :verbose => true)

puts result.inspect

map/reduce hits per day(ok, but could be taking too long)

$  ruby  src/incremental_mr.rb  {  "result"=>"visits",    "timeMillis"=>4197,    "timing"=>  {        "mapTime"=>3932,        "emitLoop"=>4170,        "total"=>4197    },    "counts"=>  {        "input"=>10000,        "emit"=>10000,        "output"=>200    },    "ok"=>1.0}

Page 92: MongoDB With Style

>  db.visits.find()                                                  

{  "_id"  :  "019640ff7952425b1b8695605459d223-­‐20110316",    "value"  :  {  "hits"  :  47  }}

{  "_id"  :  "019640ff7952425b1b8695605459d223-­‐20110317",    "value"  :  {  "hits"  :  49  }}

{  "_id"  :  "019640ff7952425b1b8695605459d223-­‐20110318",    "value"  :  {  "hits"  :  59  }  }

{  "_id"  :  "019640ff7952425b1b8695605459d223-­‐20110319",    "value"  :  {  "hits"  :  37  }  }

map/reduce hits per day(ok, every time we need to start over)

Page 93: MongoDB With Style

map/reduce hits per day(incremental with savepoints)

temporarycollection

visit-elementscollection

visitcollection

map/reduceon last changed

documentsupsert

Page 94: MongoDB With Style

map/reduce hits per day(incremental with savepoints)

db.create_collection("visit_events", :capped => true, :max => 50_000, :size => 5_000_000)

temporarycollection

visit-elementscollection

map/reduceon last changed

documents

Page 95: MongoDB With Style

FINALIZE = <<-EOF function(key, value) { db.visits.update( { "_id": key }, { $inc: { "hits": value.hits } }, true ) }EOF

map/reduce hits per day(incremental with savepoints)

temporarycollection

visitcollection

upsert

Page 96: MongoDB With Style

generate_events(number_of_events, db, now)

from = from_last_updated(db)to = to_last_inserted(db)

result = db["visit_events"].map_reduce( MAP, REDUCE, :finalize => FINALIZE, :query => { :_id => { :$gt => from, :$lte => to } }, :raw => true, :verbose => true)

db["visits"].save(:_id => "savepoint", :at => to)

map/reduce hits per day(incremental with savepoints)

Page 97: MongoDB With Style

generate_events(number_of_events, db, now)

from = from_last_updated(db)to = to_last_inserted(db)

result = db["visit_events"].map_reduce( MAP, REDUCE, :finalize => FINALIZE, :query => { :_id => { :$gt => from, :$lte => to } }, :raw => true, :verbose => true)

db["visits"].save(:_id => "savepoint", :at => to)

map/reduce hits per day(incremental with savepoints)

$  ruby  src/incremental_mr.rb  -­‐e  10000  {  "result"=>"tmp.mr.mapreduce_1300892393_60",    "timeMillis"=>4333,    "timing"=>{...},    "counts"=>{        "input"=>10000,          "emit"=>10000,        "output"=>196    },    "ok"=>1.0}

{  "_id"=>"05241f07d0e3ab6a227e67b33ea0b509-­‐20110113",          "hits"=>26}

Page 98: MongoDB With Style

generate_events(number_of_events, db, now)

from = from_last_updated(db)to = to_last_inserted(db)

result = db["visit_events"].map_reduce( MAP, REDUCE, :finalize => FINALIZE, :query => { :_id => { :$gt => from, :$lte => to } }, :raw => true, :verbose => true)

db["visits"].save(:_id => "savepoint", :at => to)

map/reduce hits per day(incremental with savepoints)

$  ruby  src/incremental_mr.rb  -­‐e  4999  {  "result"=>"tmp.mr.mapreduce_1300892399_61",    "timeMillis"=>2159,    "timing"=>{...},    "counts"=>{        "input"=>4999,        "emit"=>4999,        "output"=>146    },    "ok"=>1.0}

{  "_id"=>"05241f07d0e3ab6a227e67b33ea0b509-­‐20110113",      "hits"=>64}

Page 99: MongoDB With Style

def savepoint(db) db["visits"].find_one(:_id => "savepoint") or { "at" => BSON::ObjectId.from_time(10.years.ago) }end

def from_last_updated(db) savepoint["at"]end

def to_last_inserted(db) db["visit_events"].find.sort([:_id, Mongo::DESCENDING]).first["_id"]end

map/reduce hits per day(incremental with savepoints)

Page 100: MongoDB With Style

externalmap/reduce

query

&d

esig

n

Page 101: MongoDB With Style

master slave

replicate data

use an external mongod processto execute map/reduce jobs

Page 102: MongoDB With Style

master slave

map/reduceon last

replicateddata

use an external mongod processto execute map/reduce jobs

Page 103: MongoDB With Style

master slave

push back results

use an external mongod processto execute map/reduce jobs

Page 104: MongoDB With Style

look at the shell sourceis more powerful than you think

Page 105: MongoDB With Style

documents

embeddedor

linked?

query

&d

esig

n

Page 106: MongoDB With Style

life cycle:when root document

is deleted, he can stand for himself?

if yesembedded

if nolinked

Page 107: MongoDB With Style

if yesembedded

if nolinked

are always fetched together?

Page 108: MongoDB With Style

if yesembedded

if nolinked

his attributes are used to find the root

document?

Page 109: MongoDB With Style

if yesembedded

if nolinked

he’s small?

Page 110: MongoDB With Style

if yesembedded

if nolinked

he’s unique or there are less then

hundreds?

Page 111: MongoDB With Style

Styl

e

query

&d

esig

n

scale

Page 112: MongoDB With Style

distributedreads with

replicasets

scale

Page 113: MongoDB With Style

master

slave

slave

read/write

read

read

replicate

replicate

+ Durability+ fault tolerance

Page 114: MongoDB With Style

(seems stupid but...)

pumpyour

hardwarescale

Page 115: MongoDB With Style

(seems stupid but...)

call 10gensure they can

help :-)scale

Page 116: MongoDB With Style

Questions?

gabriele [email protected]

twitter: @gabrielelanahttp://joind.in/2943