Application Development SeriesBack to BasicsMonitoring & Performance Tuning

Daniel Roberts@dmroberts

#MongoDBBasics

2

• Recap from last session

• Monitoring Tools– MongoDB command line and shell

• Key Metrics in MongoDB

• Logs– Log levels– mtools

• Disk saturation

Agenda

3

• Virtual Genius Bar

– Use the chat to post questions

– EMEA Solution Architecture / Support team are on hand

– Make use of them during the sessions!!!

Q & A

Recap from last time….

5

• mongodump & mongorestore

• File system copy

• Files system snapshot

• Don’t use mongoimport & mongoexport!

Backup Tools & Approaches

//mongodump Exampleserver> mongodump -h myhost -d cms -c articles

6

• Creates a .bson file – (and a json metadata

file)– Over network or direct

from file system.

Mongodump

>mongodump –h myhost -d cms -c articlesconnected to: myhost 2014-04-16T12:54:56.758+0100 DATABASE: cms to dump/cms2014-04-16T12:54:56.759+0100 cms.articles to dump/cms/articles.bson2014-04-16T12:54:56.816+0100 7 documents2014-04-16T12:54:56.817+0100 Metadata for cms.articles to dump/cms/articles.metadata.json

7

• Use secondary for backup– Or hidden secondary

• Fsync+Lock

• Balance off – In sharded cluster

Backup Architecture

mongodump

Monitoring Tools

9

• Key MongoDB tools– Mongostat– Mongo shell– Mongo Management Service mms

• Mtools– Logs

• OS– iostat

Tools

10

MMS Architecture

11

MMS Alerts

12

Metrics Collection and Reporting

13

Logs and Profile data

14

• Access to key statistics– Connect to mongod or mongos for sharded cluster stats

– Shows, operations per second, memory usage, page faults, queues and more

Mongostat

>mongostat -h localhost --port 27017connected to: localhost:27017insert query update delete getmore command flushes mapped vsize res faults locked db idx miss % qr|qw ar|aw netIn netOut conn time *0 *0 *0 *0 0 1|0 0 1.67g 5.78g 29m 0 .:0.0% 0 0|0 0|0 62b 5k 2 09:08:24 *0 *0 *0 *0 0 1|0 0 1.67g 5.78g 29m 0 .:0.0% 0 0|0 0|0 62b 5k 2 09:08:25 *0 *0 *0 *0 0 1|0 0 1.67g 5.78g 29m 0 .:0.0% 0 0|0 0|0 62b 5k 2 09:08:26 *0 *0 *0 *0 0 1|0 0 1.67g 5.78g 29m 0 .:0.0% 0 0|0 0|0 62b 5k 2 09:08:27 *0 *0 *0 *0 0 1|0 0 1.67g 5.78g 29m 0 .:0.0% 0 0|0 0|0 62b 5k 2 09:08:28 *0 *0 *0 *0 0 1|0 0 1.67g 5.78g 29m 0 .:0.0% 0 0|0 0|0 62b 5k 2 09:08:29 *0 *0 *0 *0 0 1|0 0 1.67g 5.78g 29m 0 .:0.0% 0 0|0 0|0 62b 5k 2 09:08:30 *0 *0 *0 *0 0 1|0 0 1.67g 5.78g 29m 0 .:0.0% 0 0|0 0|0 62b 5k 2 09:08:31 *0 *0 *0 *0 0 1|0 0 1.67g 5.78g 29m 0 .:0.1% 0 0|0 0|0 62b 5k 2 09:08:32 *0 *0 *0 *0 0 1|0 0 1.67g 5.78g 29m 0 .:0.0% 0 0|0 0|0 62b 5k 2 09:08:33

15

• Commands for access metrics– db.serverStatus()– workingSet analyzer

Mongo shell

//to access a subset of serverStatus output specify the sub document>db.serverStatus().mem{

"bits" : 64,"resident" : 266,"virtual" : 5920,"supported" : true,"mapped" : 1712,"mappedWithJournal" : 3424

}

16

• Estimates the ‘working set’ of the database– Number of ‘pagesInMemory’ accessed in RAM over

‘overSeconds’ number of seconds.– Default Page is 4k– overSecond – delta between oldest and newest

WorkingSet analyzer

>db.serverStatus({ workingSet : 1 }){

"note" : "thisIsAnEstimate","pagesInMemory" : 723,"computationTimeMicros" : 4601,"overSeconds" : 2311

}

overSeconds decreasing or small, workingSet could be larger than RAM

MongoDB Metrics

18

• Queued readers | writers

• Page faults

• OpCounters

• Background flush process

• Memory usage

• Lock %

• Btree misses

• connections

Key Metrics

Available from :

• MMS

• db.serverStatus()

• mongostat

19

Queued Reader | Writers

• Number of operations waiting for:

• Read Lock

• Write Lock

20

Page Faults

• How often we are going to disk to retrieve data?

• Can be a problem if disk IO is saturated

• Combine with iostat

21

Background Flush Process

• Length of time to flush data back to disk

• Again can point to disk saturation.

22

Memory

• Virtual

• Mapped

• Resident

23

Lock %

• Percentage of time in the global lock

24

btree

• Accesses

• Hits

• Misses

• Bad!

• Points to Indexes not in RAM

25

connections

• Connections to the database

• Growing over time?

• Check pool logic in your application code.

Logs

27

• Increase the log level– See further insight to operation performance

– Index efficiency

– Document moves

– Operation time

Set log level

//Increase log level verbosity from the shell> db.adminCommand( { setParameter:1, logLevel:1 } ){ "was" : 0, "ok" : 1 }>

-v [ --verbose ] be more verbose (include multiple times for more verbosity e.g. -vvvvv)

28

• Pinpoint problems – In this example we can see that there has been a document move.

Log output example

2014-05-02T13:55:02.047+0100 [conn7] update cms.articles query: { _id: ObjectId('532198379fb5ba99a6bd4063') } update: { $inc: { comment_count: 1 }, $push: { comments: { $each: [ { date: new Date(1399035302013), text: "Data locality provides an amazing performance boost over relational" } ], $slice: -10, $sort: { date: 1 } } } } nscanned:1 nscannedObjects:1 nmoved:1 nMatched:1 nModified:1 keyUpdates:0 numYields:0 locks(micros) w:33529 33ms

29

• Log analysis (example syntax)– Show me queries that took more than 1000 ms from 6

am to 6 pm:

• Now, graph those queries:

mtools

prompt> mlogfilter mongodb.log --from 06:00 --to 18:00 --slow 1000 > mongodb-filtered.log

prompt> mplotqueries --logscale mongodb-filtered.log

30

mtools output

Disk saturation

32

• iostat– Disk IO usage

– Use to review % utilisation of disk– High %

• If it’s sustained, you may need to increase disk IO.– Sharding– RAID 1+0– Partitioning on different disk– Provisioned IOPS

OS Tools and Paramenters

prompt> iostat –xmt 1

Summary

34

• Monitor regularly– Set alerts for changes

• Quick check with command line and shell

• View trends with MMS graphs over time

• High levels of utilisation– Increase RAM, disk IO– Scale out– (Having first checked Indexes etc!)

• Database profiler (covered in Indexing session)

Summary