Post on 07-Jul-2015
description
Fun Teaching MongoDB Fun Teaching MongoDB New TricksNew Tricks
Robert J. MooreRobert J. MoorePresidentPresidentAllanbank Consulting, Inc.Allanbank Consulting, Inc.
Robert.J.Moore@allanbank.comRobert.J.Moore@allanbank.com
AgendaAgenda
InspirationInspiration
A Little Fun: MongoDB asA Little Fun: MongoDB as ... a Coordination Service?... a Coordination Service? ... a Message Broker?... a Message Broker?
https://github.com/allanbank/mongodb-trickshttps://github.com/allanbank/mongodb-tricks
DisclaimerDisclaimer
Don't try these at homeDon't try these at home They are hacks in the best senseThey are hacks in the best sense
Professional developer on an air gap Professional developer on an air gap laptoplaptop
Enough! On with the funEnough! On with the fun
Inspiration - TopicsInspiration - Topics
Publish / SubscribePublish / Subscribe One writer, many readersOne writer, many readers
MongoDB Capped CollectionMongoDB Capped Collection Tailable CursorTailable Cursor Can also have selectorsCan also have selectors
Inspiration - TopicsInspiration - Topics
...
Cursor
Document
Create a Capped CollectionCreate a Capped Collection Acts like a ring bufferActs like a ring buffer
Tailable Cursor returns documents as Tailable Cursor returns documents as they are added in the futurethey are added in the future
Asynchronous Java DriverAsynchronous Java Driver
WebsiteWebsite http://www.allanbank.com/mongodb-async-driverhttp://www.allanbank.com/mongodb-async-driver
Focus on performance and usabilityFocus on performance and usability Lower latency, higher throughput even under Lower latency, higher throughput even under
heavy thread contentionheavy thread contention Benefit when using the synchronous Benefit when using the synchronous
interfaceinterface Greater benefit using the asynchronous Greater benefit using the asynchronous
interfaceinterface
Asynchronous Java DriverAsynchronous Java DriverPerformancePerformance
http://www.allanbank.com/mongodb-async-driver/performance/ycsb.htmlhttp://www.allanbank.com/mongodb-async-driver/performance/ycsb.html
Coordination ServiceCoordination Service
ZooKeeperZooKeeper Notification that something of interest Notification that something of interest
changedchanged WatchesWatches
Track entities in a groupTrack entities in a group Group ManagementGroup Management
WatchesWatches
Replica Set OplogReplica Set Oplog All Insert, Update, Delete operationsAll Insert, Update, Delete operations Capped – Tailable cursors againCapped – Tailable cursors again
Want to convert from the OplogWant to convert from the Oplog Different fields contain the _id of the Different fields contain the _id of the
documentdocument
Oplog DocumentsOplog Documents InsertInsert
{ "ts" : { "t" : 1362958492000, "i" : 1 }, { "ts" : { "t" : 1362958492000, "i" : 1 },
"h" : NumberLong("5915409566571821368"), "v" : 2, "h" : NumberLong("5915409566571821368"), "v" : 2,
"op" : "i", "op" : "i",
"ns" : "test.test", "ns" : "test.test",
"o" : { "_id" : "513d189c8544eb2b5e000001" } }"o" : { "_id" : "513d189c8544eb2b5e000001" } }
DeleteDelete{ ... "op" : "d", ..., "b" : true, { ... "op" : "d", ..., "b" : true,
"o" : { "_id" : "513d189c8544eb2b5e000001" } }"o" : { "_id" : "513d189c8544eb2b5e000001" } }
UpdateUpdate{ ... "op" : "u", ..., { ... "op" : "u", ...,
"o2" : { "_id" : "513d189c8544eb2b5e000001" }, "o2" : { "_id" : "513d189c8544eb2b5e000001" },
"o" : { "$set" : { "i" : 1 } } }"o" : { "$set" : { "i" : 1 } } }
Initialize a Watcher - CodeInitialize a Watcher - Codefinal DocumentAssignable wantQuery = or( where("ns").equals(ns) .and("op").in(constant("i"), constant("d")) .and("o._id").matches(myContext), where("ns").equals(ns) .and("op").equals("u") .and("o2._id").matches(myContext));
final Find.Builder builder = new Find.Builder();if (myLastTs != null) { final DocumentBuilder tsQuery = BuilderFactory.start(); tsQuery.push(myLastTs.getName()) .add(myLastTs.withName(ComparisonOperator.GT.getToken()));
builder.setQuery(and(tsQuery, wantQuery));}else { builder.setQuery(wantQuery);}
builder.tailable();
MongoDatabase localDb = myMongoClient.getDatabase("local");MongoCollection oplog = localDb.getCollection("oplog.rs");myControls = oplog.streamingFind(new OpLogNotification(), builder.build());
Watcher - CodeWatcher - Code
Watcher watcher = new Watcher(client, collection, Pattern.compile(".*"), new WatchListener() { @Override public void changed(Operation op, String context, Document document) { if (op == Operation.DELETE) { System.out.println(context + ": " + op); } else { System.out.println(context + ": " + op + ": " + document); } } });watcher.start();
Watcher DemoWatcher Demo
Group ManagementGroup Management
Notification of:Notification of: Adding a Group MemberAdding a Group Member Removing a Group MemberRemoving a Group Member
Or Group Member DisappearsOr Group Member Disappears
Combination of:Combination of: WatcherWatcher Scheduled Task for HeartbeatScheduled Task for Heartbeat Expire old membersExpire old members
Group Manager Client - CodeGroup Manager Client - CodeGroupManager manager = new GroupManager(executor, client, collection, rootContext);manager.addListener(new GroupListener() { @Override public void memberRemoved(String context) { System.out.println(context + " - Removed"); }
@Override public void memberAdded(String context) { System.out.println(context + " - Added"); }});manager.start();
final GroupMember member = manager.addMember();
// Faster cleanup, if we can.Runtime.getRuntime().addShutdownHook(new Thread() { @Override public void run() { member.remove(); }});
Group DemoGroup Demo
QueuesQueues
Divide and ConquerDivide and Conquer One writer, one readerOne writer, one reader
MongoDB Capped CollectionMongoDB Capped Collection Want to “share” a CursorWant to “share” a Cursor
Restartable CursorsRestartable Cursors
What if we had multiple applications passing the What if we had multiple applications passing the same “getmore” requests?same “getmore” requests?
““getmore” message references a cursorgetmore” message references a cursor
We need a way to save and restore the cursorWe need a way to save and restore the cursor Lets call them Restartable Cursors, but they don't Lets call them Restartable Cursors, but they don't
exist, yetexist, yet
struct { MsgHeader header; // standard message header int32 ZERO; // 0 - reserved for future use cstring fullCollectionName; // "dbname.collectionname" int32 numberToReturn; // number of documents to return int64 cursorID; // cursorID from the OP_REPLY}http://docs.mongodb.org/meta-driver/latest/legacy/mongodb-wire-protocol/#op-get-more
QueuesQueues
...
Cursor
Document
Create a Capped CollectionCreate a Capped Collection Initialize a Tailable CursorInitialize a Tailable Cursor Share with multiple processesShare with multiple processes
Application 1 Application 3Application 2
Restartable CursorsRestartable Cursors
Added the ability to:Added the ability to: Persist an iterator or stream as a doumentPersist an iterator or stream as a doument Halt or stop the iterator stream (after Halt or stop the iterator stream (after
exhausting the already requested exhausting the already requested documents).documents).
For restart add the ability to restart the For restart add the ability to restart the iterator or streamiterator or stream
Allows conversion from an iterator to a Allows conversion from an iterator to a stream and back.stream and back.
Queues – Initialization CodeQueues – Initialization Code Initialize a Tailable CursorInitialize a Tailable Cursor Share with multiple processesShare with multiple processes
Find.Builder builder = new Find.Builder(BuilderFactory.start()); builder.setTailable(true); builder.setAwaitData(false); // Sigh. MongoIterator<Document> cursor = collection.find(builder.build());
// Graceful shutdown of the iterator locally but not on the server. cursor.stop(); while (cursor.hasNext()) { System.out.println(cursor.next()); }
collection = db.getCollection("lookup"); collection.delete(BuilderFactory.start().add("_id", collectionName)); collection.insert(BuilderFactory.start().add("_id", collectionName) .add("cursor", cursor.asDocument()));
Queues – Consumer CodeQueues – Consumer Code
MongoCollection index = client.getDatabase(args[1]).getCollection( "lookup"); Document queueLookupDoc = index.findOne(BuilderFactory.start().add( "_id", args[2])); DocumentElement cursorElement = queueLookupDoc.get( DocumentElement.class, "cursor");
MongoIterator<Document> iter = client.restart(cursorElement .getDocument());
long lastCount = 0; while (iter.hasNext()) { Document doc = iter.next(); // Do stuff }
Queues DemoQueues Demo
Restartable Cursors - GotchaRestartable Cursors - Gotcha Remember this: Remember this: builder.setAwaitData(false); // Sigh.builder.setAwaitData(false); // Sigh. That masks a small issue...That masks a small issue...
Closes the cursor on the clientCloses the cursor on the client https://jira.mongodb.org/browse/SERVER-8602https://jira.mongodb.org/browse/SERVER-8602 https://github.com/allanbank/mongo/tree/concurrent_cursor_supporthttps://github.com/allanbank/mongo/tree/concurrent_cursor_support
Pin( long long cursorid ) : _cursorid( INVALID_CURSOR_ID ) { recursive_scoped_lock lock( ccmutex ); ClientCursor *cursor = ClientCursor::find_inlock( cursorid, true ); if ( cursor ) { uassert( 12051, "clientcursor already in use? driver problem?", cursor->_pinValue < 100 ); cursor->_pinValue += 100; _cursorid = cursorid; }}
Final ThoughtsFinal Thoughts
Coordination Service Coordination Service WatchersWatchers Group ManagementGroup Management
Message BrokerMessage Broker TopicsTopics QueuesQueues
Remember the DisclaimerRemember the Disclaimer
Questions?Questions?
Contact Information:Contact Information:Robert.J.Moore@allanbank.comRobert.J.Moore@allanbank.com