Finding the right NoSQL DB for the job

The path to a non-RDBMS solution at

Who we are

• A search engine• A people

search engine• An influencer

search engine• Subscription-

George Stathis

VP Engineering14+ years of experience building full-stack web software systems with a past focus on e-commerce and publishing. Currently responsible for building engineering capability to enable Traackr's growth goals.

What’s this talk about?

• Why we picked a NoSQL database

• How we picked a NoSQL database

• My NoSQL does not do the job! What now?!

• Nirvana = the right tool for the job

Why did we pick a NoSQL DB?

There are some misconceptions around NoSQL only being appropriate when one needs to achieve

“Web Scale”

I need web scale!http://www.youtube.com/watch?v=b2F-DItXtZs

Traackr picked NoSQL; are we “Web Scale”?

• In terms of users/traffic?

Do we fit the “Web scale” profile?

Source: compete.com

Source: highscalability.com

• In terms of the amount of data?

PRIMARY> use traackrswitched to db traackrPRIMARY> db.stats(){

"db" : "traackr","collections" : 12,"objects" : 68226121,"avgObjSize" : 2972.0800625760330,"dataSize" : 202773493971,"storageSize" : 221491429671,"numExtents" : 199,"indexes" : 33,"indexSize" : 27472394891,"fileSize" : 266623699968,"nsSizeMB" : 16,"ok" : 1

That’s a quarter of a terabyte …

Wait! What? My Synology NAS at home can hold 2TB!

No need for us to track the entire web

Web Content

Influencer Content

Not at scale :-)

Alternate view of “Web Scale”

Web data is:

Heterogeneous

Unstructured (text)

Source: http://www.opte.org/

Visualization of the Internet, Nov. 23rd 2003

Data sources are

isolated islands of rich

data with lose links to

one another

How do we build a database that models all possible entities found on the web?

Modeling the web: the RDBMS way

Source: socialbutterflyclt.com

{ "realName": "David Chancogne", "title": "CTO", "description": "Web. Geek.\r\nTraackr: http://traackr.com\r\nPropz: http://propz.me", "primaryAffiliation": "Traackr", "email": "dchancogne@traackr.com", "location": "Cambridge, MA, United States", "siteReferences": [ { "siteUrl": "http://twitter.com/dchancogne", "metrics": [ { "value": 216, "name": "twitter_followers_count" }, { "value": 2107, "name": "twitter_statuses_count" } ] }, { "siteUrl": "http://traackr.com/blog/author/david", "metrics": [ { "value": 21, "name": "google_inbound_links" } ] } ]}

Influencer data as JSON

“In the old world of data analysis you knew exactly which questions you wanted to ask,

which drove a very predictable collection and storage model. In the new world of data

analysis your questions are going to evolve and change over time and as such you need to be able to collect, store and analyze data without

being constrained by resources.”— Werner Vogels, CTO/VP Amazon.com

NoSQL = schema flexibility

• In terms of the variety of the data

Traackr’s Datastore Requirements

• Schema flexibility

• Good at storing lots of variable length text

• Batch processing options

Requirement: text storage

Variable text length:

< big variance <140

character tweets

multi-page

blog posts

RDBMS’ answer to variable text length:

Plan ahead for largest value

CLOB/BLOB

Issues with CLOB/BLOG for us:

No clue what largest value is

CLOB/BLOB for tweets = wasted space

NoSQL solutions are great for text:

No length requirements (automated

chunking)

Limited space overhead

Requirement: batch processing

Some NoSQL

solutions come

with MapReduce

Source: http://code.google.com/

Requirement: batch processing

MapReduce + RDBMS:

Possible but proprietary solutions

Usually involves exporting data from

RDBMS into a NoSQL system anyway.

Defeats data locality benefit of MR

A NoSQL option is the right fit

How did we pick a NoSQL DB?

Bewildering number of optionsKey/Value Databases• Distributed hashtables• Designed for high load• In-memory or on-disk• Eventually consistent

Column Databases• Spread sheet like• Key is a row id• Attributes are columns• Columns can be grouped

into families

Document Databases• Like Key/Value• Value = Document• Document = JSON/BSON• JSON = Flexible Schema

Graph Databases• Graph Theory G=(E,V)• Great for modeling

networks• Great for graph-based

query algorithms

Bewildering number of optionsKey/Value Databases• Distributed hashtables• Designed for high load• In-memory or on-disk• Eventually consistent

into families

query algorithms

Trimming optionsKey/Value Databases• Distributed hashtables• Designed for high load• In-memory or on-disk• Eventually consistent

into families

query algorithms

Graph Databases: while we can model our domain as a graph we don’t want to pigeonhole ourselves into this structure.

We’d rather use these tools for specialized data analysis but not as the

main data store.

into families

query algorithms

Memcache: memory-based,we need true persistence

into families

query algorithms

Amazon SimpleDB: not willing to store our data in a proprietary

datastore.

into families

query algorithms

Not willing to store our data in a proprietary datastore.

Redis and LinkedIn’s Project Voldermort: no query filters,

better used as queues or distributed caches

into families

query algorithms

CouchDB: no ad-hoc queries; maturity in early 2010 made us shy away although we did try

early prototypes.

into families

query algorithms

Cassandra: in early 2010, maturity questions, no secondary indexes and no batch processing options

(came later on).

into families

query algorithms

MongoDB: in early 2010, maturity questions, adoption questions

and no batch processing options.

into families

query algorithms

Riak: very close but in early 2010, we had adoption questions.

into families

query algorithms

HBase: came across as the most mature at the time, with several deployments, a

healthy community, "out-of-the box" secondary indexes through a contrib and

support for batch processing using Hadoop/MR .

Climbing the learning curve

When Big-Data = Big Architectures

Source: http://www.larsgeorge.com/2009/10/hbase-architecture-101-storage.html

Must have a Hadoop HDFS cluster of at least 2x replication

factor nodes

Must have an odd number of

Zookeeper quorum nodes

Then you can run your Hbase nodes but it’s recommended to

co-locate regionservers with hadoop datanodes so you have

to manage resources.

Master/slave architecture means a single point of failure,

so you need to protect your master.

And then we also have to manage the MapReduce

processes and resources in the Hadoop layer.

Jokes aside, no one said open source was easy to use

To be expected

• Hadoop/Hbase are

designed to move

mountains

• If you want to move big

stuff, be prepared to

sometimes use big

equipment

What it means to a startup

Development capacity before

Development capacity after

Congrats, you are now a sysadmin…

Whatever, we can do it!

Source: http://knowyourmeme.com/memes/honey-badger

Mapping an A-List to a column store

Ranks References to influencer records

Unique key

“attributes” column family

for general attributes

“influencerId” column familyfor influencer ranks and foreign keys

Qualifiers (basically attribute names)

“name” attribute

Influencer ranks can be attribute names as well

Alist name value

Influencer id values assigned to each rank (basically foreign keys to an influencer table)

Can get pretty long so needs indexing and pagination

Problem: no out-of-the-box row-based indexing and pagination

Whatever, it’s open-source!

Jumping right into the code

MapReduce for batch scoring

• Need to re-score our influencer

database once a week

• M/R cranked through it in 15 mins

Source: http://www.charliesheentshirts.info/

a few months later…

Need to upgrade to Hbase 0.90

• Making sure to remain on recent code base

• Performance improvements

• Mostly to get the latest bug fixes

No thanks!

Looks like something is missing

Our DB indexes depend on this!

Let’s get this straight

• Hbase no longer comes with secondary

indexing out-of-the-box

• It’s been moved out of the trunk to GitHub

• Where only one other company besides us

seems to care about it

Only one other maintainer besides us

Development capacity

Congrats, you are now an hbase maintainer…

Whatever, we’ll roll our own indexing!

Homegrown Hbase Indexes

Rows have id prefixes that can be efficiently scanned using STARTROW and STOPROW filters

Row ids for Posts

Find posts for influencer_id_1234

Row ids for Posts

Find posts for influencer_id_5678

Row ids for Posts

• No longer depending on

unmaintained code

• Work with out-of-the-box Hbase

installation

You are back but you still need to

maintain indexing logic

Application layer indexes are slow and brittle. The DB should be doing this, not us.

Sort of…

a few months later…

Cracks in the data modelhuffingtonpost.com

huffingtonpost.com

http://www.huffingtonpost.com/arianna-huffington/post_1.htmlhttp://www.huffingtonpost.com/arianna-huffington/post_2.html

http://www.huffingtonpost.com/arianna-huffington/post_3.html

http://www.huffingtonpost.com/shaun-donovan/post1.htmlhttp://www.huffingtonpost.com/shaun-donovan/post2.html

http://www.huffingtonpost.com/shaun-donovan/post3.html

writes for

authored by

published under

writes for

authored by

published under

huffingtonpost.com

writes for

authored by

published under

writes for

authored by

published under

Denormalized/duplicated for fast runtime access

and storage of influencer-to-site relationship

properties

huffingtonpost.com

writes for

authored by

published under

writes for

authored by

published under

Content attribution logic could sometimes mis-attribute posts because of the

duplicated data.

huffingtonpost.com

writes for

authored by

published under

writes for

authored by

published under

Exacerbated when we started tracking people’s content on a daily basis in mid-

Fixing the cracks in the data model

huffingtonpost.com

writes for

authored by

published under

writes for

authored by

published under

huffingtonpost.com

writes for

authored by

published under

writes for

authored by

published under

Normalize the sites

• Normalization requires stronger

secondary indexing

• Our application layer indexing would

need revisiting…again!

Psych! You are back to writing indexing

Whatever, we’ll change our NoSQL!

Traackr’s Datastore Requirements (Revisited)

• Batch processing options (maybe)

• Out-of-the-box SECONDARY INDEX support!

• Simple to use and administer

NoSQL picking – Round 2Key/Value Databases• Distributed hashtables• Designed for high load• In-memory or on-disk• Eventually consistent

into families

query algorithms

into families

query algorithms

into families

query algorithms

Graph Databases: we looked at Neo4J a bit closer but passed again

for the same reasons as before.

into families

query algorithms

Memcache: still no

into families

query algorithms

Amazon SimpleDB: still no.

into families

query algorithms

Not willing to store our data in a proprietary datastore.

Redis and LinkedIn’s Project Voldermort: still no

into families

query algorithms

CouchDB: more mature but still no ad-hoc queries.

into families

query algorithms

Cassandra: matured quite a bit, added secondary indexes and batch processing

options but more restrictive in its’ use than other solutions. After the Hbase lesson,

simplicity of use was now more important.

into families

query algorithms

Riak: strong contender still but adoption questions remained.

into families

query algorithms

MongoDB: matured by leaps and bounds, increased adoption, support from 10gen, advanced indexing out-of-the-box as well as some batch processing

options, breeze to use, well documented and fit into our existing code base very nicely.

Immediate Benefits

• No more maintaining custom application-layer

secondary indexing code

Yay! I’m back!

Immediate Benefits

• Single binary installation greatly simplifies

administration

Honestly, I thought I’d never see you

guys again!

Immediate Benefits

• Single binary installation greatly simplifies

administration

• Our NoSQL could now support our domain

many-to-many relationship

Modeling an influencer

Embedded list of references to sites augmented with

influencer-specific site attributes (e.g.

percent contribution to content)

{ ”_id": "770cf5c54492344ad5e45fb791ae5d52”, "realName": "David Chancogne", "title": "CTO", "description": "Web. Geek.\r\nTraackr: http://traackr.com\r\nPropz: http://propz.me", "primaryAffiliation": "Traackr", "email": "dchancogne@traackr.com", "location": "Cambridge, MA, United States", "siteReferences": [ { "siteId": "b31236da306270dc2b5db34e943af88d", "contribution": 0.25 }, { "siteId": "602dc370945d3b3480fff4f2a541227c", "contribution": 1.0 } ]}

Modeling an influencer

siteId indexed for “find influencers

connected to site X”

> db.influencers.ensureIndex({siteReferences.siteId: 1});> db.influencers.find({siteReferences.siteId: "602dc370945d3b3480fff4f2a541227c"});

{ ”_id": "770cf5c54492344ad5e45fb791ae5d52”, "realName": "David Chancogne", "title": "CTO", "description": "Web. Geek.\r\nTraackr: http://traackr.com\r\nPropz: http://propz.me", "primaryAffiliation": "Traackr", "email": "dchancogne@traackr.com", "location": "Cambridge, MA, United States", "siteReferences": [ { "siteId": "b31236da306270dc2b5db34e943af88d", "contribution": 0.25 }, { "siteId": "602dc370945d3b3480fff4f2a541227c", "contribution": 1.0 } ]}

Embedded list of influencer references

augmented with “usernames” (useful

for content attribution)

{ ”_id": "0001e86f73cc3975a29e6a98a41a4280”, ”url": "http://traackr.com/blog/", "metrics": [ { "name": "google_inbound_links", "value": 5432 } ], "authors": [ { "username": "dchancogne", "influencerId": "770cf5c54492344ad5e45fb791ae5d52" }, { "username": ”gstathis", "influencerId": "0001e86f73cc3975a29e6a98a41a4280" } ]}

Modeling a site

Indexed for “find sites associated to

influencer X”

> db.sites.ensureIndex({authors.influencerId: 1});> db.sites.find({authors.influencerId: "0001e86f73cc3975a29e6a98a41a4280"});

{ ”_id": "0001e86f73cc3975a29e6a98a41a4280”, ”url": "http://traackr.com/blog/", "metrics": [ { "name": "google_inbound_links", "value": 5432 } ], "authors": [ { "username": "dchancogne", "influencerId": "770cf5c54492344ad5e45fb791ae5d52" }, { "username": ”gstathis", "influencerId": "0001e86f73cc3975a29e6a98a41a4280" } ]}

Other index uses

Support for alternate site URLs (a.k.a. URL aliases):{ "_id": "0001e86f73cc3975a29e6a98a41a4280", "url_hash_list": [ { "url": "http://traackr.com/blog", "hash": "770cf5c54492344ad5e45fb791ae5d52" }, { "url": "http://blog.traackr.com/", "hash": "0001e86f73cc3975a29e6a98a41a4280" } ]}

Indexed for “find sites associated to

influencer X”

Index on MD5 hash of URL

Other Benefits

• Ad hoc queries and reports became easier to write with JavaScript:

no need for a Java developer to write map reduce code to extract

the data in a usable form like it was needed with Hbase.

Ad hoc report example// File Name: retweetTotal.js// Purpose: report the count of twitter URLs for which we have// computed the the number of total retweetsprint( "NUMBER OF TWITTER URLS where retweetTotal IS SET:" );print( db.sites.find( { platformName: "twitter.com", retweetTotal: { $exists: true } } ).count() );

• Easy to execute JS report script remotely:

> mongo <hostname>:<port>/traackr --quiet retweetTotal.js

• Run as a cron job, pipe the output to a file and email it out• Also, more complex MR-based reports are easily accessible to

someone with some JavaScript knowledge

Other Benefits (cont.)

• Ad hoc queries and reports became easier to write with JavaScript:

no need for a Java developer to write map reduce code to extract

the data in a usable form like it was needed with Hbase.

• Simpler backups: Hbase mostly relied on HDFS redundancy; intra-

cluster replication is available but experimental and a lot more

involved to setup.

Same binary can be deployed several times for replication & backups

Different Availability Zones for better SPOF

tolerance

priority 0 for backup server so that it never

gets elected as primary

Using xfs_freeze before taking backups

EBS snapshots as backups are portable to new instances (e.g.

Other Benefits (cont.)

• Ad hoc queries and reports became easier to write with

JavaScript: no need for a Java developer to write map reduce code

to extract the data in a usable form like it was needed with Hbase.

• Simpler backups: Hbase mostly relied on HDFS redundancy; intra-

cluster replication is available but experimental and a lot more

involved to setup.

• Great documentation

• Great adoption and community

Mongo cursors for batch scoring

• Mongo is fast enough for our data size to

be able to serially score the DB faster

than the MapReduce jobs did in parallel.

• When we grow larger, MapReduce is still

available as an option

looks like we found the right fit!

We have more of this

And less of this

for now…

Additional takeaways

• Fearless refactoring

• Ease of use and administration cannot be

overstated for a small startup

Finding the right NoSQL DB for the job - The path to a non-RDBMS solution at Traackr

Transcript of Finding the right NoSQL DB for the job - The path to a non-RDBMS solution at Traackr

Finding the right NoSQL DB for the job - The path to a non-RDBMS solution at Traackr

Documents

Transcript of Finding the right NoSQL DB for the job - The path to a non-RDBMS solution at Traackr

RDBMS vs NoSQL - IIT Kanpur vs NoSQL Relational Database Management Systems versus Big Data Management (NoSQL) Systems ARNAB BHATTACHARYA arnabb@cse.iitk.ac.in Department of Computer

AWS re:Invent 2016| DAT318 | Migrating from RDBMS to NoSQL: How Sony Moved from MySQL to Amazon DynamoDB

SQL vs NoSQL - York University characteristics of NoSQL DBMS Popular RDBMS like Oracle can be scaled up but it is difficult. ... SQL vs NoSQL The major ...Published in: Linux Journal

Transactions Returning to Big Data (NoSQL) OR · The rise of NoSQL & Big Data •Data explosion has caused re-evaluation of RDBMS •Initially RDBMS augmented with cache •But ultimately

CSE 530A Non-Relational Databases · Non-Relational Databases Washington University Fall 2013 . NoSQL • "NoSQL" was originally the name of a specific RDBMS project that did not

Notes From the Front Line: Hadoop, NoSql, RDBMS, Katta

DATASTAX ENTERPRISE - NoSQL are the Advantages of NoSQL Over an RDBMS? 9 ... This chapter introduces the basics of NoSQL and then dives into a DBA’s ... DataStax Enterprise Graph

NoSQL Or Peles. What is NoSQL A collection of various technologies meant to work around RDBMS limitations (mostly performance) Not much of a definition...

RDBMS vs NoSQL: Performance and Scaling Comparison MSc in ...

RDBMS & noSQL

RDBMS to NoSQL: Reviewing Some Next-Generation Non ...

NoSQL DBs. Positives of RDBMS Historical positives of RDBMS: – Can represent relationships in data – Easy to understand relational model/SQL – Disk oriented.

Migration Best Practices - RDBMS to DynamoDB (1) · systems (RDBMS), but also NoSQL databases, such as Amazon DynamoDB. Certain workloads will scale better and be more cost-effective

NoSQL Databases - Software Engineering Research · PDF file · 2012-08-20Chapter 1 NoSQL Databases Johannes Zollmann 1.1 Introduction ... tional database management system (RDBMS),

DBA’s Guide to NoSQL...5 DBA’s guide to NoSQL WHY NOSQL? The RDBMS has been the de-facto standard for managing data since it first appeared from IBM in the mid-1980s. The RDBMS

Traackr at Pecha Kucha 2009

State of Modern Applications in AWS Report · 2019. 3. 13. · 8 November 2016 NoSQL Leads RDBMS Database Adoption NoSQL 58% AWS Database Adoption RDBMS 42% Context • Databases

NoSQL Data Stores - ce.uniroma2.it · • RDBMS: strong consistency – Traditional RDBMS are CA systems (or CP systems, depending on the configuration) • NoSQL systems: mostly

NoSQL - UCR Computer Science and Engineering · NoSQL relaxes some of the restrictions of RDBMS to provide a better performance 4. Advantages of NoSQL Handles Big Data Data Models

EDW Data Model Storming for Integration of NoSQL and RDBMS by Daniel Upton