Using NoSQL technologies for handling of the CMS Conditions

Roland Sipos for the CMS Collaboration

Forum on Concurrent Programming Models and Frameworks17 June 2015

Overview● Intro

○ CMS Conditions Database○ NoSQL

● Candidates○ Test framework○ Deployment

● Results● Outlook

2

Intro3

Conditions DatabaseAlignment and Calibration constants, that record a given “state” of the CMS Detector.

Essential for the analysis and reconstruction of the recorded data.

Also critical for the dataflow and need to be properly re-synchronized during the data processing.

4

CondDB - DetailsConditions are free from:● Full table scans

○ Only “by key” (or range of keys) access● Joins● Complex, nested queries● Transactions

○ Data is written once, and never deleted, altered● Absolute consistency

○ Only consistency criteria: newly appended data should be available for reads ASAP! (in less than few seconds)

5

CondDB - MotivationsFind alternative data storing technologies for the CMS Conditions data for:● Storing BLOBs● And it’s meta data● In a read-heavy environmentFurther requirements:● Durability● High availability● (Optional scalability)

Do we really need relational access for such use-case?6

Relational vs. Non-relational● Based on relational

model: relational algebra○ schema

● SQL: not necessarily but the most widespread

● Transactions○ ACID

● Well tested, "proven"

● Does not based on the relational model.○ schema free

● Query languages may differ:○ Datalog, XPath, etc.

● Unique operations (eg. CRUD)○ BASE

● Many, quite new solutions.○ beta phase versions, etc. 7

NoSQL - ACID vs. BASEACID

● Atomicity○ "all or nothing"

transactions● Consistency

○ data is always valid● Isolation

○ transactions are independent

● Durability○ permanent state

BASE● Basic Availability

○ it’s OK to give approximate answers

● Soft state○ easier (schema)

evolution● Eventual consistency

○ stored data achieves consistent state with time

8

NoSQL - CAP Theorem

Consistencyaka."All client see the same data at the same time!"

Availabilityaka."Every request got a response about success or failure!"

Partition toleranceEventualconsistency

RDBMSs

Enforced consistency(PAXOS)

You can choose only two by the following three attributes.

- Eric Brewer at 2000

10+ years already passed, many misleading/false information were born based on the theorem.

9

Partitioning - General● Distributed memory cache● Clustering

○ scaling the persistency layer● Separate operations (reads and writes)

○ dedicated master for write, group of slaves for read● Sharding - horizontal partitioning

○ the storage volume is distributed among many nodes

Scale up: "put more RAM or better processor to the server" Scale out: distribute data among computing elements

Scale out = partitioning10

Partitioning - Vertical

Splitting up stored data in one entity, into multiple ones. (It’s a bit more than normalization.)

“Different columns on different resources.”

E.g.: Frequently accessed columns in separate table, cached in-memory. Rarely used columns stored on disk.

11

Partitioning - Horizontal

One entity, however it’s data may be distributed on many storage elements.

“Different rows on different resources.”

E.g.: Year based distribution on different storage elements based on an “insertion time” constraint.

12

NoSQL - GeneralNoSQL in keywords:● Only a buzzword

○ Meaning: “One size does not fit all!”● CAP Theorem● ACID vs BASE● Different models

○ Doc. store, Key-Value, Column oriented, BigTable

NoSQL means: “we have options”!Not against relational DBs, but a complement to those!

13

NoSQL - Options

Options

Non-Relational Relational

Operational

Analytic

NewSQLNoSQL

Document

Key-value DaaS

Column oriented Graph

Oracle IBM DB2 JustOneDB MS SQL Server

HadoopCloudera Hadapt

Oracle TimesTen IBM InfosphereSAP (Hana, Sybase IQ) HP Vertica

SPARK

Lotus Notes

CouchDB MongoDB

MySQL PostgreSQL JustOneDB

ProgressObjectivity Versant

McObjectMarkLogic

SQL Azure RavenDB Amazon RDS

XeroundFathomDB NuoDB

Riak Redis Voldemort BerkleyDB

Cassandra Accumulo

BigTable HyperTable HBase

Neo4j

Couchbase SimpleDBApp Engine

Brand new RDBS Add-on

Clustrix VoltDB SnakeSQL

ScaleDBMySQL Cluster GenieDB TokutekDrizzle

Flat, Hierarchical, Network, etc...

source: Tim Gasper - Big Data Right Now: Five trendy open source technologies

14

Challenges 1.How to chose? Rule of thumb: Benchmarking!But an exact way for NoSQL benchmarking could not exist. (It's not like TPC-X for RDBMSs.) Even if we want to compare them, we must fight with different...

○ problems and possibilities,○ APIs,○ partitioning techniques, etc ...

15

Challenges 2.The main issue is that the design and preferred use-cases of the NoSQL databases are REALLY different. There is no "x better than y" argument for EVERY use-case.

Benchmarks are based on several use cases:● different computing elements,● compared by write/read/update/scan

operations. (mixed with a ratio: 90%/10%/0%)16

Prototypes - The candidates17

SelectionIn multiple phases...

Find:● Showstopper problems (no-go)● Barely usable (some issues)● Promising candidates

Preliminary testing.18

CandidatesNo-go

● HBase (/w HDFS)○ BLOB size problem.

● CouchDB○ Drivers

● Hypertable○ In development

● etc.: app layer needs, CAP characteristics, durability problems.

Promising● MongoDB

● Cassandra

So-so● RIAK

○ Query routing!● (Couchbase)

19

DeploymentAutomated virtual environments on OpenStack.

○ Personal tenant - biased by user interactions○ Thanks to the collaboration with CERN IT, the

evaluation was made on dedicated resources○ Also SSD cached vs. disk comparisons were made

Details:○ No overcommit○ Instances are “equally” distributed on the

hypervisors. (for 5 node: 2-2-1 on 3 hypervisors)○ 1 GBit NICs (shared between co-hosted VMs)

20

Evaluation

Empirical evaluation: Check if a given prototype meets the usability and performance criterias of the desired solution.

If more of them passes the criteria, choose the best, based on essential features and performance characteristics.

21

CustomSamplers 1.An extension for JMeter, with CMS specific needs, in order to measure the performance of different databases.For each candidate the extension has:● Deployers

○ To build up the data model● QueryHandlers

○ Simulate the CMS workflow● ConfigElements

○ Configure persistency objects● Samplers

○ Report to the testplan listeners22

CustomSamplers 2.Testplans are XML configurations that set up the behaviour of the testing engine by controlling:● number of threads● ramp up time of thread creation● configure connection layers● assign requests to threads

1 TPS : 1200 threads started in 1200 second, result in 20 min. constant stage tests. 23

ResultsIncreasing request numbers: 1-9 TPS(For both remote and single testplans)● Exploring limits for saturating factors like:

○ Network bandwidth○ Access of persistency objects○ Storage elements (Ephemeral disk/SSD, Ceph)

● Scaling out (different cluster setups):○ Node numbers (5 x m1.large, 4 x m1.medium) ○ Routing techniques (Round robin, Token-aware)○ Distributed testing (4 JMeter engine)

24

Single nodeNormal test - DISK

25

Single nodeRemote test - DISK

26

Single nodeNormal test - SSD

27

Single nodeRemote test - SSD

28

Medium clusterRemote test - DISK

29

Medium clusterRemote test - SSD

30

Large clusterRemote test - DISK

31

Large clusterRemote test - SSD

32

Ceph volume

33

Outro - Present and future34

Application layerThe current implementation of the session layer is highly modular and extendable with alternative storage backends. Steps:● Handling persistency objects

○ Extending the software framework with NoSQL support

● Implement the Session interfaces○ Implementing the “equivalent” CondDB queries

● Testing 35

Integration● Release validation● Find differences between the current

solution and the prototypes○ Using real data○ Real use-cases - using CMSSW

This will be the final performance comparison between different deployments.

36

Outlook● Understand and eliminate issues during the

release validation● Fine-tuning critical performance factors● Formal evaluation and comparison of the

different solutions

Long term project!Not a “by tomorrow” change, but for LS2.

37

The endThank you for your attention!

Any questions are welcome!

From: http://geek-and-poke.com/38