Spark with Couchbase to Electrify Your Data Processing – Couchbase Live New York 2015

Couchbase & Sparkefficient data crunching in a fast moving world

©2015 Couchbase Inc. 2

The Big Picture


Apache Spark… is a fast and general purpose engine for small and large scale data processing …


Components: Spark Core

Resilient Distributed DatasetsClusteringExecution


Components: Spark SQL

Structured through DataFrames

Distributed querying with SQL


Components: Spark Streaming

Fault-tolerant streaming applications


Components: Spark MLib

Built-In Machine Learning Algorithms


Components: Spark GraphX

Graph processing and graph-parallel computations


How does it work?

Source: http://spark.apache.org/docs/latest/cluster-overview.html


Spark Benefits

Linearly scalable to 1000+ worker nodes Simpler to use than Hadoop MR Only partial recompute on failure

For developers and data scientists– machine learning– R integration

Tight but not mandatory Hadoop integration– Sources, Sinks– Scheduler


Spark vs Hadoop

Spark is RAM while Hadoop is mainly HDFS (disk) bound

Fully compatible with Hadoop Input/Output

Easier to develop against thanks to functional composition

Hadoop certainly more mature, but Spark ecosystem growing fast


Couchbase in the Spark Landscape Transparent generation and persistence of

– RDDs– DataFrames– Dstreams

Spark SQL and N1QL are a natural fit Linearly scale your data and application layer Share data between Spark Applications

The perfect storage companion for your spark applications.

Source: http://spark.apache.org/docs/latest/cluster-overview.html


Cluster Communication

STORAGE

Couchbase Server 1

SHARD7

SHARD9

SHARD5

SHARDSHARDSHARD

Managed Cache

Cluster ManagerCluster

Manager

Managed Cache

Storage

Data Service

Index Service

Query Service STORAGE

Couchbase Server 2

SHARD7

SHARD9

SHARD5

SHARDSHARDSHARD

Managed Cache


Manager

Managed Cache

Storage

Data Service

Index Service


Couchbase Server 3

SHARD7

SHARD9

SHARD5

SHARDSHARDSHARD

Managed Cache


Manager

Managed Cache

Storage

Data Service

Index Service


Couchbase Server 4

SHARD7

SHARD9

SHARD5

SHARDSHARDSHARD

Managed Cache


Manager

Managed Cache

Storage

Data Service

Index Service


Couchbase Server 5

SHARD7

SHARD9

SHARD5

SHARDSHARDSHARD

Managed Cache


Manager

Managed Cache

Storage

Data Service

Index Service


Couchbase Server 6

SHARD7

SHARD9

SHARD5

SHARDSHARDSHARD

Managed Cache


Manager

Managed Cache

Storage

Data Service

Index Service

Query Service

Spark Worker Spark Worker


Ecosystem Flexibility

RDBMS

StreamsWeb APIs

DCPKVN1QLViews

BatchingData Archive

OLTP Data


Infrastructure Consolidation


The Connector


Couchbase Connector Spark Core

– Automatic Cluster and Resource Management– Creating and Persisting RDDs– Java APIs in addition to Scala

Spark SQL– Easy JSON handling and querying– Tight N1QL Integration

Spark Streaming– Persisting DStreams– DCP source (experimental)


Facts Current Version: 1.0.0-beta

Code: https://github.com/couchbaselabs/couchbase-spark-connector

Docs until GA: http://developer.couchbase.com/documentation/server/4.0/connectors/spark-1.0/spark-intro.html

https://github.com/couchbaselabs/couchbase-spark-connector

https://github.com/couchbaselabs/couchbase-spark-connector

http://developer.couchbase.com/documentation/server/4.0/connectors/spark-1.0/spark-intro.html




Connection Management


Creating RDDs


Persisting RDDs


Spark SQL Integration


Spark Streaming with DCP


What‘s next?


Couchbase Connector 1.1.0 plans

– Upgrade to Spark 1.5– Stabilize DCP Support– Extend, Optimze, Fix bugs…

We need your feedback!

Spark with Couchbase to Electrify Your Data Processing – Couchbase Live New York 2015

Software

Transcript of Spark with Couchbase to Electrify Your Data Processing – Couchbase Live New York 2015