Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz...

91
Nathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Transcript of Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz...

Page 1: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Nathan MarzTwitter

Distributed and fault-tolerant realtime computation

Storm

Page 2: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Basic info

• Open sourced September 19th

• Implementation is 15,000 lines of code

• Used by over 25 companies

• >2400 watchers on Github (most watched JVM project)

• Very active mailing list

• >1800 messages

• >560 members

Page 3: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Before Storm

Queues Workers

Page 4: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Example

(simplified)

Page 5: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Example

Workers schemify tweets and append to Hadoop

Page 6: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Example

Workers update statistics on URLs by incrementing counters in Cassandra

Page 7: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

ScalingDeploy

Reconfigure/redeploy

Page 8: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Problems

• Scaling is painful

• Poor fault-tolerance

• Coding is tedious

Page 9: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

What we want

• Guaranteed data processing

• Horizontal scalability

• Fault-tolerance

• No intermediate message brokers!

• Higher level abstraction than message passing

• “Just works”

Page 10: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Storm

Guaranteed data processing

Horizontal scalability

Fault-tolerance

No intermediate message brokers!

Higher level abstraction than message passing

“Just works”

Page 11: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Streamprocessing

Continuouscomputation

DistributedRPC

Use cases

Page 12: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Storm Cluster

Page 13: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Storm Cluster

Master node (similar to Hadoop JobTracker)

Page 14: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Storm Cluster

Used for cluster coordination

Page 15: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Storm Cluster

Run worker processes

Page 16: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Starting a topology

Page 17: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Killing a topology

Page 18: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Concepts

• Streams

• Spouts

• Bolts

• Topologies

Page 19: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Streams

Unbounded sequence of tuples

Tuple Tuple Tuple Tuple Tuple Tuple Tuple

Page 20: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Spouts

Source of streams

Page 21: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Spout examples

• Read from Kestrel queue

• Read from Twitter streaming API

Page 22: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Bolts

Processes input streams and produces new streams

Page 23: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Bolts

• Functions

• Filters

• Aggregation

• Joins

• Talk to databases

Page 24: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Topology

Network of spouts and bolts

Page 25: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Tasks

Spouts and bolts execute as many tasks across the cluster

Page 26: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Task execution

Tasks are spread across the cluster

Page 27: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Task execution

Tasks are spread across the cluster

Page 28: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Stream grouping

When a tuple is emitted, which task does it go to?

Page 29: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Stream grouping

• Shuffle grouping: pick a random task

• Fields grouping: mod hashing on a subset of tuple fields

• All grouping: send to all tasks

• Global grouping: pick task with lowest id

Page 30: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Topology

shuffle

[“url”]

shuffle

shuffle

[“id1”, “id2”]

all

Page 31: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Streaming word count

TopologyBuilder is used to construct topologies in Java

Page 32: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Streaming word count

Define a spout in the topology with parallelism of 5 tasks

Page 33: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Streaming word count

Split sentences into words with parallelism of 8 tasks

Page 34: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Consumer decides what data it receives and how it gets grouped

Streaming word count

Split sentences into words with parallelism of 8 tasks

Page 35: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Streaming word count

Create a word count stream

Page 36: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Streaming word count

splitsentence.py

Page 37: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Streaming word count

Page 38: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Streaming word count

Submitting topology to a cluster

Page 39: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Streaming word count

Running topology in local mode

Page 40: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Demo

Page 41: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Distributed RPC

Data flow for Distributed RPC

Page 42: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

DRPC Example

Computing “reach” of a URL on the fly

Page 43: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Reach

Reach is the number of unique peopleexposed to a URL on Twitter

Page 44: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Computing reach

URL

Tweeter

Tweeter

Tweeter

Follower

Follower

Follower

Follower

Follower

Follower

Distinct follower

Distinct follower

Distinct follower

Count Reach

Page 45: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Reach topology

Page 46: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Reach topology

Page 47: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Reach topology

Page 48: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Reach topology

Keep set of followers foreach request id in memory

Page 49: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Reach topology

Update followers set whenreceive a new follower

Page 50: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Reach topology

Emit partial count afterreceiving all followers for a

request id

Page 51: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Demo

Page 52: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Guaranteeing message processing

“Tuple tree”

Page 53: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Guaranteeing message processing

• A spout tuple is not fully processed until all tuples in the tree have been completed

Page 54: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Guaranteeing message processing

• If the tuple tree is not completed within a specified timeout, the spout tuple is replayed

Page 55: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Guaranteeing message processing

Reliability API

Page 56: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Guaranteeing message processing

“Anchoring” creates a new edge in the tuple tree

Page 57: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Guaranteeing message processing

Marks a single node in the tree as complete

Page 58: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Guaranteeing message processing

• Storm tracks tuple trees for you in an extremely efficient way

Page 59: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Transactional topologies

How do you do idempotent counting with anat least once delivery guarantee?

Page 60: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Won’t you overcount?

Transactional topologies

Page 61: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Transactional topologies solve this problem

Transactional topologies

Page 62: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Built completely on top of Storm’s primitivesof streams, spouts, and bolts

Transactional topologies

Page 63: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Batch 1 Batch 2 Batch 3

Transactional topologies

Process small batches of tuples

Page 64: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Batch 1 Batch 2 Batch 3

Transactional topologies

If a batch fails, replay the whole batch

Page 65: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Batch 1 Batch 2 Batch 3

Transactional topologies

Once a batch is completed, commit the batch

Page 66: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Batch 1 Batch 2 Batch 3

Transactional topologies

Bolts can optionally be “committers”

Page 67: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Commit 1

Transactional topologies

Commits are ordered. If there’s a failure during commit, the whole batch + commit is retried

Commit 1 Commit 2 Commit 3 Commit 4 Commit 4

Page 68: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Example

Page 69: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Example

New instance of this objectfor every transaction attempt

Page 70: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Example

Aggregate the count forthis batch

Page 71: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Example

Only update database if transaction ids differ

Page 72: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Example

This enables idempotency sincecommits are ordered

Page 73: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Example

(Credit goes to Kafka devs for this trick)

Page 74: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Transactional topologies

Multiple batches can be processed in parallel, but commits are guaranteed to be ordered

Page 75: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

• Will be available in next version of Storm (0.7.0)

• Requires a source queue that can replay identical batches of messages

• storm-kafka has a transactional spout implementation for Kafka

Transactional topologies

Page 76: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Storm UI

Page 77: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Storm on EC2

https://github.com/nathanmarz/storm-deploy

One-click deploy tool

Page 78: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Starter code

https://github.com/nathanmarz/storm-starter

Example topologies

Page 79: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Documentation

Page 80: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Ecosystem

• Scala, JRuby, and Clojure DSL’s

• Kestrel, AMQP, JMS, and other spout adapters

• Serializers

• Multilang adapters

• Cassandra, MongoDB integration

Page 81: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Questions?

http://github.com/nathanmarz/storm

Page 82: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Future work

• State spout

• Storm on Mesos

• “Swapping”

• Auto-scaling

• Higher level abstractions

Page 83: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Implementation

KafkaTransactionalSpout

Page 84: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Implementation

all

all

all

Page 85: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Implementation

all

all

all

TransactionalSpout is a subtopology consisting of a spout and a bolt

Page 86: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Implementation

all

all

all

The spout consists of one task that coordinates the transactions

Page 87: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Implementation

all

all

all

The bolt emits the batches of tuples

Page 88: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Implementation

all

all

all

The coordinator emits a “batch” streamand a “commit stream”

Page 89: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Implementation

all

all

all

Batch stream

Page 90: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Implementation

all

all

all

Commit stream

Page 91: Storm - University of California, Berkeleycloud.berkeley.edu/data/storm-berkeley.pdfNathan Marz Twitter Distributed and fault-tolerant realtime computation Storm

Implementation

all

all

all

Coordinator reuses tuple tree framework to detectsuccess or failure of batches or commits and replays

appropriately