Spark Streaming
Much easier than StormReplaces Storm spouts/bolts with Akka Actors
Better API(make time part of API) and integrationHadoop 2.3/Spark 0.9.1
Sbt setup
Create a separate sbt project; sbt run Includes the jars and sets the class path
Batch and Streaming, http://spark.apache.org/docs/latest/quick-start.html
Create a project directory Add dependencies; scalaized maven
libraryDependencies += "org.apache.hadoop" % "hadoop-client" % "2.3.0"
scalaVersion:="2.10.3"
Manage the sbt/scala versions locally
Maven setup
Run the demo using maven/eclipse Easier, maven central to find jars/artifacts Add the external libs using maven to local repo
and mvn package in spark source distro Eclipse: add Scala Nature, Maven project
Demo
Connect to twitter stream and process Test Twitter4j connection w/Java first. Print out a
twitter stream Batch Mode: sc.stop(); RealTime Streaming
stream.awaitTermination(). Dstream/scala lazy evaluation
Create a stream using #:: like the recursive List operator. (#iphone,1)#:(#andriod,3)#(#apple,10). Unlike a list head/tail behave differently. Head is a val.
Spark Streams
StreamingContext start scheduler JobScheduler.scala: starts JobGenerator and runs
them in a thread pool JobGenerator.scala: Starts event actor, checkpoint
writer, for each thread Storage:
DStream appends to blockgenerator BlockGenerator.scala: Spark BlockGenerator w/2
threads. On termination wait for blockpush thread to join.
Kafka Streaming Demo
KafkaUtils/Consumer connection IOItec connection lib Need to add more features/testing for faults Read source how to fill out params Start zookeeper, start a producer, define a
topic, etc...
Send data from the producer
Demo Output showing console producer to Spark Consumer
Producer/Executor
Match the broker-id in the server conf file with groupID in the consumer call
val kafkaInputs = (1 to 5).map { _ =>
KafkaUtils.createStream(stream,"localhost:2181", "1", Map("testtopic" -> 1))
Producer
Use awaitTermination() to get infinite loop so you can see what you enter into the producer; Start w/1 executor
val stream = new StreamingContext("local[2]","TestObject", Seconds(1)) val kafkaMessages=
KafkaUtils.createStream(stream,"localhost:2181","1",Map("testtopic"->1)) //create 5 executors val kafkaInputs = (1 to 5).map { _ => KafkaUtils.createStream(stream,"localhost:2181", "1", Map("testtopic" -> 1)) kafkaMessages.print() stream.start() stream.awaitTermination()
Top Related