Post on 14-Apr-2017
The How and Why of Fast Data Analytics with Apache Spark
with Justin Pihony @JustinPihony
Today’s agenda:▪ Concerns▪ Why Spark?▪ Spark basics▪ Common pitfalls▪ We can help!
2
Target Audience
3
Concerns▪ Am I too small?
4
▪ Will switching be too costly?▪ Can I utilize my current infrastructure?▪ Will I be able to find developers?▪ Are there enough resources available?
Why Spark?
5
grep?
Why Spark?
6
object WordCount{
def main(args: Array[String])){
val conf = new SparkConf()
.setAppName("wordcount")
val sc = new SparkContext(conf)
sc.textFile(args(0))
.flatMap(_.split(" "))
.countByValue
.saveAsTextFile(args(1))
}}
7
public class WordCount { public static class Map extends Mapper<LongWritable, Text, Text, IntWritable> { private final static IntWritable one = new IntWritable(1); private Text word = new Text(); public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException { String line = value.toString(); StringTokenizer tokenizer = new StringTokenizer(line); while (tokenizer.hasMoreTokens()) { word.set(tokenizer.nextToken()); context.write(word, one); } } } public static class Reduce extends Reducer<Text, IntWritable, Text, IntWritable> { public void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException { int sum = 0; for (IntWritable val : values) { sum += val.get(); } context.write(key, new IntWritable(sum)); } } public static void main(String[] args) throws Exception { Configuration conf = new Configuration(); Job job = new Job(conf, "wordcount"); job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class); job.setMapperClass(Map.class); job.setReducerClass(Reduce.class); job.setInputFormatClass(TextInputFormat.class); job.setOutputFormatClass(TextOutputFormat.class); FileInputFormat.addInputPath(job, new Path(args[0])); FileOutputFormat.setOutputPath(job, new Path(args[1])); job.waitForCompletion(true); } }
Tiny CodeBig CodeWhy Spark?
Why Spark?
8
Readability
Expressiveness
Fast
Testability
Interactive
Fault Tolerant
Unify Big Data
9
The MapReduce Explosion
10
“Spark will kill MapReduce, but save Hadoop.”
- http://insidebigdata.com/2015/12/08/big-data-industry-predictions-2016/
Big Data Unified API
13
Spark Core
SparkSQL
SparkStreaming
MLlib(machinelearning)
GraphX(graph)
DataFrames
14
Yahoo!
Who Is Using Spark?
Spark Mechanics
15
Worker WorkerWorker
Driver
Spark Mechanics
16
Spark Context
Worker WorkerWorker
Driver
Spark Context
17
Task creatorSchedulerData localityFault tolerance
RDD
18
▪ Resilient Distributed Dataset▪ Transformations
-map- filter-…
▪ Actions-collect-count- reduce-…
Expressive and Interactive
19
Built-in UI
20
Common Pitfalls
▪ Functional▪ Out of memory▪ Debugging▪ …
21
Concerns▪ Am I too small?
22
▪ Will switching from MapReduce be too costly?▪ Can I utilize my current infrastructure?▪ Will I be able to find developers?▪ Are there enough resources available?
Q & A
23
EXPERT SUPPORT Why Contact Typesafe for Your Apache Spark Project?
Ignite your Spark project with 24/7 production SLA, unlimited expert support and on-site training:
• Full application lifecycle support for Spark Core, Spark SQL & Spark Streaming
• Deployment to Standalone, EC2, Mesos clusters • Expert support from dedicated Spark team • Optional 10-day “getting started” services
package
Typesafe is a partner with Databricks, Mesosphere and IBM.
Learn more about on-site trainingCONTACT US
©Typesafe 2016 – All Rights Reserved