From MapReduce to Apache Spark

5
From MapReduce to Apache Spark: An Ecosystem Evolves Doug Cutting (@cutting) Chief Architect / Apache Hadoop Co-founder

Transcript of From MapReduce to Apache Spark

Page 1: From MapReduce to Apache Spark

1©Cloudera,Inc.Allrightsreserved.

FromMapReduce toApacheSpark:AnEcosystemEvolvesDougCutting(@cutting)ChiefArchitect/ApacheHadoopCo-founder

Page 2: From MapReduce to Apache Spark

2©Cloudera,Inc.Allrightsreserved.

TheMergingEcosystems

Page 3: From MapReduce to Apache Spark

3©Cloudera,Inc.Allrightsreserved.

Genomics(Broad Institute)

Physics(CERN)

Healthcare Delivery(CHOA)

TheNewUseCasesareAmazing

Page 4: From MapReduce to Apache Spark

4©Cloudera,Inc.Allrightsreserved.

BringYourQuestions AboutLivytotheClouderaBooth

Livy.io

• ForsubmittingSparkjobsfromanyweb/mobileapp,withnoSparkclientinstall — enablingnewarchitecturesandusecases

• Providesmulti-tenancyandfaulttolerancetosupportmultipleusersreliably

• Works instandalonemode,withYARN,orwithMesos

• Nocodechangesneeded

• ApacheLicense.Initialcontributors includeemployeesofCloudera,Intel, andMicrosoft+morewanted!

NewContribution:Livy,anOpenSourceRESTServiceforSpark(Alpha)

Page 5: From MapReduce to Apache Spark

5©Cloudera,Inc.Allrightsreserved.

Thankyou@cutting

cloudera.com/spark


Developing Apache Spark Applications - Cloudera · Apache Spark Quick Start Apache Spark Overview Apache Spark Programming Guide Using the Spark DataFrame API A DataFrame is a distributed

Developing Apache Spark Applications - Cloudera · Apache Spark Quick Start Apache Spark Overview Apache Spark Programming Guide Using the Spark DataFrame API A DataFrame is a distributed

Spark vs. MapReduce

Spark vs. MapReduce

Unified Big Data Processing with Apache Spark · Unified Big Data Processing with Apache Spark ... MapReduce was great for batch processing, ... Spark can leverage most of the latest

Unified Big Data Processing with Apache Spark · Unified Big Data Processing with Apache Spark ... MapReduce was great for batch processing, ... Spark can leverage most of the latest

Online Course Syllabus Template · Storing & Retrieving data with Apache Hadoop HDFS, MapReduce, and Hive Storing & Retrieving data with Apache Hadoop HDFS and Spark Data lakes: A

Online Course Syllabus Template · Storing & Retrieving data with Apache Hadoop HDFS, MapReduce, and Hive Storing & Retrieving data with Apache Hadoop HDFS and Spark Data lakes: A

Overview of MapReduce and Spark

Overview of MapReduce and Spark

Budapest Spark Meetup - Apache Spark @enbrite.ly

Budapest Spark Meetup - Apache Spark @enbrite.ly

MapReduce - Cornell Universitymajor performance degradations. MapReduce vs Spark. Perspectives • MapReduce, databases, Spark, all differ: • In engineering needs for a particular

MapReduce - Cornell Universitymajor performance degradations. MapReduce vs Spark. Perspectives • MapReduce, databases, Spark, all differ: • In engineering needs for a particular

Introduction to Apache Spark - tropars.github.io · I Hadoop MapReduce, Apache Spark, Apache Flink, etc 25. Agenda Computing at large scale Programming distributed systems MapReduce

Introduction to Apache Spark - tropars.github.io · I Hadoop MapReduce, Apache Spark, Apache Flink, etc 25. Agenda Computing at large scale Programming distributed systems MapReduce

Big Data Analysis with Apache - Broad Institute · PDF fileOutline The big data problem MapReduce Apache Spark How people are using it

Big Data Analysis with Apache - Broad Institute · PDF fileOutline The big data problem MapReduce Apache Spark How people are using it

MapReduce Programming With Apache Hadoop

MapReduce Programming With Apache Hadoop

Managed Solutions Apache Spark® · Apache Spark® Apache Spark™ is a high performing engine for large-scale analytics and data processing, While Apache Spark™ provides advanced

Managed Solutions Apache Spark® · Apache Spark® Apache Spark™ is a high performing engine for large-scale analytics and data processing, While Apache Spark™ provides advanced

Comparing Spark with MapReduce - Glowworm Swarm ...cs.ndsu.edu/~siludwig/Publish/papers/IGI2018.pdfapplications that companies such as Facebook work with (Apache Spark @Scale, 2017).

Comparing Spark with MapReduce - Glowworm Swarm ...cs.ndsu.edu/~siludwig/Publish/papers/IGI2018.pdfapplications that companies such as Facebook work with (Apache Spark @Scale, 2017).

Integrating Apache Hive with Kafka, Spark, and BI...Community Connection: Integrating Apache Hive with Apache Spark--Hive Warehouse Connector Apache Spark-Apache Hive connection configuration

Integrating Apache Hive with Kafka, Spark, and BI...Community Connection: Integrating Apache Hive with Apache Spark--Hive Warehouse Connector Apache Spark-Apache Hive connection configuration

Developing Apache Spark Applications · Apache Spark Introduction Introduction Apache Spark enables you to quickly develop applications and process jobs. Apache Spark is designed

Developing Apache Spark Applications · Apache Spark Introduction Introduction Apache Spark enables you to quickly develop applications and process jobs. Apache Spark is designed

Performance of Spark vs MapReduce

Performance of Spark vs MapReduce

Big Data Platforms for Artifical Intelligence · MapReduce and Apache Pig (HDD),Apache Spark(RAM) I Distributed SQL database queries for analytics: Apache Hive, Spark SQL, Cloudera

Big Data Platforms for Artifical Intelligence · MapReduce and Apache Pig (HDD),Apache Spark(RAM) I Distributed SQL database queries for analytics: Apache Hive, Spark SQL, Cloudera

Introduction to MapReduce, Hadoop, Spark, Shark … to MapReduce, Hadoop, Spark, Shark and Mahout Christan Grant cgrant@cise.ufl.edu

Introduction to MapReduce, Hadoop, Spark, Shark … to MapReduce, Hadoop, Spark, Shark and Mahout Christan Grant [email protected]fl.edu

Exploring Adverse Drug Effect Data with Apache Spark ... · 23/09/2015  · Apache Spark Overview Open source, Java-based alternative to Hadoop MapReduce platform Execution engine

Exploring Adverse Drug Effect Data with Apache Spark ... · 23/09/2015  · Apache Spark Overview Open source, Java-based alternative to Hadoop MapReduce platform Execution engine

Community Spotlight Apache MapReduce - Intel · Apache Hadoop* Community Spotlight Apache* MapReduce “This ‘data locality’ is a key design criterion of the MapReduce framework.

Community Spotlight Apache MapReduce - Intel · Apache Hadoop* Community Spotlight Apache* MapReduce “This ‘data locality’ is a key design criterion of the MapReduce framework.

Scrap Your MapReduce - Apache Spark

Scrap Your MapReduce - Apache Spark