Introduction to the Hortonworks YARN Ready Program
-
Upload
hortonworks -
Category
Technology
-
view
106 -
download
2
description
Transcript of Introduction to the Hortonworks YARN Ready Program
Page 1 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Introduction to YARN Ready
John Kreisa – VP Strategic MarketingAjay Singh - Director, Partner Solutions
Page 2 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
YARN Ready Program OverviewJohn Kreisa
Page 3 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Why YARN Ready? -> Apache Hadoop YARN
• YARN is the architecture center of Apache Hadoop 2 enabling more efficient cluster utilization
• More and more customers are asking about tools and applications that are integrated into YARN to maximize the value of their Hadoop cluster
• YARN Ready is a program offered by Hortonworks the leading developers of YARN
– YARN was conceived of by Arun Murthy Hortonworks founder– He submitted JIRA MapReduce 279 in January 2008 and has been working on it since– Hortonworks has written 80% of the YARN code
Page 3
Page 4 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
The Two YARN Ready Program Goals
Expand the Hadoop ecosystem by enabling ISVs to build & validate YARN-based applications, using Apache technologies
Allow enterprises to know which technologies maximize Hadoop resources in an ecosystem-friendly way
Page 5 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Two Achievements Available
HDP Certified
YARN Ready
Badge Description
HDP (Certification) Integrates with HDP 2.x
YARN Ready (Validation) Uses the YARN resource scheduler
Page 6 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
YARN Technical Overview and ValueAjay Singh
Page 7 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Hadoop 1.0: Batch
HADOOP 1.0Built for Web-Scale Batch Apps
Single App
BATCH
HDFS
Single App
INTERACTIVE
Single App
BATCH
HDFS
• All other usage patterns must leverage that same infrastructure
• Forces the creation of silos for managing mixed workloads
Single App
BATCH
HDFS
Single App
ONLINE
Page 8 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Hadoop 2.0: Multi Workload
HADOOP 1.0
HDFS(redundant, reliable storage)
MapReduce(cluster resource management
& data processing)
HDFS2(redundant, reliable storage)
YARN(cluster resource management)
MapReduce(data processing)
Others(data processing)
HADOOP 2.0
Single Use SystemBatch Apps
Multi Purpose PlatformBatch, Interactive, Online, Streaming, …
Page 9 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Key Driver Of Hadoop Adoption: Enterprise Data Lake
FlexibleEnables other purpose-built data processing models beyond MapReduce (batch), such as interactive and streaming
EfficientDouble processing IN Hadoop on the same hardware while providing predictable performance & quality of service
SharedProvides a stable, reliable, secure foundation and shared operational services across multiple workloads
Data Processing Engines Run Natively IN HadoopBATCH
MapReduceINTERACTIVE
TezSTREAMING
StormIN-MEMORY
SparkGRAPHGiraph
ONLINEHBase, Accumulo
OTHERS
HDFS: Redundant, Reliable Storage
YARN: Cluster Resource Management
Page 10 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
YARN Development Framework
YARN : Data Operating System
°1 ° ° ° ° ° ° ° °
° ° ° ° ° ° ° ° °
°
°°
° ° ° ° ° ° °
° ° ° ° ° ° N
HDFS (Hadoop Distributed File System)
System
BatchMapReduce
InteractiveTez
Engine Long RunningSlider
Direct
ISV Apps
Scripting
Pig
SQL
Hive
Cascading
JavaScala
NoSQL
HBaseAccumulo
Stream
StormAPIISV
AppsISVAps
Applications
Others
Spark ISV Apps
ISVApps
Page 11 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Tez – Next Generation Data Processing Framework
YARN ApplicationMaster to run DAG of Tez Tasks
Task with pluggable Input, Processor and Output
Tez Task - <Input, Processor, Output>
Task
ProcessorInput Output
Page 12 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Slider – Onboarding Long Running Services To YARN
A YARN application to deploy existing distributed applications, monitor them and make them larger or smaller as desired - even while the cluster is running
Has a plug-in provider architecture to support different applications
Automatic Deploy, Manage, Monitor via Ambari– Standard management APIs for Ambari to use
– Standard ways to collect metrics
Clusters can be stopped, "frozen" and restarted, "thawed" later– Distribution of the deployed application across the YARN cluster is persisted.
Page 13 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
# of Apps / Ease of Integration
Con
trol
Integration Option Tradeoffs
YARN Native Apps
Tez Apps
Services on YARN (Slider)
MapReduce Apps
Hive
Page 14 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
HADOOP
YARN
HDFS
SQL
Datanode
HDFS
Visual Data & Analytics Workflow
Actian Analytics Platform – YARN Native Integration ExampleTransform Hadoop into a High Performance Analytics Platform
Datanode
HDFS
Datanode
HDFS
Datanode
HDFS
X100X100X100
ReadLoad
Actian VectorBlend &Enrich
Data Science & Analytics
Datanode
HDFS
X100
HDFS
Vector
• Original file format• Standard block
replication
• Column-based blocks
• Binary• Compressed• Partitioned
• Faster Loading• Faster SQL• Standard SQL• Better Scaling
NamenodeHigh Performance, Industrialized SQL
Database
High Performance, Data Science &
Analytics
Page 15 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Partners with YARN Ready Apps
Page 16 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
YARN Ready Program DetailsJohn Kreisa
Page 17 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Partner Benefits - Marketing
Recognition Recognition as Hortonworks Certified Technology Partner on our website along with the YARN Ready
badge
Hortonworks quote for reference in partner marketing assets
Promotion to customers in Hortonworks led events and activities
Joint Marketing & Field Alignment
Invitation to Modern Data Architecture Webinar Series
Hortonworks Sales Briefing and alignment with Field BD
Automatic participation in digital marketing initiatives
LinkedIn and FB campaigns
Twitter and blog activity
Page 18 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Partner Benefits - Technical Tech Expertise YARN implementation guide and sample code
Integration best practice guidance
Access to the architects and builders of YARN
Roadmap Input YARN, Tez and Stinger roadmap visibility
Input into the direction of YARN and the YARN engines
Access to pre-release builds to speed integration with new features
Page 19 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Process for Partners
Become a Hortonworks partner Sign up at http://hortonworks.com/partners/become-a-partner/
Become Certified Download the certification kit (a .zip archive) from the partner portal
Validate with HDP (Sandbox OK)
Complete the provided templates (samples are included) for
Solution Overview
Reference Architecture
Certification Report
Submit the three documents to [email protected]. SLA: 10 working days for Hortonworks response.
Collaborate with Hortonworks on joint marketing – via [email protected]
Page 20 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Certification Kit
A .zip file available to partners
Contents:
Program Guide
Solution Overview template (DOC)
Reference Architecture template (PPT)
Certification Report template & sample (DOC)
Sample YARN App code*
Sample Slider provider code*
YARN app test recommendations*
Logos (available after certification)
PR Template (available after certification)
README file (text)
*YARN Ready, Coming – target July
Page 21 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Calls to Action
Engage us on your YARN initiative
Become HDP Certified and YARN Ready
Develop a joint Sandbox Tutorial
Collaborate on “Art of Possible” Demonstration
Co-Market & Co-Sell
Page 22 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Questions?Join us for the next series of webinars on integrating to YARN:YARN native July 24 @ 9am PTSlider integration August 7 @ 9am PTTez integration August 21 @ 9am PT
Page 22
Page 23 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Thank You