Experiences Evolving a New Analytical Platform: What Works and What's Missing

Saturday, June 12, 2010

Evolving a New Analytical PlatformWhat Works and What’s Missing

Jeff HammerbacherChief Scientist, ClouderaJune 8, 2010


My BackgroundThanks for Asking

▪ [email protected]▪ Studied Mathematics at Harvard▪ Worked as a Quant on Wall Street▪ Conceived, built, and led Data team at Facebook▪ Nearly 30 amazing engineers and data scientists▪ Several open source projects and research papers

▪ Founder of Cloudera▪ Chief Scientist▪ Also, check out the book “Beautiful Data”


mailto:[email protected]

mailto:[email protected]

Presentation Outline▪ BI: Science for Profit▪ Need tools for whole research cycle▪ SQL Server 2008 R2: defining the platform

▪ State of the Platform Ecosystem▪ New Foundations: Hadoop▪ Boiling the Frog▪ Future developments

▪ Questions and Discussion


BI is looking more like science (for profit)


Jim Gray: Science entering Fourth Paradigm“We have to do better at producing tools to

support the whole research cycle”


RDBMS only a small part of this tool set


Example: SQL Server 2008 R2


RDBMS: SQL Server


RDBMS: SQL ServerETL: SQL Server Integration Services



Reporting: SQL Server Reporting Services



Reporting: SQL Server Reporting ServicesAnalysis: SQL Server Analysis Services




Search: Full-Text Search





CEP: StreamInsight





CEP: StreamInsight

OLAP: PowerPivot





CEP: StreamInsight

OLAP: PowerPivot

MDM: Master Data Services





CEP: StreamInsight

OLAP: PowerPivot

MDM: Master Data ServicesCollaboration: SharePoint


What do we call this unified suite?


For today: Analytical Data Platform


Who makes up the platform ecosystem?


Platform Providers


Platform ProvidersInfrastructure Providers



Application Developers



Application Developers

Content Providers



Application DevelopersEnd Users

Content Providers


What is new about the ecosystem today?


Content Providers1. > 95% of enterprise data is unstructured

2. Data volumes growing rapidly


Infrastructure Providers1. Cloud

2. Warehouse-Scale Computers


Platform Providers1. Open source

2. Driven by consumer web properties


Application Developers1. Data Scientists

2. Diversity of languages


End Users1. Move beyond reporting to analytics2. Make use of all enterprise data


New foundations: HDFS and MapReduce


(This is what boiling a frog feels like)


2005: Doug/Mike start project inside Nutch


2006: Doug joins Yahoo!


2007: Make Hadoop scale


2007: Make Hadoop scaleYahoo! makes Pig open source


2007: Make Hadoop scaleJim Gray’s “Fourth Paradigm” lecture

Yahoo! makes Pig open source




Randy Bryant’s “DISC” lecture




Randy Bryant’s “DISC” lecture

Powerset makes HBase open source


2008: Make Hadoop fast


2008: Make Hadoop fastYahoo! wins Daytona terabyte sort benchmark


2008: Make Hadoop fastFirst Hadoop Summit

Yahoo! wins Daytona terabyte sort benchmark



Yahoo! wins Daytona terabyte sort benchmarkYahoo! builds production webmap with Hadoop




Facebook makes Hive open source




Facebook makes Hive open source“MapReduce: A Major Step Backwards”


2009: Insert Hadoop into the enterprise


2009: Insert Hadoop into the enterpriseCloudera releases CDH



First Hadoop World NYC



First Hadoop World NYCYahoo! sorts a petabyte with Hadoop




Cloudera adds training, support, services




Cloudera adds training, support, services

“The Unreasonable Effectiveness of Data”


2010: Integrate Hadoop into the enterprise


2010: Integrate Hadoop into the enterpriseIBM announces InfoSphere BigInsights



Yahoo! completes enterprise-class security




Datameer and Karmasphere funded





Teradata, Pentaho, and others integrate





Teradata, Pentaho, and others integrateHive adds JDBC and ODBC


Hadoop will be an Analytical Data Platform


What’s Next?


Capture: Log collection and CEP


Curate: Workflow and Scheduling


Curate: Secondary and Full-Text Indexing


Curate: Learn Structure from Data


Analyze: Mesos-enabled frameworks


Analyze: Link local and global data


All behind a single pane of glass


Cloudera DesktopMaking Many Computers Feel Like One


Experiences Evolving a New Analytical Platform: What Works and What's Missing

Technology

Transcript of Experiences Evolving a New Analytical Platform: What Works and What's Missing