BUILDING TOOLS FOR THE HADOOP DEVELOPER matt winkler @mwinkle.
-
Upload
cecilia-hallas -
Category
Documents
-
view
217 -
download
1
Transcript of BUILDING TOOLS FOR THE HADOOP DEVELOPER matt winkler @mwinkle.
![Page 1: BUILDING TOOLS FOR THE HADOOP DEVELOPER matt winkler @mwinkle.](https://reader035.fdocuments.us/reader035/viewer/2022062619/5516c3f8550346f0208b59e8/html5/thumbnails/1.jpg)
BUILDING TOOLS FOR THE HADOOP DEVELOPER
matt winkler@mwinkle
![Page 2: BUILDING TOOLS FOR THE HADOOP DEVELOPER matt winkler @mwinkle.](https://reader035.fdocuments.us/reader035/viewer/2022062619/5516c3f8550346f0208b59e8/html5/thumbnails/2.jpg)
Making Hadoop easy to provision, secure, manage and use
Windows Azure Parallel DataWarehouseAppliance
Windows Server Single NodeDeveloper
Environment
![Page 3: BUILDING TOOLS FOR THE HADOOP DEVELOPER matt winkler @mwinkle.](https://reader035.fdocuments.us/reader035/viewer/2022062619/5516c3f8550346f0208b59e8/html5/thumbnails/3.jpg)
Authoring Jobs App Integration
Building Developer Tools
Core Hadoop
Consistent REST API’s
Breadth of Clients (Java, JS, .NET, etc)
Authoring frameworks and languages
End User Tooling (IDE’s, Analyst tools, Command lines)
ConnectivityProgrammabilitySecurityLoosely coupled
LightweightLow cost to extendScenario oriented
Innovation flows upward
New compute modelsPerf enhancements
Extend breadth & depthEnable new scenariosIntegrate with current tool chains
![Page 4: BUILDING TOOLS FOR THE HADOOP DEVELOPER matt winkler @mwinkle.](https://reader035.fdocuments.us/reader035/viewer/2022062619/5516c3f8550346f0208b59e8/html5/thumbnails/4.jpg)
Existing Ecosystem Actively contributing to:
Core Pig Hive HCatalog
Branching to other projects Simple one-box developer install
on Windows
![Page 5: BUILDING TOOLS FOR THE HADOOP DEVELOPER matt winkler @mwinkle.](https://reader035.fdocuments.us/reader035/viewer/2022062619/5516c3f8550346f0208b59e8/html5/thumbnails/5.jpg)
.NET
Map/Reduce LINQ to Hive Client API’s
WebHCat Ambari WebHDFS Azure
Visual Studio Tooling Local debugging support
![Page 6: BUILDING TOOLS FOR THE HADOOP DEVELOPER matt winkler @mwinkle.](https://reader035.fdocuments.us/reader035/viewer/2022062619/5516c3f8550346f0208b59e8/html5/thumbnails/6.jpg)
JavaScript MRjs – Map/Reduce in JavaScript Node.js client API’s
WebHCat WebHDFS Ambari Azure
![Page 7: BUILDING TOOLS FOR THE HADOOP DEVELOPER matt winkler @mwinkle.](https://reader035.fdocuments.us/reader035/viewer/2022062619/5516c3f8550346f0208b59e8/html5/thumbnails/7.jpg)
Management
UI Tooling Cluster usage Job authoring Result consumption in common tools
PowerShell & Cross platform scripting
API Surface RDFE – Azure provisioning Ambari – Cluster monitoring WebHCatalog – Metadata and job submission WebHDFS, Blob Storage – Storage
Management Portal
>_Scripting
(Windows, Linux and Mac)
REST API
![Page 8: BUILDING TOOLS FOR THE HADOOP DEVELOPER matt winkler @mwinkle.](https://reader035.fdocuments.us/reader035/viewer/2022062619/5516c3f8550346f0208b59e8/html5/thumbnails/8.jpg)
Sources http://hadoopsdk.codeplex.com http://www.github.com/windowsazure
NuGet packages Microsoft.Hadoop.MapReduce Microsoft.Hadoop.Hive Microsoft.Hadoop.WebHDFS => WebClient
NPM packages Azure Azure-cli Hadoop REST clients pending…
open
![Page 9: BUILDING TOOLS FOR THE HADOOP DEVELOPER matt winkler @mwinkle.](https://reader035.fdocuments.us/reader035/viewer/2022062619/5516c3f8550346f0208b59e8/html5/thumbnails/9.jpg)
Big Data. Small Data. All Data.
www.microsoft.com/bigdata
Learn More:February 289:05am Microsoft Keynote
by Dave Campbell
9:45am Microsoft Keynoteby Kate Crawford
10:40am Microsoft Sponsored Session
in Partnership with Ascribe
11:30am Dave Campbell’s Office Hours