Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …
-
Upload
mortardata -
Category
Technology
-
view
1.093 -
download
1
description
Transcript of Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …
![Page 1: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …](https://reader033.fdocuments.us/reader033/viewer/2022060111/5566966ad8b42a78708b4e6c/html5/thumbnails/1.jpg)
![Page 2: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …](https://reader033.fdocuments.us/reader033/viewer/2022060111/5566966ad8b42a78708b4e6c/html5/thumbnails/2.jpg)
Open-source softwaredevelopment frameworks
![Page 3: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …](https://reader033.fdocuments.us/reader033/viewer/2022060111/5566966ad8b42a78708b4e6c/html5/thumbnails/3.jpg)
Open-source software dev frameworks
(and many more...)
![Page 4: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …](https://reader033.fdocuments.us/reader033/viewer/2022060111/5566966ad8b42a78708b4e6c/html5/thumbnails/4.jpg)
• Avoid repeat work• Remain 100% customizable• Escape vendor lock-in
Every platform needsopen-source, code-based dev frameworks
![Page 5: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …](https://reader033.fdocuments.us/reader033/viewer/2022060111/5566966ad8b42a78708b4e6c/html5/thumbnails/5.jpg)
But for data?Nothing.
![Page 6: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …](https://reader033.fdocuments.us/reader033/viewer/2022060111/5566966ad8b42a78708b4e6c/html5/thumbnails/6.jpg)
Data work today• Proprietary / GUI / No framework• Not reusable• Little collaboration• Too many errors, too slow to fix
![Page 7: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …](https://reader033.fdocuments.us/reader033/viewer/2022060111/5566966ad8b42a78708b4e6c/html5/thumbnails/7.jpg)
an open source dev framework for data
![Page 8: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …](https://reader033.fdocuments.us/reader033/viewer/2022060111/5566966ad8b42a78708b4e6c/html5/thumbnails/8.jpg)
“Mortar takes something complex and makes it simple and intuitive.”
—Jon Coveney, Twitter
![Page 9: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …](https://reader033.fdocuments.us/reader033/viewer/2022060111/5566966ad8b42a78708b4e6c/html5/thumbnails/9.jpg)
What big data system should I use? Hadoop, HPCC, Disco, Storm… What Hadoop distro should I use?How much will this cost? Should I do it some other way?How long is this going to take to learn? Should I do it some other way?How many machines should I buy?Can I run in the cloud?How should I interact with it? Pig, Hive, Cascading, Scalding, CascalogCan I use libraries I need?How do I install everything?How can my team share code?How can I reuse code?Is my workload typical?How can I safely deploy?How do I know if what I've written is correct?Are there any libraries I could start with?Can I connect to my key-value store?Can I do machine learning in Hadoop?Is it secure?What if I need help?
Weeks
![Page 10: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …](https://reader033.fdocuments.us/reader033/viewer/2022060111/5566966ad8b42a78708b4e6c/html5/thumbnails/10.jpg)
More Weeks
![Page 11: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …](https://reader033.fdocuments.us/reader033/viewer/2022060111/5566966ad8b42a78708b4e6c/html5/thumbnails/11.jpg)
Even More Weeks
![Page 12: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …](https://reader033.fdocuments.us/reader033/viewer/2022060111/5566966ad8b42a78708b4e6c/html5/thumbnails/12.jpg)
“Our focus in designing Pig has always been to make Hadoop easy...
Mortar's approach is right on—they extend our quick start and ease of use focuses with pre-built Hadoop clusters, clear examples, code organization templates, and github for social sharing of the code.”
OPEN SOURCED PIG—Alan Gates, Hortonworks co-founder
![Page 13: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …](https://reader033.fdocuments.us/reader033/viewer/2022060111/5566966ad8b42a78708b4e6c/html5/thumbnails/13.jpg)
Mortar is for analyzing lots of data in AWS.
Can Mortar help you?
![Page 14: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …](https://reader033.fdocuments.us/reader033/viewer/2022060111/5566966ad8b42a78708b4e6c/html5/thumbnails/14.jpg)
Mortar serves companies of all sizes from any industry.
Who is Mortar for?
![Page 15: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …](https://reader033.fdocuments.us/reader033/viewer/2022060111/5566966ad8b42a78708b4e6c/html5/thumbnails/15.jpg)
“...Mortar fits right in with our vision of the future... With this exciting launch, MongoDB users can now also seamlessly use Mortar.”
—Dwight MerrimanFOUNDER OF 10GEN (MAKER OF MONGODB), DOUBLECLICK (ACQ. GOOGLE), SHOPWIKI, BUSINESS INSIDER, GILT GROUPE
![Page 16: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …](https://reader033.fdocuments.us/reader033/viewer/2022060111/5566966ad8b42a78708b4e6c/html5/thumbnails/16.jpg)
By and for engineers and data scientists
![Page 17: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …](https://reader033.fdocuments.us/reader033/viewer/2022060111/5566966ad8b42a78708b4e6c/html5/thumbnails/17.jpg)
> gem install mortar
![Page 18: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …](https://reader033.fdocuments.us/reader033/viewer/2022060111/5566966ad8b42a78708b4e6c/html5/thumbnails/18.jpg)
> mortar new my_project
![Page 19: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …](https://reader033.fdocuments.us/reader033/viewer/2022060111/5566966ad8b42a78708b4e6c/html5/thumbnails/19.jpg)
![Page 20: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …](https://reader033.fdocuments.us/reader033/viewer/2022060111/5566966ad8b42a78708b4e6c/html5/thumbnails/20.jpg)
> git clone your_project
![Page 21: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …](https://reader033.fdocuments.us/reader033/viewer/2022060111/5566966ad8b42a78708b4e6c/html5/thumbnails/21.jpg)
> mortar run your_project
![Page 22: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …](https://reader033.fdocuments.us/reader033/viewer/2022060111/5566966ad8b42a78708b4e6c/html5/thumbnails/22.jpg)
![Page 23: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …](https://reader033.fdocuments.us/reader033/viewer/2022060111/5566966ad8b42a78708b4e6c/html5/thumbnails/23.jpg)
![Page 24: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …](https://reader033.fdocuments.us/reader033/viewer/2022060111/5566966ad8b42a78708b4e6c/html5/thumbnails/24.jpg)
Pig is easy to learn(and we’ve made it easier)
![Page 25: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …](https://reader033.fdocuments.us/reader033/viewer/2022060111/5566966ad8b42a78708b4e6c/html5/thumbnails/25.jpg)
![Page 26: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …](https://reader033.fdocuments.us/reader033/viewer/2022060111/5566966ad8b42a78708b4e6c/html5/thumbnails/26.jpg)
![Page 27: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …](https://reader033.fdocuments.us/reader033/viewer/2022060111/5566966ad8b42a78708b4e6c/html5/thumbnails/27.jpg)
Illustrate is awesome• Find your mistakes• Understand code before collaborating• Automated tests: a way to test every condition
![Page 28: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …](https://reader033.fdocuments.us/reader033/viewer/2022060111/5566966ad8b42a78708b4e6c/html5/thumbnails/28.jpg)
Hadoop & Python are powerful data science tools
![Page 29: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …](https://reader033.fdocuments.us/reader033/viewer/2022060111/5566966ad8b42a78708b4e6c/html5/thumbnails/29.jpg)
...but they haven’t worked together before.
![Page 30: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …](https://reader033.fdocuments.us/reader033/viewer/2022060111/5566966ad8b42a78708b4e6c/html5/thumbnails/30.jpg)
Now you can use Hadoop & real Python on Mortar
![Page 31: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …](https://reader033.fdocuments.us/reader033/viewer/2022060111/5566966ad8b42a78708b4e6c/html5/thumbnails/31.jpg)
![Page 32: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …](https://reader033.fdocuments.us/reader033/viewer/2022060111/5566966ad8b42a78708b4e6c/html5/thumbnails/32.jpg)
![Page 33: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …](https://reader033.fdocuments.us/reader033/viewer/2022060111/5566966ad8b42a78708b4e6c/html5/thumbnails/33.jpg)
• Installed Mortar• Made a new project• Cloned a project• Ran the project• Illustrated project• Use Python and other libraries on Hadoop
What you just saw
![Page 34: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …](https://reader033.fdocuments.us/reader033/viewer/2022060111/5566966ad8b42a78708b4e6c/html5/thumbnails/34.jpg)
- Web Projects: zero install, in the browser
2 options for using Mortar:
- Git Projects: modularity, testability, code sharing, local dev, and revision control.
![Page 35: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …](https://reader033.fdocuments.us/reader033/viewer/2022060111/5566966ad8b42a78708b4e6c/html5/thumbnails/35.jpg)
![Page 36: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …](https://reader033.fdocuments.us/reader033/viewer/2022060111/5566966ad8b42a78708b4e6c/html5/thumbnails/36.jpg)
• Use your browser• Minutes to connect data• Productive in one hour
One-hour challenge
![Page 37: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …](https://reader033.fdocuments.us/reader033/viewer/2022060111/5566966ad8b42a78708b4e6c/html5/thumbnails/37.jpg)
As a good citizen, Mortar has a rich API
How does Mortar fit with other
![Page 38: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …](https://reader033.fdocuments.us/reader033/viewer/2022060111/5566966ad8b42a78708b4e6c/html5/thumbnails/38.jpg)
Full speed, directly on Hadoop
How about speed?
![Page 39: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …](https://reader033.fdocuments.us/reader033/viewer/2022060111/5566966ad8b42a78708b4e6c/html5/thumbnails/39.jpg)
• Easy start• Keeps you productive• Collaborate with data• No lock-in• Easy to budget
Mortar revolutionizes your data pipeline.
![Page 40: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …](https://reader033.fdocuments.us/reader033/viewer/2022060111/5566966ad8b42a78708b4e6c/html5/thumbnails/40.jpg)
• Free | Service use unlimited | 10 node-hours• Pay as you Go | $0.89/node-hour | support• Enterprise | $3,000/month | $0.69/node-hour | live
support
Tiers
![Page 41: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …](https://reader033.fdocuments.us/reader033/viewer/2022060111/5566966ad8b42a78708b4e6c/html5/thumbnails/41.jpg)
![Page 42: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …](https://reader033.fdocuments.us/reader033/viewer/2022060111/5566966ad8b42a78708b4e6c/html5/thumbnails/42.jpg)
mortardata.com / @mortardata