Hadoop & devOps : better together
-
Upload
maxime-lanciaux -
Category
Engineering
-
view
283 -
download
3
Transcript of Hadoop & devOps : better together
![Page 1: Hadoop & devOps : better together](https://reader033.fdocuments.us/reader033/viewer/2022051006/58cfaf021a28ab223a8b4651/html5/thumbnails/1.jpg)
1 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Hadoop & DevOps : better togetherMaxime LanciauxDevOps System ArchitectFebruary 2017
![Page 2: Hadoop & devOps : better together](https://reader033.fdocuments.us/reader033/viewer/2022051006/58cfaf021a28ab223a8b4651/html5/thumbnails/2.jpg)
2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
![Page 3: Hadoop & devOps : better together](https://reader033.fdocuments.us/reader033/viewer/2022051006/58cfaf021a28ab223a8b4651/html5/thumbnails/3.jpg)
3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
AgendaAutomate & Speed up deployment
![Page 4: Hadoop & devOps : better together](https://reader033.fdocuments.us/reader033/viewer/2022051006/58cfaf021a28ab223a8b4651/html5/thumbnails/4.jpg)
4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Automate & Speed up deployment
Continuous integration workflow
Dev + OpsSet of practices that emphasize the collaboration
and communication of both software developers and information technology professionals
while automating the process of software delivery and infrastructure changes.
Automate deployment with tools to ensure quality
![Page 5: Hadoop & devOps : better together](https://reader033.fdocuments.us/reader033/viewer/2022051006/58cfaf021a28ab223a8b4651/html5/thumbnails/5.jpg)
5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Automate & Speed up deployment
Continuous integration + Build machine– Jenkins + Maven
Configuration management tool (Hadoop + non Hadoop tools)– Ambari + Ansible
Configuration management (configuration, projects, Jenkins pipeline)– Git
Big Data Platform + Database + reporting– HDP + PostgreSQL + Grafana + Zeppelin
Others– Custom log4J config, bash template, custom python scripts
Typical set of tool
You may want to use virtualization, containerization, continuous inspection, code collaboration tool as well
![Page 6: Hadoop & devOps : better together](https://reader033.fdocuments.us/reader033/viewer/2022051006/58cfaf021a28ab223a8b4651/html5/thumbnails/6.jpg)
6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Automate & Speed up deploymentTypical Architecture
DN
DN
DN
MN
MN
Database
Reporting
Edge Node
Edge Node
Configuration Management
(Tag / Branches)
BuildAutomation
Continuous integration
(Parameter, template)
Configuration and Management platform
+
data
Configuration / workflow / tests
![Page 7: Hadoop & devOps : better together](https://reader033.fdocuments.us/reader033/viewer/2022051006/58cfaf021a28ab223a8b4651/html5/thumbnails/7.jpg)
7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Automate & Speed up deploymentTypical Architecture
Database
ReportingConfiguration Management
BuildAutomation
Continuous integration
Configuration and Management platform
Dev + DR
Prod + Validation
+
![Page 8: Hadoop & devOps : better together](https://reader033.fdocuments.us/reader033/viewer/2022051006/58cfaf021a28ab223a8b4651/html5/thumbnails/8.jpg)
8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Automate & Speed up deployment
Typical Organization HDFS directory for Dev, Validation, Production, Backup Hive databases and Kafka topics with naming convention HBase dedicated namespace YARN Capacity Scheduler with corresponding queue (Dev, Validation, Production) Ranger rules (Project, team, environment with groups)
All automatically created during init of project (On Dev by user, on Prod by admin)
![Page 9: Hadoop & devOps : better together](https://reader033.fdocuments.us/reader033/viewer/2022051006/58cfaf021a28ab223a8b4651/html5/thumbnails/9.jpg)
9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Automate & Speed up deployment
On the Git side Codify everything Version everything Organize infrastructure as code
![Page 10: Hadoop & devOps : better together](https://reader033.fdocuments.us/reader033/viewer/2022051006/58cfaf021a28ab223a8b4651/html5/thumbnails/10.jpg)
10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
AgendaAutomate & Speed up deployment
Enforce testing and quality
![Page 11: Hadoop & devOps : better together](https://reader033.fdocuments.us/reader033/viewer/2022051006/58cfaf021a28ab223a8b4651/html5/thumbnails/11.jpg)
11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Enforce testing and quality
Use of multi-tenant (1+n) Hadoop clusters– Which means naming convention and parameter / security
Environments / Context
![Page 12: Hadoop & devOps : better together](https://reader033.fdocuments.us/reader033/viewer/2022051006/58cfaf021a28ab223a8b4651/html5/thumbnails/12.jpg)
12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Enforce testing and quality
Use of multi-tenant Hadoop cluster– Which means naming convention and parameter / security
Environments / Context
Dev Prex Prod Dev Dev
![Page 13: Hadoop & devOps : better together](https://reader033.fdocuments.us/reader033/viewer/2022051006/58cfaf021a28ab223a8b4651/html5/thumbnails/13.jpg)
13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Enforce testing and quality
Use of multi-tenant Hadoop cluster– Which means naming convention and parameter / security
Environments / Context
![Page 14: Hadoop & devOps : better together](https://reader033.fdocuments.us/reader033/viewer/2022051006/58cfaf021a28ab223a8b4651/html5/thumbnails/14.jpg)
14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Enforce testing and quality
Use of multi-tenant Hadoop cluster– Which means naming convention and parameter / security
Environments / Context
![Page 15: Hadoop & devOps : better together](https://reader033.fdocuments.us/reader033/viewer/2022051006/58cfaf021a28ab223a8b4651/html5/thumbnails/15.jpg)
15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Enforce testing and quality
Hadoop logs Monitoring of full stacks component Application logs Evolution of performance Code review with or without tools Test are include in the bundle (for non-reg)
Massive use of Pareto
![Page 16: Hadoop & devOps : better together](https://reader033.fdocuments.us/reader033/viewer/2022051006/58cfaf021a28ab223a8b4651/html5/thumbnails/16.jpg)
16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
AgendaAutomate & Speed up deployment
Enforce testing and quality
Protect your enterprise asset
![Page 17: Hadoop & devOps : better together](https://reader033.fdocuments.us/reader033/viewer/2022051006/58cfaf021a28ab223a8b4651/html5/thumbnails/17.jpg)
17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Environments Configuration management Automation Impact analysis 5W
Protect your enterprise asset
Protect your enterprise asset Your company
TMA #1 TMA #2 TMA #n
DevOps
![Page 18: Hadoop & devOps : better together](https://reader033.fdocuments.us/reader033/viewer/2022051006/58cfaf021a28ab223a8b4651/html5/thumbnails/18.jpg)
18 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
AgendaAutomate & Speed up deployment
Enforce testing and quality
Protect your enterprise asset
Ease Hadoop migration
![Page 19: Hadoop & devOps : better together](https://reader033.fdocuments.us/reader033/viewer/2022051006/58cfaf021a28ab223a8b4651/html5/thumbnails/19.jpg)
19 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Install, configure and deploy All test to be schedule / run Follow performance evolution Can be as easy as configuring access to a new edge node + configuration / password /
SSH / certificates / security Multi-component environments
Ease Hadoop migration
A lovely hand
![Page 20: Hadoop & devOps : better together](https://reader033.fdocuments.us/reader033/viewer/2022051006/58cfaf021a28ab223a8b4651/html5/thumbnails/20.jpg)
20 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
AgendaAutomate & Speed up deployment
Enforce testing and quality
Protect your enterprise asset
Ease Hadoop migration
Empower Administrator and Developer
![Page 21: Hadoop & devOps : better together](https://reader033.fdocuments.us/reader033/viewer/2022051006/58cfaf021a28ab223a8b4651/html5/thumbnails/21.jpg)
21 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Developer to get a better feedback on their application Administrator to check / push back best practices Numerous action to be completely automated (ssh) Focus more high level task, with direct access to information Get directly access to template (no need to reinvent the wheels)
Empower Administrator and Developer
Same tools, same discussion
![Page 22: Hadoop & devOps : better together](https://reader033.fdocuments.us/reader033/viewer/2022051006/58cfaf021a28ab223a8b4651/html5/thumbnails/22.jpg)
22 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
AgendaAutomate & Speed up deployment
Enforce testing and quality
Protect your enterprise asset
Ease Hadoop migration
Empower Administrator and Developer
With full monitoring for Human
![Page 23: Hadoop & devOps : better together](https://reader033.fdocuments.us/reader033/viewer/2022051006/58cfaf021a28ab223a8b4651/html5/thumbnails/23.jpg)
23 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
With full monitoring for human
Per project, per environment, per Hadoop cluster or on all of them
![Page 24: Hadoop & devOps : better together](https://reader033.fdocuments.us/reader033/viewer/2022051006/58cfaf021a28ab223a8b4651/html5/thumbnails/24.jpg)
24 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
With full monitoring for human
Per project, per environment, per Hadoop cluster or on all of them
![Page 25: Hadoop & devOps : better together](https://reader033.fdocuments.us/reader033/viewer/2022051006/58cfaf021a28ab223a8b4651/html5/thumbnails/25.jpg)
25 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
With full monitoring for human
Per project, per environment, per Hadoop cluster or on all of them
![Page 26: Hadoop & devOps : better together](https://reader033.fdocuments.us/reader033/viewer/2022051006/58cfaf021a28ab223a8b4651/html5/thumbnails/26.jpg)
26 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
AgendaAutomate & Speed up deployment
Enforce testing and quality
Protect your enterprise asset
Ease Hadoop migration
Empower Administrator and Developer
With full monitoring for Human
Demo
![Page 27: Hadoop & devOps : better together](https://reader033.fdocuments.us/reader033/viewer/2022051006/58cfaf021a28ab223a8b4651/html5/thumbnails/27.jpg)
27 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
We are Recruiting !Email me :[email protected]
![Page 28: Hadoop & devOps : better together](https://reader033.fdocuments.us/reader033/viewer/2022051006/58cfaf021a28ab223a8b4651/html5/thumbnails/28.jpg)
28 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Thank You