Jayush Luniya

Apache Ambari PMC

Principal Software Engineer @ Hortonworks

MPACK BASED OPERATIONS MODEL

Agenda

Agenda

Agenda

Stack Based Operations Model

Management Packs – V1

Limitations

Mpacks Based Operations Model

Summary

Q&A

Stack Based Operations Model

Agenda

Stack Based Operations Model

Stack Based Operations Model

Ambari built on a standardized stack based operations model

Pluggable architecture; no vendor lock-in

Stacks wrap services of all shapes and sizes

Consistent definition & lifecycle control layer

Consistent approach for managing & monitoring services

Stacks embedded in Ambari

Management Packs – V1

Agenda

Management Packs – V1

Introduced in 2.4

V1 release based on existing stack based operations model

First take on decoupling stack releases

Bundles stacks, add-on services

Final stack definition is overlay of multiple mpacks

Stacks don’t need to be embedded in Ambari

Management Packs – V1

Service Level Extension Points

Extension Apache JIRA Apache Wiki

Stack Advisor Extensions AMBARI-15226 Ambari Wiki

Role Command Order Extensions AMBARI-9363 Ambari Wiki

Quick Links Extensions AMBARI-11268 Ambari Wiki

Service Repos Extensions AMBARI-15538 Ambari Wiki

Upgrade Pack Extensions AMBARI- 15388 Ambari Wiki

Limitations

Agenda

Limitations

Cluster attached to single stack Cannot deploy HDFS from HDP, NiFi from HDF in same cluster

Entire cluster upgrade

Stacks are monolithic

Ambari upgrade for refreshing stacks

No clear upgrade story for add-on services

Ownership of service deployment scripts

Mpack Based Operations Model

Agenda

Goals

Replacement for monolithic stacks

Cluster to multiple mpacks

Scenario: Deploy HDP +HDF cluster

Decouple Ambari core release

Independent release cycles

First class support for add-on services

Support dynamic installation

Modular upgrades

New Concepts

Deployment artifact

Individual service ownership

Independent release cycles

Examples

HDFS-3.0.0-packlet

SPARK-2.0.0-packlet

Distribution artifact

Composition of packlets

Modular release & upgrades

Examples

ODPi-2.0.0-mpack

HDP-3.0.0-mpack

HDF-3.1.0-mpack

Packlets Management Packs (Mpacks V2)

New Concepts

Catalog of management packs

Compatibility between mpacks

Search and discover mpacks

Support multiple mpack repos

Examples

ODPi Mpack Repository

Hortonworks Mpack Repository

Mpack Repository

Mpack Based Operations Model

Mpack Processing Engine

Mpack Repository

DistroX3.0.0

DistroY3.2.0

Release Management

Completely decouple Ambari releases

Service ownership with domain experts

Independent release cadence

Mpack Based Upgrades

Modular upgrades

Upgrade subset of services in a cluster that belong to mpack

Upgrade packs defined at mpack level

Download new version of mpack and kickoff upgrade

Upgrade Pack

“upgrade-from” instead of “upgrade-to”

Add-on Mpacks

First class support for 3rd party services

Mpacks provide a 3rd party onboarding platform

Add-on services not overlaid into stack

Independent release management

Independent and modular upgrades

Summary

Agenda

Summary

Modular build & release

Independent release mgmt

Modular upgrades

3rd party onboarding platform

Multi-everything vision

Service level refactoring

Resource management library

No network connectivity

Ownership

Testing

Key Takeaways Challenges

Summary

Future YARN Services

Umbrella Apache Epic

AMBARI-14714 – Multi Everything Architecture

Apache Epic

AMBARI-19621 – Mpack Based Operations Model

Dataworks Summit Talk

Future of Apache Ambari – Jaimin, Jayush

Q&A

Agenda

Backup Slides

Agenda