Performance Aware DevOps - BME-HITbuttyan/atnc/20160122_Casale_Devops.pdf · •Agile development...

Performance Aware DevOps

Giuliano CasaleDepartment of Computing (DoC)

Imperial College [email protected]

The Road to Agility

• Agile development is a success story

• But development (Dev) hits a bottleneck during operations (Ops)

• Programmable infrastructures provide a solution (e.g., Puppet, Chef, …)1/24/2016 G. Casale – Performance Aware DevOps 2

Is DevOps Just IT Automation?• Automation is the answer• The question is how to improve:• Collaborationof Dev & Ops

• Sharing of responsibilities

• System‐oriented mindset

1/24/2016 G. Casale – Performance Aware DevOps 3

Embracing DevOps


Development ProductionTools

DevelopmentTools

IT Operations

Testing

BeforeDevOps

Development Production

ToolsDevelopment & IT Operations

Testing

DevOps

Continuous Integration Continuous Testing Continuous Monitoring

QA – Quality Assurance

Unified Processes

Unified Tooling

Source: What is DevOps? E. Di Nitto et al.

Quality Assurance in Today’s DevOps

• Mainly functional testing• What about performance tests?

• “Just 13% companies run automated performance tests within continuous integration servers” [Neotys]


Performance Engineering (PE)

1/24/2016 6

Requirements

SLAs

Compare Alternatives

Load testing

Cost Tradeoffs

Monitoring

Capacity Management

Incident Analysis Deployment

ProfilingSPE

Regression

Bottleneck Identification

Root CauseAnalysis

?

User behaviour

Adaptation

• Somewhat similar to early 2000s• Dev teams write UML models• PE focuses on analytical models• How to facilitatethe interaction?

• Model‐Driven Engineering (MDE) is the only game in town

A New Challenge?


Platform‐Independ. Model

ArchitectureModel

Platform‐Specific Model

Java

Performance Model

Performance Aware MDEPerformance Annotations

• UML MARTE profile, UML DAM profile


FailureProbability

UsageProfileSystem

Behaviour


Model‐to‐Model TransformationFrom Architecture to Analytical Models


Model‐to‐Model TransformationFrom Architecture to Analytical Models

database CPUadminServer CPU

SVNagent

HTTPagent

UserThink Time

database CPUadminServer CPU

SVNagent

HTTPagentUser behaviour 1(class 1)

User behaviour 2 (class 2)

ThroughputsResponse times

Resource Utilization

1s .3s

.7s

2s

So What About Continuous Delivery?Measurement‐Driven DevOps


MySQL1

MySQL2

MySQL3

Tomcat 1

Tomcat 2UsersDev

jenkins

chef

incident report

release

Ops

incident report

(performanceunit tests)

So What About Continuous Delivery?Today’s Performance‐Aware MDE methods


MySQL1

MySQL2

MySQL3

Tomcat 1

Tomcat 2UsersDev

jenkins

chef

incident report

early-stage performance assessment

Ops

incident report

release


So What About Continuous Delivery?Performance Aware DevOps


MySQL1

MySQL2

MySQL3

Tomcat 1

Tomcat 2UsersDev

jenkins

chef

incident report& model correlation

continuousperformance engineering(a “shared system view”)

Ops

incident report

continuous parameterization

release


Performance Aware DevOps

1/24/2016 14

Requirements

SLAs

Compare Alternatives

Load testing

Cost Tradeoffs

Monitoring

Capacity Management

Incident Analysis Deployment

ProfilingSPE

Regression

Bottleneck Identification

Root CauseAnalysis

continuousperformanceengineering

User behaviour

Adaptation

Research Roadmap

1. Shared MDE models for both Dev and Ops2. Extensions of existing tools for performance

validation to model Ops environment3. Continuous parameterization methods4. Traceability and anomaly detection


Research Roadmap1. Shared MDE models for both Dev and Ops

• Annotations for recent technologies:• Cloud computing• Internet‐of‐Things • Cyber‐Physical Systems• Mobile devices• Big data

• Some relevant outputs from large EU projects


Research Roadmap




• Tools: LQNS, LINE, PerfCenter, JMT, ...

• Challenge: model new resource types (e.g., spot & micro VM, …) and platform services

Extension of Existing Tools Model‐Driven Prediction


Physical Host

Lo Hi

Virtualization Layer

VM

VM

VMmy VM

Cloud CPU steal modelCPUSteal

Up Kill

Spot instance price Simulation &Analytical Models

Extension of Existing Tools Performance Antipatterns

• Tools: PANDA‐Aemilia

• Challenge: create libraries of anti‐patterns for deployment models relevant to Ops


Pattern-basedIdentification

and Refactoring

References: Trubiani et al.

Research Roadmap




• Tools: LibReDe, Filling‐the‐Gap (FG)

• Challenges: inference synchronized to new commits; inference for all model parameters

Continuous ParameterizationResource Consumption Inference

21

Ripartizione dei Costi

Util

izzo

CPU

- %

020

4060

8010

0

Lun Mar Mer Gio Ven Sab DomMo Tu We Th Fr Sa Su

Util

izat

ion

[0-1

00%

]

Usage& Costs Splitting

Research Roadmap




Performance TraceabilityModel‐Driven Monitoring

• Tools: Kieker• AOP used to correlate monitoring and models

• Challenges: low‐overhead continuous monitoring, model‐based monitoring config.

23References: Van Hoorn et al.;

Model Traceability Tracking Runtime Adaptations

• Tools: Models@Runtime

• Challenges: model co‐evolution and adaptivity24

Cloud API Connectors

Deployer

Model

FacadeMOF CMD NotifierNotifier

WebSocket REST Shell

High level commands such as scale out, deploy, load a

model

ApplicationDeployment

Model

Based on the model

dependencies compute the list of actions to be performed by the connectors

A set of connectors to communicate with IaaS and PaaS providers and Cloud services

Reference: Ferry et al.

Performance Anomaly Detection Application Performance Signatures

• Methods: ACE, Demand‐Based, ...

• Challenge: continuous model‐aided generation of performance signatures

25

UtilizationSamples

Throughput

References: Sansottera et al.; Cherkasova et al.

• MODAclouds Project 9M€ (2012‐2015)• DevOps‐inspired tools for multi‐cloud software• Tools: http://www.modaclouds.eu/software

26

A Case Study

Access rating agencies

Order analysis

Get stock prices

CIM

Place order

Wait for ack from the stock market

Update customer trading account

OK

Fail

CCIM

MDE for Multi‐Cloud SoftwareFunctional Modelling


Order analysis

Get stock prices

Availability 24h/day

CIM

Place order



OK

Fail

Response time <0.5s

CCIM

MDE for Multi‐Cloud Software QoS Requirements


Order analysis

Get stock prices

CPIM

key-valued DB

A Reliable

ResourceB

High perf. Resource

Place order



OK

Fail

C Reliable

Resource

MDE for Multi‐Cloud Software Platform‐Independent Models



Order analysis

Get stock prices

Inst. numb. >2

CPIM

CPSM

key-valued DB

SimpleDB

A Reliable

ResourceB

High perf. Resource

A-1 Medium CPU

Instance

B-1 Large CPU

Instance

A-2 Worker Role

Large

B-2 Worker Role

Large

C-2Worker RoleExtra Large

Table Storage

Place order



OK

Fail

C Reliable

Resource

C-1 Large Memory

Instance

MDE for Multi‐Cloud Software Platform‐ and Technology‐Specific Models


Runtime Engine

DevOps‐Inspired Approach

Space 4Clouds(QoS modelling and analysis

tool)

CloudML 4Clouds(Deployment and Provisioning tool)

Venues 4Clouds(Decision SupportSystem)

Dev

Ops

Energizer4Clouds (runtime platform)

Data Mapping

tool

Runtime GUI

MODACloudMLModelling tool

Feasibility Study

Engineer

Application Developer

QoSEngineer

App Provider

Cloud App Admin

Cloud App

Cloud App

Resource model DB(incl info of providers)

Creator 4Clouds (design‐time IDE)

31

Tower4Clouds

(Monitoring)

SpaceOps4Clouds

(Self‐Adaptivity)

Addapters4Clouds(Runtime Services)

Filling the Gap Analysis

Ops‐Aware MDEFunctional Modelling

• MDE with cloud annotations: (MODA)CloudML• MDE extensions to describe cloud computing

32

1/24/2016 33

Ops‐Aware MDEExample: Amazon RDS

1/24/2016 Giuliano Casale ‐ Department of Computing 34

MonitoringRule

ViolationTrigger

CPIM DeploymentDiagram

MonitoringAnnotation

TraceabilityConfigurable Monitoring Rules

Performance Engine

• Transformation toQueueing Networks

• LINE fluid solver• SLA percentiles

SPACE4CloudsOptimization Engine

• Quantitative Analysis• Optimization QoS & SLA

Service and QoS model • Suggest deployment• Types of VMs• Number of VMsAmazon

Azure

Amazon

Azure

Google

Candidate Cloud Providers

Creator4Clouds IDE

Venues4Clouds

Search‐Based Deployment SPACE4Clouds+LINE

1/24/2016

1/24/2016 36

Search‐Based Deployment SPACE4Clouds

Solution Manager

MILP solver

Initial Solution Builder

MODACloudMLModel Initial solution

Optimized Solution

SPACE4Clouds

LINE

Multi‐thread connectors

MILP Solver connector

Cost and Feasibility Evaluator

Reference: D. Ardagna et al. A Multi-Model Optimization Framework for the Model Driven Design of Cloud Applications. SSBSE 2014. 61-76.

VM startuptimesCPU stealSLAs…

line5_2_b.mp4

VIDEO


Conclusion

Questions?

[email protected]

Performance Aware DevOps - BME-HITbuttyan/atnc/20160122_Casale_Devops.pdf · •Agile development...

Documents

Transcript of Performance Aware DevOps - BME-HITbuttyan/atnc/20160122_Casale_Devops.pdf · •Agile development...