Automating ChaosExperiments In Production

Ali Basiri - Chaos Team@abasiri

Netflix

CDNControlPlane

Movie Bits Website, Apps,Signup, Login,

Browsing, SearchPlayback control,

Bookmarks, ...

Ali BasiriSoftware Engineer @ Netflix

● Chaos Engineer● Distributed Systems Engineer● Co-author of Principles of Chaos

Chaos Monkey

ServiceAvailability

Anatomy of a Failure

MovieInfo

APICDN

Selection

MovieInfo

APICDN

Selection

MovieInfo

APICDN

Selection

Fallback

MovieInfo

APICDN

Selection

Fallback

MovieInfo

APICDN

Selection

Fallback

MovieInfo

APICDN

Selection

Fallback

FIT

Request LevelFailure Injection

Request LevelFailure Injection

MovieInfo

APICDN

Selection

APIGatewayPersona-lization

Is API resilient to failure of Personalization?

APIGatewayPersona-lization

Randomly select 10% of requests to participate in experiment

APIGatewayPersona-lization

APIGatewayPersona-lization

if (shouldFail == true)

APIGatewayPersona-lization

if (shouldFail == true)

Even MoreAvailabilityFIT

Chaos Engineering is the discipline of experimenting on a distributed

system in order to build confidence in the system’s capability to withstand turbulent conditions in production.

ENGINEERINGCH∀OS

Stream Starts Per Second (SPS)

● Build a Hypothesis around Steady State Behavior

● Vary Real-world Events● Run Experiments in Production● Automate Experiments to Run

Continuously

Principles Of Chaos Engineeringhttp://principlesofchaos.org

Stream Starts Per Second (SPS)

ChAP

Goal:Chaos All The Things

APIGatewayPersona-lization

APIGatewayPersona-lization

API Control

API Exp

APIGatewayPersona-lization

API Control

API Exp

APIGatewayPersona-lization

API Control

API Exp

APIGatewayPersona-lization

API Control

API Exp

Select 1% of requests for controlSelect 1% of requests for experiment

APIGatewayPersona-lization

API Control

API Exp

APIGatewayPersona-lization

API Control

API Exp

if(shouldRoute == true)

APIGatewayPersona-lization

API Control

API Exp1%

1%

98%

APIGatewayPersona-lization

API Control

API Exp

if(shouldFail == true)

APIGatewayPersona-lization

API Control

API Exp

Stream Starts Per Second (SPS)

Fallback Metrics

Fallback Metrics

Fallback Metrics

CPU Utilization

Future Work on ChAP

Automated Canary Analysis

Detect divergence and stop early

Integrate with continuous delivery system

Clone multiple services to run an experiment

BA

B Con

B Exp

DC

C Con

C Exp

http://principlesofchaos.orghttp://chaos.community

Questions?