Expect the Unexpected - USENIX...Expect the Unexpected. Help > About John Arthorne...

21
Expect the Unexpected

Transcript of Expect the Unexpected - USENIX...Expect the Unexpected. Help > About John Arthorne...

Page 1: Expect the Unexpected - USENIX...Expect the Unexpected. Help > About John Arthorne Developer/Manager/SRE at Shopify Shopify Software for Commerce 30 → 120 SREs Average $1700 GMV

Expect the Unexpected

Page 2: Expect the Unexpected - USENIX...Expect the Unexpected. Help > About John Arthorne Developer/Manager/SRE at Shopify Shopify Software for Commerce 30 → 120 SREs Average $1700 GMV

Help > About

John Arthorne

● Developer/Manager/SRE at Shopify

Shopify

● Software for Commerce● 30 → 120 SREs● Average $1700 GMV / second

Page 3: Expect the Unexpected - USENIX...Expect the Unexpected. Help > About John Arthorne Developer/Manager/SRE at Shopify Shopify Software for Commerce 30 → 120 SREs Average $1700 GMV

●●●

What We’ll Cover

Page 4: Expect the Unexpected - USENIX...Expect the Unexpected. Help > About John Arthorne Developer/Manager/SRE at Shopify Shopify Software for Commerce 30 → 120 SREs Average $1700 GMV

high failure novelty rate

Page 5: Expect the Unexpected - USENIX...Expect the Unexpected. Help > About John Arthorne Developer/Manager/SRE at Shopify Shopify Software for Commerce 30 → 120 SREs Average $1700 GMV

●

●

●

Page 6: Expect the Unexpected - USENIX...Expect the Unexpected. Help > About John Arthorne Developer/Manager/SRE at Shopify Shopify Software for Commerce 30 → 120 SREs Average $1700 GMV

Transparent Response

●

●

●

Page 7: Expect the Unexpected - USENIX...Expect the Unexpected. Help > About John Arthorne Developer/Manager/SRE at Shopify Shopify Software for Commerce 30 → 120 SREs Average $1700 GMV

Incident Simulation

●

●

●

Page 8: Expect the Unexpected - USENIX...Expect the Unexpected. Help > About John Arthorne Developer/Manager/SRE at Shopify Shopify Software for Commerce 30 → 120 SREs Average $1700 GMV

Game Days

●

●

●

Page 9: Expect the Unexpected - USENIX...Expect the Unexpected. Help > About John Arthorne Developer/Manager/SRE at Shopify Shopify Software for Commerce 30 → 120 SREs Average $1700 GMV

Turn Rusty Knobs

●

●

●

Page 10: Expect the Unexpected - USENIX...Expect the Unexpected. Help > About John Arthorne Developer/Manager/SRE at Shopify Shopify Software for Commerce 30 → 120 SREs Average $1700 GMV

Automated Failure Tests

●

●

●

Page 11: Expect the Unexpected - USENIX...Expect the Unexpected. Help > About John Arthorne Developer/Manager/SRE at Shopify Shopify Software for Commerce 30 → 120 SREs Average $1700 GMV

Software Change Rate

People Change Rate

High

High

Low

Low

Page 12: Expect the Unexpected - USENIX...Expect the Unexpected. Help > About John Arthorne Developer/Manager/SRE at Shopify Shopify Software for Commerce 30 → 120 SREs Average $1700 GMV

Software Change Rate

People Change Rate

High

High

Low

Low

Incident TransparencyAutomated Tests

Game Days

Incident SimulationsRusty Knobs

Page 13: Expect the Unexpected - USENIX...Expect the Unexpected. Help > About John Arthorne Developer/Manager/SRE at Shopify Shopify Software for Commerce 30 → 120 SREs Average $1700 GMV

novel failures?

Page 14: Expect the Unexpected - USENIX...Expect the Unexpected. Help > About John Arthorne Developer/Manager/SRE at Shopify Shopify Software for Commerce 30 → 120 SREs Average $1700 GMV

Magic Recipe for Novel Failures

Page 15: Expect the Unexpected - USENIX...Expect the Unexpected. Help > About John Arthorne Developer/Manager/SRE at Shopify Shopify Software for Commerce 30 → 120 SREs Average $1700 GMV

Training Exercise Formula

●●●●●

Page 16: Expect the Unexpected - USENIX...Expect the Unexpected. Help > About John Arthorne Developer/Manager/SRE at Shopify Shopify Software for Commerce 30 → 120 SREs Average $1700 GMV
Page 17: Expect the Unexpected - USENIX...Expect the Unexpected. Help > About John Arthorne Developer/Manager/SRE at Shopify Shopify Software for Commerce 30 → 120 SREs Average $1700 GMV
Page 18: Expect the Unexpected - USENIX...Expect the Unexpected. Help > About John Arthorne Developer/Manager/SRE at Shopify Shopify Software for Commerce 30 → 120 SREs Average $1700 GMV
Page 19: Expect the Unexpected - USENIX...Expect the Unexpected. Help > About John Arthorne Developer/Manager/SRE at Shopify Shopify Software for Commerce 30 → 120 SREs Average $1700 GMV
Page 20: Expect the Unexpected - USENIX...Expect the Unexpected. Help > About John Arthorne Developer/Manager/SRE at Shopify Shopify Software for Commerce 30 → 120 SREs Average $1700 GMV

Summing Up

Page 21: Expect the Unexpected - USENIX...Expect the Unexpected. Help > About John Arthorne Developer/Manager/SRE at Shopify Shopify Software for Commerce 30 → 120 SREs Average $1700 GMV

Thank You!github.com/jarthorn/lego-incident-response

https://github.com/jarthorn/lego-incident-response

Acute Gynecological Findings on CT: Expect the Unexpected

Acute Gynecological Findings on CT: Expect the Unexpected

Expect the Unexpected - Issue and Crisis Management

Expect the Unexpected - Issue and Crisis Management

Hospitality Risk Management: Expect The Unexpected

Hospitality Risk Management: Expect The Unexpected

Lomo "Expect The Unexpected" Campaign

Lomo "Expect The Unexpected" Campaign

Expect the Unexpected! - USENIX · Expect the Unexpected! A process for managing unplanned work. Dan O’Boyle @Selfcommit Brian Artschwager @artschwagerb brian.artschwager.com

Expect the Unexpected! - USENIX · Expect the Unexpected! A process for managing unplanned work. Dan O’Boyle @Selfcommit Brian Artschwager @artschwagerb brian.artschwager.com

What to Expect When the Unexpected Happens: Social Media Communications ... · What to Expect When the Unexpected Happens: Social Media Communications Across Crises Alexandra Olteanu

What to Expect When the Unexpected Happens: Social Media Communications ... · What to Expect When the Unexpected Happens: Social Media Communications Across Crises Alexandra Olteanu

Expect the unexpected: Prepare for failures in microservices

Expect the unexpected: Prepare for failures in microservices

Expect the Unexpected! …And the Expected!

Expect the Unexpected! …And the Expected!

Expect the unexpected: qualitative and quantitative tools ... · Justen, Schippl, Höltl, Fleischer: Expect the unexpected – qualitative and quantitative tools and methods to detect

Expect the unexpected: qualitative and quantitative tools ... · Justen, Schippl, Höltl, Fleischer: Expect the unexpected – qualitative and quantitative tools and methods to detect

ManageMent excellence Expect the Unexpected

ManageMent excellence Expect the Unexpected

Expect the Unexpected - assets.kpmg€¦ · Expect the Unexpected: Building business value in a changing world: Executive Summary | 3 challenges. Growth could be . compromised and

Expect the Unexpected - assets.kpmg€¦ · Expect the Unexpected: Building business value in a changing world: Executive Summary | 3 challenges. Growth could be . compromised and

Expect the Unexpected Fish, Shellfish & Crustacean

Expect the Unexpected Fish, Shellfish & Crustacean

sewing and embroidery machine Expect the unexpected! AS ...

sewing and embroidery machine Expect the unexpected! AS ...

Expect the Unexpected: An Interactive Conversation on ......Expect the Unexpected: An Interactive Conversation on Marketing to Donors staffing and Structure Association Foundation

Expect the Unexpected: An Interactive Conversation on ......Expect the Unexpected: An Interactive Conversation on Marketing to Donors staffing and Structure Association Foundation

Expect the Unexpected - UNCW Faculty and Staff Web Pagespeople.uncw.edu/berlinert/research/documents/... · Expect the Unexpected Psycho and the Types of Planting and Payoff Todd

Expect the Unexpected - UNCW Faculty and Staff Web Pagespeople.uncw.edu/berlinert/research/documents/... · Expect the Unexpected Psycho and the Types of Planting and Payoff Todd

08 JICA mimura 20160225 Expect the Unexpected · 2016. 2. 25. · Expect the Unexpected DisastersExpect the Unexpected Disasters-Mainstreaming DRR into Development-20162252016.2.25

08 JICA mimura 20160225 Expect the Unexpected · 2016. 2. 25. · Expect the Unexpected DisastersExpect the Unexpected Disasters-Mainstreaming DRR into Development-20162252016.2.25

Expect the Unexpected? - World Trade Organization · "Expect the Unexpected"? LDC GATS Commitments as Internationally Credible Policy Indicators? The Example of Mali Dale Honeck WTO

Expect the Unexpected? - World Trade Organization · "Expect the Unexpected"? LDC GATS Commitments as Internationally Credible Policy Indicators? The Example of Mali Dale Honeck WTO

EXPECT THE UNEXPECTED

EXPECT THE UNEXPECTED

Age of Serendipity I expect the unexpected

Age of Serendipity I expect the unexpected

Your 2017 festival guide Your 2017 Expect the unexpected ... · PDF fileOVER 200 PERFORMANCES Your 2017 festival guide Expect the unexpected Your 2017 festival guide Expect the unexpected

Your 2017 festival guide Your 2017 Expect the unexpected ... · PDF fileOVER 200 PERFORMANCES Your 2017 festival guide Expect the unexpected Your 2017 festival guide Expect the unexpected