1

FRED HEBERT

THE ZEN OF ERLANG

2

LET IT CRASH

EurekaPresentation 3

BLOW IT UP

ROCKET SCIENCE

EurekaPresentation 4

CONTROLLED BURNS

FIGHT FIRE WITH FIRE

5

PROCESSES

FULLY ISOLATED

EurekaPresentation 6

ASYNCHRONOUS

MESSAGE PASSING

7

LINKS & MONITORS

CODIFYING DEPENDENCIES

EurekaPresentation 8

PREEMPTIVE SCHEDULING

DON’T TRUST THE DEVS

EurekaPresentation 9

FAILOVERS

NETWORK AWARE

10

LET IT CRASH

EurekaPresentation 11

SUPERVISION TREES

STRUCTURE INTERACTIONS

12

Old

New

1 13 32

Old

New

1 3 1 2 3

Old

New

1 1 13 2 3

SUPERVISORSst rategies

13

WHY RESTARTING

WORKSHEISENBUGS & FRIENDS

14

REPEATABLE TRANSIENT

CORE FEATURE EASY HARD

SECONDARY FEATURE

EASY, OFTEN OVERLOOKED

HARD

Why restart ing works

EASE OF FINDING BUGS IN DEVELOPMENT

15

REPEATABLE TRANSIENT

CORE FEATURE SHOULD NEVER ALL THE TIME

SECONDARY FEATURE PRETTY OFTEN ALL THE TIME

Why restart ing works

BUGS THAT HAPPEN IN PRODUCTION

16

REPEATABLE TRANSIENT

CORE FEATURE NO YES

SECONDARY FEATURE DEPENDS YES

Why restart ing works

BUGS HANDLED BY RESTARTS

17

“I LIKE STATICALLY TYPED LANGUAGES, AND I RESTART MY

DAEMON AFTER UNHANDLED EXCEPTIONS. WHAT IS ERLANG

GONNA WIN ME ON FAULT TOLERANCE?”

- FORUM USER “Notorious b.s.d.”

18

ROOT

TALLY

DISTRICT

QUEBEC ONTARIO …

COUNT OCR

C_OCR

WORKER POOL

STORAGE

CACHE

POOL WORKER

POOL WORKER

POOL WORKER

WEB SERVER

LIVE REPORTS

SESSIONS

…WEB

WORKERWEB

WORKERWEB

WORKER

SUPERVISION TREES

19

ROOT

TALLY

DISTRICT

QUEBEC ONTARIO …

COUNT OCR

C_OCR

WORKER POOL

STORAGE

CACHE

POOL WORKER

POOL WORKER

POOL WORKER

WEB SERVER

LIVE REPORTS

SESSIONS

…WEB

WORKERWEB

WORKERWEB

WORKER

SUPERVISION TREES

20

ROOT

TALLY

DISTRICT

QUEBEC ONTARIO …

COUNT OCR

C_OCR

WORKER POOL

STORAGE

CACHE

POOL WORKER

POOL WORKER

POOL WORKER

WEB SERVER

LIVE REPORTS

SESSIONS

…WEB

WORKERWEB

WORKERWEB

WORKER

SUPERVISION TREES

REPORT_APPTALLY_APP

STORAGE_APP

21

but probably not…

HOPEFULLY

SLEEP AT NIGHT

EurekaPresentation 22

EXPECT FAILURE

23

DANGER ZONE

laid-back area

The r i sk of gett ing comfortable

WELL-DEFINED OPERATING CONDITIONS ERODE OVER TIME

EurekaPresentation 25

HOW THINGS INTERACT

IN A NUTSHELL

27

Total ly advert is ing f ree books to you a l l

QUESTIONS?

@MONONCQC