{

Erlang :Because S**t happens

Mahesh Paolini-Subramanya (@dieswaytoofast)V.P. Ubiquiti Networks

My Vacation

(Actually, the day before)

A small failure…

The Horror! The Horror!

Why are my calls failing?

You better call me back!

I’m still p***ed off!

And you’re stupid Apps don’t work!

The Horror! The Horror!

Surely you Tested?

1000 year floods

Fault Tolerance

Concurrency

The Big SixFrom http://www.erlang.org/download/armstrong_thesis_2003.pdf

Concurrency

Fault detection

The Big SixFrom http://www.erlang.org/download/armstrong_thesis_2003.pdf

Concurrency

Fault detection

Fault identification

The Big SixFrom http://www.erlang.org/download/armstrong_thesis_2003.pdf

Concurrency

Fault detection

Fault Identification

Error Encapsulation

The Big SixFrom http://www.erlang.org/download/armstrong_thesis_2003.pdf

Concurrency

Fault detection

Fault Identification

Error Encapsulation

Code upgrade

The Big SixFrom http://www.erlang.org/download/armstrong_thesis_2003.pdf

Concurrency

Fault detection

Fault Identification

Error Encapsulation

Code upgrade

Stable Storage

The Big SixFrom http://www.erlang.org/download/armstrong_thesis_2003.pdf

erlang…

Concurrency

Fault detection

Fault Identification

Error Encapsulation

Code upgrade

Stable Storage

The Big SixFrom http://www.erlang.org/download/armstrong_thesis_2003.pdf

Concurrency Oriented

Concurrency Oriented

Concurrency

Fault detection

Fault Identification

Error Encapsulation

Code upgrade

Stable Storage

The Big SixFrom http://www.erlang.org/download/armstrong_thesis_2003.pdf

Fault Detection

Concurrency

Fault detection

Fault Identification

Error Encapsulation

Code upgrade

Stable Storage

The Big SixFrom http://www.erlang.org/download/armstrong_thesis_2003.pdf

Stack Traces?

Immutable Variables

X = 1.

Immutable Variables

X = 1.

X = 2.

Immutable Variables

X = 1.

X = 2.

X = X + 1.

Fault Identification

Concurrency

Fault detection

Fault Identification

Error Encapsulation

Code upgrade

Stable Storage

The Big SixFrom http://www.erlang.org/download/armstrong_thesis_2003.pdf

Let It Crash

BEAM!

Faster to create

JVM is not necessarily your friend!

Concurrency

Fault detection

Fault Identification

Error Encapsulation

Code upgrade

Stable Storage

The Big SixFrom http://www.erlang.org/download/armstrong_thesis_2003.pdf

Code Upgrade

Live!

Hot SwappingCode Upgrade

Concurrency

Fault detection

Fault Identification

Error Encapsulation

Code upgrade

Stable Storage

The Big SixFrom http://www.erlang.org/download/armstrong_thesis_2003.pdf

The Intangibles

4x – 10x less code

Code Size

Faster to create

4x – 10x less code

Faster to create

Easier to reason about

4x – 10x less code

Faster to create

Easier to reason about

Fewer bugs

4x – 10x less code

Faster to create

Easier to reason about

Fewer bugs

Speedy refactoring

4x – 10x less code

The Shell is our friend

Live Debugging

Predictability

Performance

Fault Tolerance - Systems

Romney 2012

Fault Tolerance - Systems

Concurrency

Error encapsulation

Fault detection

Fault identification

Code upgrade

Stable Storage

The Big Six - Systems

Concurrency

Error encapsulation

Fault detection

Fault identification

Code upgrade

Stable Storage

The Big Six - Systems

Concurrency

Error encapsulation

Fault detection

Fault identification

Code upgrade

Stable Storage

The Big Six - Systems

Loose Coupling?

Breeds Trust

Loose Coupling

Loose Coupling

Breeds Trust

Devote more brainpower to specific areas

Loose Coupling

Loose Coupling

Breeds Trust

Devote more brainpower to specific areas

No. of bugs/line is constant

Loose Coupling

Performance

60 - 90% of all SW projects fail

10 – 25% of all SW projects get abandoned

Fault Tolerance

Concurrency

Error encapsulation

Fault detection

Fault identification

Code upgrade

Stable Storage

The Big Six - Systems

Monitoring?

Dashboards

Monitoring?

Dashboards

Out of band systems

Monitoring?

Supervision

Dashboards

Out of band systems

Polyglot safety

Monitoring?

Concurrency

Error encapsulation

Fault detection

Fault identification

Code upgrade

Stable Storage

The Big Six - Systems

Concurrency

Error encapsulation

Fault detection

Fault identification

Code upgrade

Stable Storage

The Big Six - Systems

Concurrency

Error encapsulation

Fault detection

Fault identification

Code upgrade

Stable Storage

The Big Six - Systems

No battle plan survives contact with the enemy

Not just about Systems

Fault Tolerance

Fault Tolerance

People

Vendors

Fault Tolerance

People

Vendors

Fraud

Fault Tolerance

The BusinessBeware the Black Swan

Is It Safe?

erlang…

mahesh@dieswaytoofast.com @dieswaytoofastQuestions

Coda

Active Queue Management

Queues

Queues

Queues

Queues

Can you recover quickly?

Buffer-bloat doesn’t matter, right?

Once up, can you deal with the backlog?

Back-pressure isn’t an issue, right?

Queues

Can you recover quickly?

Buffer-bloat doesn’t matter, right?

Once up, can you deal with the backlog?

Back-pressure isn’t an issue, right?

Queues

Programmable

Behavioral

Self Managed

Something’s gotta give

Tail Drop

God(category – TCP/IP)

RED

RED

Newark Airport

FRED

RED-PD

WRED

RED – Many many more

SRED RRED ARED (and Blue!) CHOKe

Special Mention

RED in a different Light

SERIOUSLY!

RED in a different Light

CoDel and fq_codel

mahesh@dieswaytoofast.com @dieswaytoofastQuestions