V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Drinking from the Firehose
Drinking from the
FirehoseMahesh Paolini-Subramanya (@dieswaytoofast)V.P. R&D, Ubiquiti Networks
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Drinking from the Firehose
Drinking from the
Firehose
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Drinking from the Firehose
Drinking from the
FirehoseAka: “Scale Matters”
Drinking from the Firehose
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
ARCHITECTURE
DEVICE
Drinking from the Firehose
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Architecture
DEVICE
STATUS
Drinking from the Firehose
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Scale-Fail
DEVICE
STATUS
Drinking from the Firehose
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Data Stores
Drinking from the Firehose
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Data Stores
• State of the World
Drinking from the Firehose
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Data Stores
• State of the World
• Historical Data
Drinking from the Firehose
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Data Stores
• State of the World
• Historical Data
• Configurations
Drinking from the Firehose
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Data Stores
• State of the World
• Historical Data
• Configurations
• Billing
AGILITY
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Drinking from the Firehose
Fault Tolerance
From
http://www.erlang.org/download/armstrong_thesis_2003.pdf
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Drinking from the Firehose
The Big Six
Concurrency
From
http://www.erlang.org/download/armstrong_thesis_2003.pdf
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Drinking from the Firehose
The Big Six
Concurrency
Fault detection
From
http://www.erlang.org/download/armstrong_thesis_2003.pdf
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Drinking from the Firehose
The Big Six
Concurrency
Fault detection
Fault Identification
From
http://www.erlang.org/download/armstrong_thesis_2003.pdf
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Drinking from the Firehose
The Big Six
Concurrency
Fault detection
Fault Identification
Error Encapsulation
From
http://www.erlang.org/download/armstrong_thesis_2003.pdf
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Drinking from the Firehose
The Big Six
Concurrency
Fault detection
Fault Identification
Error Encapsulation
Code upgrade
From
http://www.erlang.org/download/armstrong_thesis_2003.pdf
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Drinking from the Firehose
The Big Six
Concurrency
Fault detection
Fault Identification
Error Encapsulation
Code upgrade
Stable Storage
From
http://www.erlang.org/download/armstrong_thesis_2003.pdf
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Drinking from the Firehose
The Big Six
Concurrency
Fault detection
Fault Identification
Error Encapsulation
Code upgrade
Stable Storage
From
http://www.erlang.org/download/armstrong_thesis_2003.pdf
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Drinking from the Firehose
The Big Six
Concurrency
Fault detection
Fault Identification
Error Encapsulation
Code upgrade
Stable Storage
From
http://www.erlang.org/download/armstrong_thesis_2003.pdf
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Drinking from the Firehose
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Drinking from the Firehose
** exception error: no match of
right hand side value
{error,unhappy}
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Drinking from the Firehose
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Drinking from the Firehose
The Big Six
Concurrency
Fault detection
Fault Identification
Error Encapsulation
Code upgrade
Stable Storage
From
http://www.erlang.org/download/armstrong_thesis_2003.pdf
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Drinking from the Firehose
** exception error: no match of
right hand side value
{error,unhappy}
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Drinking from the Firehose
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Drinking from the Firehose
The Big Six
Concurrency
Fault detection
Fault Identification
Error Encapsulation
Code upgrade
Stable Storage
From
http://www.erlang.org/download/armstrong_thesis_2003.pdf
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Drinking from the Firehose
The Big Six
Concurrency
Fault detection
Fault Identification
Error Encapsulation
Code upgrade
Stable Storage
From
http://www.erlang.org/download/armstrong_thesis_2003.pdf
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Drinking from the Firehose
The Big Six
Concurrency
Fault detection
Fault Identification
Error Encapsulation
Code upgrade
Stable Storage
From
http://www.erlang.org/download/armstrong_thesis_2003.pdf
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Drinking from the Firehose
The Fault Tolerant System
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Drinking from the Firehose
The Big Six
Concurrency
Fault detection
Fault Identification
Error Encapsulation
Code upgrade
Stable Storage
From
http://www.erlang.org/download/armstrong_thesis_2003.pdf
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Drinking from the Firehose
The Big Six
Concurrency
Fault detection
Fault Identification
Error Encapsulation
Code upgrade
Stable Storage
From
http://www.erlang.org/download/armstrong_thesis_2003.pdf
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Drinking from the Firehose
The Big Six
Concurrency
Fault detection
Fault Identification
Error Encapsulation
Code upgrade
Stable Storage
From
http://www.erlang.org/download/armstrong_thesis_2003.pdf
LOOSE C
OUPLING
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Drinking from the Firehose
Loose Coupling
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Drinking from the Firehose
Loose Coupling
Breeds Trust
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Drinking from the Firehose
Loose Coupling
Breeds Trust
Devotes Brainpower appropriately
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Drinking from the Firehose
Loose Coupling
Breeds Trust
Devotes Brainpower appropriately
Minimizes bugs
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Drinking from the Firehose
Fault Tolerance
60 - 90% of all SW projects fail
10 – 25% of all SW projects get abandoned
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Drinking from the Firehose
The Big Six - Systems
Concurrency
Fault detection
Fault Identification
Error Encapsulation
Code upgrade
Stable Storage
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Drinking from the Firehose
The Big Six - Systems
Concurrency
Fault detection
Fault Identification
Error Encapsulation
Code upgrade
Stable Storage
MONIT
ORING
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Drinking from the Firehose
Monitoring?
Monitoring?• Dashboards
Monitoring?• Dashboards
• Out of band systems
Monitoring?• Dashboards
• Out of band systems
• Polyglot safety
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Drinking from the Firehose
The Big Six - Systems
Concurrency
Fault detection
Fault Identification
Error Encapsulation
Code upgrade
Stable Storage
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Drinking from the Firehose
The Big Six - Systems
Concurrency
Fault detection
Fault Identification
Error Encapsulation
Code upgrade
Stable Storage
Polyglo
t
Persis
tence
Drinking from the Firehose
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Polyglot Persistence
Scale-Fail
DEVICE
STATUS
Drinking from the Firehose
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Data Stores
• State of the World
• Historical Data
• Configurations
• Billing
Drinking from the Firehose
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Data Stores
• State of the World
Drinking from the Firehose
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Data Stores
• State of the World ElasticSearch
Drinking from the Firehose
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Data Stores
• State of the World ElasticSearch
• Historical Data
Drinking from the Firehose
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Data Stores
• State of the World ElasticSearch
• Historical Data Cassandra
Drinking from the Firehose
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Data Stores
• State of the World ElasticSearch
• Historical Data Cassandra
• Configurations
Source: Wholly unscientific research by me.
Drinking from the Firehose
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Data Stores
• State of the World ElasticSearch
• Historical Data Cassandra
• Configurations Riak
Drinking from the Firehose
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Data Stores
• State of the World ElasticSearch
• Historical Data Cassandra
• Configurations Riak
• Billing
Drinking from the Firehose
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Data Stores
• State of the World ElasticSearch
• Historical Data Cassandra
• Configurations Riak
• Billing Postgres
Drinking from the Firehose
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Data Stores
• State of the World ElasticSearch
• Historical Data Cassandra
• Configurations Riak
• Billing Postgres
Polyglo
t
Persis
tence
Drinking from the Firehose
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Data Stores
• State of the World ElasticSearch
• Historical Data Cassandra
• Configurations Riak
• Billing Postgres
LOOSE C
OUPLING
Drinking from the Firehose
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Data Stores
• State of the World ElasticSearch
• Historical Data Cassandra
• Configurations Riak
• Billing Postgres
MONIT
ORING
1
1
1
Drinking from the Firehose
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Split Brain
• Its ubiquitous (thanks aphyr)
Drinking from the Firehose
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Split Brain
• Its ubiquitous (thanks aphyr)
• Zookeeper doesn’t necessarily help
Drinking from the Firehose
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Split Brain
• Its ubiquitous (thanks aphyr)
• Zookeeper doesn’t necessarily help
• Moving the problem around doesn’t necessarily help
Cheating classroom)
Drinking from the Firehose
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Cheating
• Stale status messages are OK
Drinking from the Firehose
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Cheating
• Stale status messages are OK
• Charts can miss data
Drinking from the Firehose
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Cheating
• Stale status messages are OK
• Charts can miss data
• Billing is sharded
Drinking from the Firehose
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Cheating
• Stale status messages are OK
• Charts can miss data
• Billing is sharded
• Availability zones (google.co.uk)
Scaling
Drinking from the Firehose
V 1.0 © Ubiquiti Networks, Inc. All Rights Reserved
Drinking from the Firehose
Mahesh Paolini-Subramanya
(@dieswaytoofast)
V.P. R&D, Ubiquiti Networks
Top Related