Espresso - SDN for Public InternetSam K. AldrinOn behalf of Google Technical Infrastructure and Google Cloud Platform

IETF, July 19th, 2017

Google Global Cache edge nodes

FASTER (US, JP, TW) 2016

Unity (US, JP) 2010SJC (JP, HK, SG) 2013

Points of presence >100

Network fiber

Google NetworkMore than a collection of data centers

#

#

Future regions and number of zones

Current regions and number of zones

3

3

2

3

3 3

3

3

24

3

3

2

Frankfurt

Singapore

S Carolina

N Virginia

Belgium

London

TaiwanMumbai

Sydney

OregonIowa

São Paulo

Finland

Tokyo

Montreal

California

Netherlands

3

3

33

Google Cloud RegionsAdding 11 new regions

Ubiquitous Cloud...10x Scaling

Datacenter

Next-gen disaggregation of storage, memory and compute

Campus & MetroCloud regions and campus expansion driving DC interconnect

WANCloud replication and bandwidth intensive cloud services (e.g., turnkey video, IoT)

10x10x 10x

Step Function Disruptions: Bandwidth, Latency, Availability, Predictability

B4WAN

Interconnect

Andromeda NFV and network

virtualization

JupiterDatacenter Networking

The Pillars of SDN @ Google

B4: [Jain et al, SIGCOMM 13] BwE: [Jain et al, SIGCOMM 15]

B4: Google's Software Defined WAN

10.1.4/24

VNET: 5.4/16

VNET: 192.168.32/24

VNET: 10.1.1/24 Load Balancing

DoS

ACLs

VPN

NFVInternal Network

Andromeda

ToR

Google Infrastructure Services

10.1.1/24

ToR

10.1.2/24

ToR

10.1.3/24

ToR

Watchtower

Saturn

Firehose 1.1

Google Datacenter Network InnovationAnd hardware scale that we could not buy

8

Time

Capa

city

Firehose 1.0

Jupiter

4 Post

1.3Pb/s clusters in 2013

B4WAN

Interconnect

Andromeda NFV and network

virtualization

JupiterDatacenter Networking

The Pillars of SDN @ Google

PublicInternet?

B4WAN

Interconnect

Andromeda NFV and network

virtualization

JupiterDatacenter Networking

The Pillars of SDN @ Google

Espresso SDN for public

Internet

Espresso in Context

B4

Jupiter Data CenterGoogle

Espresso in Context

B4

B2

Peering Metro

Jupiter Data CenterGoogle

Google

Espresso in Context

B4Espresso

B2

Internet

Peering Metro

User

Jupiter Data CenterGoogle

Google

Cloud 1.0Espresso

SDNPeering

RouterCentric

Protocols

Espresso: Before and After

Local viewConnectivity firstCoarse fault recovery

Per-metro and global viewApplication signalsReal-time optimization

Espresso Architecture Overview

Label-switched Fabric

BGP speaker

External Peer

Espresso Metro

Peering Fabric

eBGP Peering

Espresso Architecture Overview

Label-switched Fabric

HostHostHostHostHost

Host

Packet Processor

BGP speaker

External PeereBGP Peering

Espresso Metro

Labeled packets specify egress

HostHostHostHostHost

Peering Fabric

Espresso Architecture Overview

Label-switched Fabric

HostHostHostHostHost

Host

Packet Processor

LocalControl

Global Controller

BGP speaker

External PeereBGP Peering

Espresso Metro

Application Signals

Labeled packets specify egress

HostHostHostHostHost

Peering Fabric

• First things first: an insecure infrastructure is an unavailable infrastructure• Stability is more important than efficiency• Network management is critical• Configuration is hard• Automation matters but can be counter to availability

“Evolve or Die: High-Availability Design Principles Drawn from Google’s Network Infrastructure.” SIGCOMM 2016.

Availability is Paramount

The next wave of computing• Serverless compute in Cloud 3.0• IoT• Tightly coupled, general purpose

distributed computing

It’s time to put it all together• Agile Scale• Jitter• Isolation• Performance is great, but only

meaningful with availability, manageability, and velocity

Next Decade Challenges in Networking

Thank You!Thank You!