March 9, 2004 1

PlanetLab: An Open Laboratory for Introducing Disruptive Technology

into the Internet

David CullerUniversity of California, Berkeley

http://www.cs.berkeley.edu/~culler

March 9, 2004 2

Q: From where will the next 10 major advances in the internet emerge?

March 9, 2004 3

Innovator’s Dilemma• The Internet is an enormous success

story– commercially– impact on our daily lives– global reach

• Success has an unexpected cost: ossification– difficult to deploy disruptive technologies

correct vulnerabilities introduce new capabilities

March 9, 2004 4

A: ideas honed through process of “rough consensus and running code”

March 9, 2004 5

…PlanetLab

Intel BerkeleyIntel BerkeleyICIRICIR

MITMIT

PrincetonPrincetonCornellCornell

DukeDuke

UTUT

ColumbiaColumbiaUCSBUCSBUCBUCB

UCSDUCSDUCLAUCLA

UWUW

Intel SeattleIntel Seattle

KYKY

CambridgeCambridge

HarvardHarvard

GITGIT

UppsalaUppsalaCopenhagenCopenhagen

CMUCMU

UPennUPennWIWIChicagoChicagoUtahUtah

Intel ORIntel OR

UBCUBC

WashuWashu

ISIISI

IntelIntel

RiceRice

BolognaBologna

LancasterLancaster

St. LouisSt. Louis

UAUA

CanterburyCanterbury

SydneySydney

BerlinBerlinBelasBelas

TorontoToronto

CUHKCUHK

HUJHUJ

UCLUCL

TaiwanTaiwan

NizhnyNizhny

Towards 1,000 widely-distributed nodes– today: 360 machines, 148 sites, 20 countries, – 110 networks– at edge sites and network cross-roads– Universities, Companies, Nat’l labs

March 9, 2004 6

Today’s Internet

Limitations– the Internet is “opaque”

making it difficult to adapt to network conditions

– applications typically split into two pieces: client and server

Best-Effort Packet

Delivery Service

March 9, 2004 7

Internet services

March 9, 2004 8

Tomorrow’s Internet

Opportunities– multiple vantage points

anomaly detection, robust routing Adapt to load, failures, $

– Proximity: close wherever you are low latency, high bandwidth, predictable, reliable content distribution, data fusion

– multiple, independent domains survivable storage

Collection ofPlanetary-ScaleServices

March 9, 2004 9

Towards Planetary-Scale Services

• CDN and P2P just tip of the iceberg

• Research Community developing the architectural building blocks to enable many kinds of distributed services–scalable translation, –dist. storage, –dist. events, –instrumentation, –management

March 9, 2004 10

Key missing element – real hands-on experience

• Researchers had no vehicle to try out their next n great ideas in this space

• Lot’s of simulations • Lot’s of emulation on large clusters

– emulab, millennium, modelnet

• Lot’s of folks calling their 17 friends before the next deadline

– RON testbed

• but not the surprises and frustrations of experience at scale to drive innovation

March 9, 2004 11

Quick Growth• “Underground” meeting March 2002• Intel seeds effort

– First 100 nodes, Operational support

• First node up July 2002• By SOSP (March 2003) 25% of accepted

papers refer to PlanetLab• Each following conference

has seen dramatic load– OSDI– NDSI

March 9, 2004 12

Some Example Services

March 9, 2004 13

Berkeley: OceanStore`RAID’ distributed over the whole Internet

March 9, 2004 14

Washington - ScriptRoute

• Internet measurement tool• open, unauthenticated, community measurement

infrastructure vs closed, engineered service

UWGoogle

March 9, 2004 15

Princeton: CoDeeNOpen Content Distribution Network

BBB

B

B

B

B

A

A

A

A

AA

C

C

C

C

CCC

March 9, 2004 16

Berkeley – Internet Indirection Infrastructure

• Multicast, anycast, mobility, …

src

dstPut v,IDGet ID

March 9, 2004 17

NetBait Serendipity• Brent Chun built a simple http server on port 80 to

explain what planetlab was about and to direct inquiries to planet-lab.org

• It also logged requests• Sitting just outside the firewall of ~40 universities...• the worlds largest honey pot• the number of worm probes from compromised

machines was shocking• imagine the the epidemiology

• see netbait.planet-lab.org

March 9, 2004 18

One example

• The monthly code-red cycle in the large?• What happened in March 2003?

0

50

100

150

200

250

1/5/

2003

1/10

/200

3

1/15

/200

3

1/20

/200

3

1/25

/200

3

1/30

/200

3

2/4/

2003

2/9/

2003

2/14

/200

3

2/19

/200

3

2/24

/200

3

3/1/

2003

3/6/

2003

3/11

/200

3

3/16

/200

3

Pro

be

s p

er

da

y

Code Red

Nimda

March 9, 2004 19

No, not Iraq

• A new voracious worm appeared and displaced the older Code Red

0

200

400

600

800

1000

1200

1400

3/1/

2003

3/2/

2003

3/3/

2003

3/4/

2003

3/5/

2003

3/6/

2003

3/7/

2003

3/8/

2003

3/9/

2003

3/10

/200

3

3/11

/200

3

3/12

/200

3

3/13

/200

3

3/14

/200

3

3/15

/200

3

3/16

/200

3

3/17

/200

3

3/18

/200

3

3/19

/200

3

3/20

/200

3

Pro

be

s p

er

da

y

Code Red

Nimda

Code Red II.F

March 9, 2004 20

Intel Netbait - March 2003

March 9, 2004 21

Internet Public Health

March 9, 2004 22

Key Concept: Overlay networks

March 9, 2004 23

Overlay network routing

March 9, 2004 24

Evolving the Internet• Add a new layer to the network architecture

– overlay networks

• Challenge– how to innovate & deploy at scale

purpose-built virtual networks that use the existing Internet for transmission

the Internet was once deployed as an overlay on

top of the telephony network

overlay

Internet

March 9, 2004 25

PlanetLab is…

Intel BerkeleyIntel BerkeleyICIRICIR

MITMIT

PrincetonPrincetonCornellCornell

DukeDuke

UTUT

ColumbiaColumbiaUCSBUCSBUCBUCB

UCSDUCSDUCLAUCLA

UWUW

Intel SeattleIntel Seattle

KYKY

CambridgeCambridge

HarvardHarvard

GITGIT

UppsalaUppsalaCopenhagenCopenhagen

CMUCMU

UPennUPennWIWIChicagoChicagoUtahUtah

Intel ORIntel OR

UBCUBC

WashuWashu

ISIISI

IntelIntel

RiceRice

BolognaBologna

LancasterLancaster

St. LouisSt. Louis

UAUA

CanterburyCanterbury

SydneySydney

BerlinBerlinBelasBelas

TorontoToronto

CUHKCUHK

HUJHUJ

UCLUCL

TaiwanTaiwan

NizhnyNizhny

More than a bunch of machines spread around the world

March 9, 2004 26

PlanetLab is…

A common software architecture• OS running on each node• Remote installation mechanism• Remote management capability• Central account management service

March 9, 2004 27

PlanetLab is…

A network testbed • 450 active research projects

– experiment at scale– experiment under real-world conditions– potential for real workloads and users

A deployment platform• 15 continuously running services

design deploy

measure

March 9, 2004 28

PlanetLab is…

A microcosm of the next Internet

• Fold services back into PlanetLab– evolve core technologies to support overlays and

slices– discover common sub-services

• Long-term goals– become the way users interact with the Internet– define standards that support multiple

“PlanetLabs”

March 9, 2004 29

Software Architecture

• Distributed virtualization– slice a network of virtual machines– isolation

isolate services from each other protect the Internet from PlanetLab

• Unbundled Management– OS defines only local (per-node) behavior

global (network-wide) behavior implemented by services

– multiple competing services running in parallel shared, unprivileged interfaces

March 9, 2004 30

Slices

March 9, 2004 31

Per-Node View

Virtual Machine Monitor (VMM)

NodeMgr

LocalAdmin

VM1 VM2 VMn…

March 9, 2004 32

Virtualization• Hypervisors (e.g., VMWare)

– don’t scale well– don’t need multi-OS functionality

• Paravirtualization (e.g., Xen, Denali)– not yet mature

• Virtualize at system call interface (e.g., Jail, Vservers)– reasonable compromise– doesn’t provide the isolation that hypervisors do

• Unix processes– isolation is problematic

• Java Virtual Machine– too high-level

March 9, 2004 33

PlanetLab VMM• Linux: significant mind-share• Vservers: virtualizes at the system call

interface– each vserver runs in its own security context

private UID/GID name space limited superuser capabilities (e.g., no CAP_NET_RAW)

– uses chroot for file system isolation– scales to 1000 vservers per node (29MB each)

• Plkmod: enforces isolation– processor and link schedulers, disk quotas– network virtualization

safe raw sockets (UDP, TCP, ICMP, GRE) rate limits exceptional packets, unique addrs/ports

March 9, 2004 34

Infrastructure Services• Brokerage Services

– buy, sell, trade, pool resources– least mature / greatest potential (market-based)

• Environment Services– create / maintain programming environments– challenge: space efficiency

• Monitoring Services– resource discovery– network / service health– adaptive applications

• Routing Underlay– discover network topology

March 9, 2004 35

PLC: Brokerage Service

create_slice(name, credentials)

delete_slice(name, credentials)

assign_owners(name, ssh_keys[ ], credentials)

set_state(name, boot_script, credentials)

set_resources(name, share, credentials)

instantiate_slice(name, nodes[ ], credentials)

rcap = acquire(rspec)

bind(name, rcap)

March 9, 2004 36

Stork: Environment Service

• Defines a boot_script to be run when VM inits– invokes Stork

• Maintains a repository of packages– those registered by a client service + dependencies

• Shared package area on each node– retrieves one copy of each required package– installation scripts run in context of client VM

• Optionally…– manages user accounts for client slices– establishes tunnels to neighboring nodes

March 9, 2004 37

Monitoring Service – Sophia, Pier,…

S S S… A A A…

Information Plane

observe analyze react

Distributed query processor

March 9, 2004 38

Pluto: Routing Underlay

Service Overlay Networks

Library of Routing Services

Topology Probing Kernel

Raw Topology Information

Primitives

Expense

Scope

March 9, 2004 39

Representative Sample of the Internet?

March 9, 2004 40

A Rich Research Agenda

• Global System Architecture– Slices, management, distribution,

• Network measurement– Scriptroute, PlanetProbe, I3, etc.

• Application-level multicast– ESM, Scribe, TACT, etc.

• Distributed Hash Tables– Chord, Tapestry, Pastry, Bamboo,

etc.• Wide-area distributed storage

– Oceanstore, SFS, CFS, Palimpsest, IBP

• Resource allocation– Sharp, Slices, XenoCorp,

Automated contracts• Distributed query processing

– PIER, IrisLog, Sophia, etc.

• Content Dist. Networks– CoDeeN, ESM, UltraPeer emulation,

Gnutella mapping• Management and Monitoring

– Ganglia, InfoSpect, Scout Monitor, BGP Sensors, etc.

• Overlay Networks– RON, ROM++, ESM, XBone, ABone,

etc.• Virtualization and Isolation

– Xen, Denali, VServers, SILK, Mgmt VMs, etc.

• Router Design implications– NetBind, Scout, NewArch, Icarus,

etc. • Testbed Federation

– NetBed, RON, XenoServers• Etc., etc., etc.

March 9, 2004 41

Confluence of Technologies • Cluster-based scalable distribution, remote execution,

management, monitoring tools– UCB Millennium, OSCAR, ..., Utah Emulab, ...

• CDNS and P2Ps– Gnutella, Kazaa, ...

• Proxies routine• Virtual machines & Sandboxing

– VMWare, Janos, Denali,... web-host slices (EnSim)• Overlay networks becoming ubiquitous

– xBone, RON, Detour... Akamai, Digital Island, .... • Service Composition Frameworks

– yahoo, ninja, .net, websphere, Eliza• Established internet ‘crossroads’ – colos• Web Services / Utility Computing• Authentication infrastructure (grid)• Packet processing (layer 7 switches, NATs, firewalls)• Internet instrumentation

March 9, 2004 42

Current Institutions (partial)Academia Sinica, TaiwanBoston UniversityCaltechCarnegie Mellon UniversityChinese Univ of Hong KongColumbia UniversityCornell UniversityDatalogisk Institut CopenhagenDuke UniversityGeorgia TechHarvard UniversityHP LabsIntel ResearchJohns HopkinsLancaster UniversityLawrence Berkeley LaboratoryMITMichigan State UniversityNational Tsing Hua Univ.New York University

Northwestern University

Princeton UniversityPurdue UniversityRensselaer Polytechnic Inst.Rice UniversityRutgers UniversityStanford UniversityTechnische Universitat BerlinThe Hebrew Univ of JerusalemUniversity College LondonUniversity of ArizonaUniversity of BaselUniversity of BolognaUniversity of British ColumbiaUC BerkeleyUCLAUC San DiegoUC Santa BarbaraUniversity of CambridgeUniversity of CanterburyUniversity of ChicagoUniversity of Illinois

University of KansasUniversity of KentuckyUniversity of MarylandUniversity of MassachusettsUniversity of MichiganUniversity of North CarolinaUniversity of PennsylvaniaUniversity of RochesterUSC / ISIUniversity of Technology SydneyUniversity of TennesseeUniversity of Texas

University of Toronto

University of Utah

University of Virginia

University of Washington

University of Wisconsin

Uppsala University, Sweden

Washington University in St Louis

Wayne State University

March 9, 2004 43

What Planet-Lab is about?• Create the open infrastructure for invention of the

next generation of wide-area (“planetary scale”) services– post-cluster, post-yahoo, post-CDN, post-P2P, ...

• Potentially, the foundation on which the next Internet can emerge– think beyond TCP/UDP/IP + DNS + BGP + OSPF... as to what the net

provides– building-blocks upon which services and applications will be based– “the next internet will be created as an overlay in the current one” (NRC)

• A different kind of network testbed– not a collection of pipes and giga-pops– not a distributed supercomputer – geographically distributed network services– alternative network architectures and protocols

• Focus and Mobilize the Network / Systems Research Community to define the emerging internet

March 9, 2004 44

Join the fun ... • It is just beginning

– towards a representative sample of the internet (1,000s)

• PlanetLab consortium recently formed– Princeton, UCB, UWash– Intel, HP, Google, …

• Hands-on experience with wide-area services at scale is mothering tremendous innovation

– nothing “just works” in the wide-area at scale– Joining forces we all learn more

www.planet-lab.org