PlanetLab: An open platform for developing,

deploying, and accessing planetary-scale services

http://www.planet-lab.org

Overview

Adapted from Peterson et al’s talks

Overview

• PlanetLab is a global research network that supports the development of new network services. Since the beginning of 2003, more than 1,000 researchers at top academic institutions and industrial research labs have used PlanetLab to develop new technologies for distributed storage, network mapping, peer-to-peer systems, distributed hash tables, and query processing.

PlanetLab Today

www.planet-lab.org

•838 machines spanning 4140 sites and 40 countries

• Supports distributed virtualization each of 600+ network services running in their own slice

Long-Running Services• Content Distribution

– CoDeeN: Princeton– Coral: NYU, Stanford– Cobweb: Cornell

• Storage & Large File Transfer– LOCI: Tennessee– CoBlitz: Princeton

• Information Plane– PIER: Berkeley, Intel– PlanetSeer: Princeton– iPlane: Washington

• DHT– Bamboo (OpenDHT): Berkeley, Intel– Chord (DHash): MIT

Services (cont)• Routing / Mobile Access

– i3: Berkeley– DHARMA: UIUC– VINI: Princeton

• DNS– CoDNS: Princeton– CoDoNs: Cornell

• Multicast– End System Multicast: CMU– Tmesh: Michigan

• Anycast / Location Service– Meridian: Cornell– Oasis: NYU

Services (cont)• Internet Measurement

– ScriptRoute: Washington, Maryland

• Pub-Sub– Corona: Cornell

• Email– ePost: Rice

• Management Services– Stork (environment service): Arizona– Emulab (provisioning service): Utah– Sirius (brokerage service): Georgia– CoMon (monitoring service): Princeton– PlanetFlow (auditing service): Princeton– SWORD (discovery service): Berkeley, UCSD

Usage Stats

• Slices: 600+• Users: 2500+• Bytes-per-day: 4 TB• IP-flows-per-day: 190M• Unique IP-addrs-per-day: 1M

Slices

Slices

Slices

User Opt-in

ServerNAT

Client

Per-Node View

Virtual Machine Monitor (VMM)

NodeMgr

LocalAdmin

VM1 VM2 VMn…

Virtualization

Virtual Machine Monitor (VMM)

NodeMgr

OwnerVM

VM1 VM2 VMn…

Linux kernel (Fedora Core)+ Vservers (namespace isolation)+ Schedulers (performance isolation)+ VNET (network virtualization)

Auditing serviceMonitoring servicesBrokerage servicesProvisioning services

Global View

…

…

…

PLC

Design Challenges• Minimize centralized control

without violating trust assumptions.

• Balance the need for isolation with the reality of scarce resources.

• Maintain a stable and usable system while continuously evolving it.

PlanetLab Architecture

• Node manger (one per node)– Create slices for service managers

• When service managers provide valid tickets

– Allocate resources for virtual-servers

• Resource Monitor (one per node)– Track node’s available resources– Tell agents about available resources

PlanetLab Architecture (cont’)• Agents (centralized)

– Track nodes’ free resources– Advertise resources to resource

brokers– Issue tickets to resource brokers

• Tickets may be redeemed with node managers to obtain the resource

PlanetLab Architecture (cont’)

• Resource Broker (per service)– Obtain tickets from agents on behalf of

service managers

• Service Managers (per service)– Obtain tickets from broker– Redeem tickets with node managers to

acquire resources– If resources can be acquired, start

service

Slice Management: Obtaining a Slice

Agent

Service Manager

Broker

Obtaining a Slice

Agent

Service Manager

BrokerResource Monitor

Obtaining a Slice

Agent

Service Manager

BrokerResource Monitor

Obtaining a Slice

Agent

Service Manager

BrokerResource Monitor

ticket

Obtaining a Slice

Agent

Service Manager

Broker

ticket

Resource Monitor

Resource Monitor

Obtaining a Slice

Agent

Service Manager

Broker

ticket

Resource Monitor

Resource Monitor

ticket

ticket

Obtaining a Slice

Agent

Service Manager

Broker

ticket

ticket

ticket

Obtaining a Slice

Agent

Service Manager

Broker

ticket

ticket

ticket

Obtaining a Slice

Agent

Service Manager

Broker

ticket

ticket

ticket

Obtaining a Slice

Service Manager

Broker

ticket

ticket

ticket

Obtaining a Slice

Agent

Service Manager

Broker

ticket

ticket

ticketNode Manager

Node Manager

Obtaining a Slice

Agent

Service Manager

Broker

ticket

Obtaining a Slice

Agent

Service Manager

Broker

ticket

Trust RelationshipsPrincetonBerkeleyWashingtonMITBrownCMUNYUEPFLHarvardHP LabsIntelNEC LabsPurdueUCSDSICSCambridgeCornell…

princeton_codeennyu_dcornell_beehiveatt_mcashcmu_esmharvard_icehplabs_donutlabidsl_pseprirb_phiparis6_landmarksmit_dhtmcgill_cardhuji_enderarizona_storkucb_bambooucsd_shareumd_scriptroute…

N x N

TrustedIntermediary

(PlanetLab Central )(PLC, agent)

Trust Relationships (cont)

Node Owner

PLCAgent

Service Developer(User)1

2

3

4

1) PLC expresses trust in a user by issuing it credentials to access a slice

2) Users trust PLC to create slices on their behalf and inspect credentials

3) Owner trusts PLC to vet users and map network activity to right user

4) PLC trusts owner to keep nodes physically secure

1. Each node boots from an immutable file system, loading a boot manager, a public key for PLC, and a node-specific secret key

2. User accounts are created through an authorized PI associated with each site

3. PLC runs an auditing service that records info about packet flows.

Decentralized Control• Owner autonomy

– owners allocate resources to favored slices– owners selectively disallow un-favored slices

• Delegation– PLC grants tickets that are redeemed at

nodes– enables third-party management services

• Federation– create “private” PlanetLabs

• now distribute MyPLC software package

– establish peering agreements

Resource Allocation• Decouple slice creation and resource

allocation– given a “fair share” (1/Nth) by default when

created– acquire/release additional resources over

time, including resource guarantees• Protect against thrashing and over-use

– Link bandwidth, upper bound on sustained rate (protect campus bandwidth)

– Memory, kill largest user of physical memory when swap at 85%

CoMon

• Performance monitoring services, providing a monitoring statistics for Planetlab at both a node level and a slice level.

• Two daemons running on each node, one for node-centric data and the other for slice-centric data

• The archived files contain the results of the checks every 5 minutes. They are available as http://summer.cs.princeton.edu/status/dump_cotop_YYYYMMDD.bz2 for the slice-centric data and http://summer.cs.princeton.edu/status/dump_comon_YYYYMMDD.bz2 for the node-centric data.

• The data collected can be accessed via query interface • http://comon.cs.princeton.edu

Node Availability