The CoDeeN Content Distribution Network

Vivek S. Pai, Limin Wang, KyoungSoo Park, Ruoming Pang, Larry PetersonPrinceton UniversityAugust 12, 2003

Aug 12, 2003 CoDeeN Overview - IRIS/PlanetLab 2

Content Distribution Networks

Replicates Web content broadlyRedirects clients to “best” copy

Load, locality, proximityOffloads work from origin serversMultiplexes load spikes

Reduces overprovisioningEx: Akamai, Mirror Image, Speedera

Aug 12, 2003 CoDeeN Overview - IRIS/PlanetLab 3

What Does It Do?An Academic Content Distribution Network

Redirects/caches HTTP requestsBased on our OSDI 2002 paper on CDN performance

An Open Proxy NetworkProbably the largest in existence

Aug 12, 2003 CoDeeN Overview - IRIS/PlanetLab 4

Who Is The Target Audience?Now

Users wanting better performancePeople seeking “anonymity”

NextContent providers seeking load sharing

LaterGeneral support for absorbing flash crowdsAvoid the “Slashdot Effect”

Aug 12, 2003 CoDeeN Overview - IRIS/PlanetLab 5

How Does It Work?Server surrogates (proxies) on most North American sites

Originally everywhere, but we cut backClients specify proxy to use

Cache hits served locallyCache misses forwarded to CoDeeN nodes• Maybe forwarded to origin servers

Aug 12, 2003 CoDeeN Overview - IRIS/PlanetLab 6

Request Forwarding

Aug 12, 2003 CoDeeN Overview - IRIS/PlanetLab 7

When Will It Be Ready?January – development started

Reliability & stability major concernsMarch – stable enough for daily useApril – security problems begin

Shut down for one monthJune – Restarted “beta”Expecting “production” soon

Aug 12, 2003 CoDeeN Overview - IRIS/PlanetLab 8

Decisions – Good & BadUse commercial proxy with API [USITS 2003]

Good – mostly layer 7 concernsBad – limits deployment size (donated licenses)

Deployment on PlanetLabGood – otherwise impossible“Bad” – vulnerable to other experiments

Allow open accessGood – generates real trafficBad – some traffic just plain mean

Aug 12, 2003 CoDeeN Overview - IRIS/PlanetLab 9

Lots of Malicious TrafficSpammers

SMTP tunnels, POST forms, IRC channelsBandwidth hogs

Google crawls, steganographers, X-PacificHackers & Spreaders

Yahoo dictionary attacks, IIS vuln testsContent thieves

E-journals/databases, local content

Restrict ports & HTTP methods

Multi-scale req & bw accounting

Signature database & Robot test

Determine location & privilege

Aug 12, 2003 CoDeeN Overview - IRIS/PlanetLab 10

Protecting Privilege

Aug 12, 2003 CoDeeN Overview - IRIS/PlanetLab 11

Attempted SMTP Tunnels/Day

Aug 12, 2003 CoDeeN Overview - IRIS/PlanetLab 12

By The Numbers…Restarted in late May

In continuous operationStats from first 8 weeks

Over 59,000 unique IPs as clientsOver 24 million requests servicedValid rates up to 15K reqs/hourRoughly 1 million reqs/day aggregate

Aug 12, 2003 CoDeeN Overview - IRIS/PlanetLab 13

More Production InfoAbout 2000 lines of code

About ¼ is actual decision logicUptimes limited by upgrades

Generally 1-2 times/weekDowntimes of 20 seconds/node

Currently on ~40 nodes

Aug 12, 2003 CoDeeN Overview - IRIS/PlanetLab 14

Daily Requests (Serviced)

Aug 12, 2003 CoDeeN Overview - IRIS/PlanetLab 15

Welcome

Aug 12, 2003 CoDeeN Overview - IRIS/PlanetLab 16

Avoiding

sorted by # avoiding

Aug 12, 2003 CoDeeN Overview - IRIS/PlanetLab 17

Load

sorted by # load average

Aug 12, 2003 CoDeeN Overview - IRIS/PlanetLab 18

Total

sorted by # total req rate

Aug 12, 2003 CoDeeN Overview - IRIS/PlanetLab 19

Users

sorted by # users

Aug 12, 2003 CoDeeN Overview - IRIS/PlanetLab 20

The Troubles We’ve CausedRoutinely trigger open proxy alerts

Educating sysadmins, othersResource checks generate noise

Got onto planetlab-supportReally good honeypots

6000 SMTP flows/minute at CMUSpammers do ~1M HTTP ops/day

Aug 12, 2003 CoDeeN Overview - IRIS/PlanetLab 21

What We’ve LearnedParallel ssh is a must

General commands/queriesBasis for parallel scpUsed to detect out-of-date files

Monitoring is a mustToo hard to see anomalies in 40+ nodesAlmost looks like a demo

Be careful accepting outside requests

Aug 12, 2003 CoDeeN Overview - IRIS/PlanetLab 22

What We Still NeedBetter layer 4 tools

Hard to tell why things dieBuilding complete heartbeats isn’t fun

Better isolation on most resourcesCPU/OS: Java, VServers, ???Others: FD exhaustion, disk space

Aug 12, 2003 CoDeeN Overview - IRIS/PlanetLab 23

What We Wouldn’t Mind…Customizable DNS mapping

Map project.planet-lab.org to some nodeProjects could provide feedback• Node availability, utility, etc

Most IP geolocation seems locked up

24CoDeeN Overview - IRIS/PlanetLabAug 12, 2003

More Infohttp://codeen.cs.princeton.edu