How Akamai Handles Large Events Bruce Maggs Carnegie Mellon Duke Akamai Technologies.

How Akamai Handles Large How Akamai Handles Large EventsEvents

Bruce MaggsCarnegie Mellon

DukeAkamai Technologies

Traffic StatisticsTraffic Statistics

• 30,000+ domains

• 1.1 Tbps daily peak traffic

• 6,419 terabytes / day

• 274 billion hits / day

• 274 million unique client IP addresses / day

• in 2009 expect to deliver more bits than in 1998-2008 combined

Network DeploymentNetwork Deployment

40000+Servers

950+Networks

67+Countries

1450+POPs

Assigning Clients to ServersAssigning Clients to Servers

• ““High level”: Map client to a cluster High level”: Map client to a cluster based on client’s nameserver’s IP based on client’s nameserver’s IP address. address. Algorithm: stable marriage Algorithm: stable marriage with multi-dimensional hierarchical with multi-dimensional hierarchical capacity constraints.capacity constraints.

• ““Low level”: Assign client to specific Low level”: Assign client to specific server or servers within cluster based server or servers within cluster based on content requested. on content requested. Algorithm: Algorithm: consistent hashing.consistent hashing.

Embedded Image DeliveryEmbedded Image Delivery (e.g., Amazon) (e.g., Amazon)

<html><html><head><head><title>Welcome to xyz.com!</title><title>Welcome to xyz.com!</title></head></head><body><body><img src=“<img src=“<img src=“ <img src=“ <h1>Welcome to our Web site!</h1><h1>Welcome to our Web site!</h1><a href=“page2.html”>Click here to enter</a><a href=“page2.html”>Click here to enter</a></body></body></html></html>

http://www.xyz.com/logos/logo.gifhttp://www.xyz.com/logos/logo.gif”>”>http://www.xyz.com/jpgs/navbar1.jpghttp://www.xyz.com/jpgs/navbar1.jpg”>”>

Embedded URLs are Converted to ARLsEmbedded URLs are Converted to ARLs

akak

End User

Akamai DNS ResolutionAkamai DNS Resolution

Akamai High-Level DNS Servers

4 ak.xyz.com

1

Browser’s Cache

OS

2

Local Name Server

310

9

520.20.123.55

Akamai Low-Level DNS Servers

6 ak.xyz.com

30.30.123.5 7

8

select cluster

select servers within cluster

MappingMapping

• Maps IP address of client’s name server Maps IP address of client’s name server and type of content being requested to and type of content being requested to an Akamai cluster.an Akamai cluster.

• Note: Doesn’t depend on content Note: Doesn’t depend on content provider (although indicated by provider (although indicated by ak.xyz.com).ak.xyz.com).

• Special cases: Akamai Accelerated Special cases: Akamai Accelerated Network Partners (AANPs)Network Partners (AANPs)

• General case: “Core Point” analysisGeneral case: “Core Point” analysis

Core Points Core Points

• Core point X is the first router at which all Core point X is the first router at which all paths to nameservers 1, 2, 3, and 4 paths to nameservers 1, 2, 3, and 4 intersect.intersect.

• Traceroute once per day from 300 Traceroute once per day from 300 clusters to 280,000 nameservers.clusters to 280,000 nameservers.

X

12 3 4

Akamai cluster 3Akamai cluster 2

Akamai cluster 1

Core PointsCore Points

280,000 nameservers (98.8% of requests) 280,000 nameservers (98.8% of requests)

reduced toreduced to

30,000 core points 30,000 core points

ping core points every 6 minutes

Akamai ClusterAkamai Cluster

Servers pool resources

•RAM

•Disk

•Throughput

HashingHashing

E.g., h(x) = (((a x + b) mod P) mod |B|) , whereP is prime, P > |U|a,b chosen uniformly at random from ZP

x is a serial number (designates content provider)

Universe U of all possible objects, set B of buckets.

object: set of web objects with same serial numberbucket: web server

Hash function h: U B

Assigns objects to buckets

f(d) = d + 1 mod 5

Difficulty changing number of Difficulty changing number of bucketsbuckets

5 7 10 11 27 29 36 38 40 43

4

3

2

1

0

bucket

object

f(d) = d + 1 mod 4

Consistent HashingConsistent Hashing

Idea: Map both objects and buckets to unit circle.

object

bucket

Assign object to next bucket on circle in clockwise order.

new bucket

How it’s really doneHow it’s really done

a212: 10.10.10.1 10.10.10.4 10.10.10.3 10.10.10.2

a213: 10.10.10.3 10.10.10.4 10.10.10.2 10.10.10.1

a214: 10.10.10.1 10.10.10.2 10.10.10.3 10.10.10.4

a215: 10.10.10.2 10.10.10.1 10.10.10.4 10.10.10.3

random permutations of servers

Why? To spread load for one serial number.

Strategies for Handling Large EventsStrategies for Handling Large Events

• Flash crowds (no advance Flash crowds (no advance notice): Rely on rapid load notice): Rely on rapid load balancing. Cross fingers.balancing. Cross fingers.

• Planned events: “pre-warm” Planned events: “pre-warm” caches with objects to be caches with objects to be served.served.

Content Delivery on 9/11 (2001)Content Delivery on 9/11 (2001)

• Akamai’s network had capacity for all Akamai’s network had capacity for all content providers requesting servicecontent providers requesting service

• Total bits served on September 11 Total bits served on September 11 was approximately 3.5 times normalwas approximately 3.5 times normal

• Traffic was higher on September 12Traffic was higher on September 12

• (But not as high as January 7, 2002)(But not as high as January 7, 2002)

News Site A – Image TrafficNews Site A – Image Traffic

News Site A – StreamingNews Site A – Streaming

News Site B – HTML TrafficNews Site B – HTML Traffic

News Site B – Image TrafficNews Site B – Image Traffic

News Site B – StreamingNews Site B – Streaming

Portal A – Image TrafficPortal A – Image Traffic

Sports Site A – Image TrafficSports Site A – Image Traffic

Steve Jobs Keynote (2002)Steve Jobs Keynote (2002)

Akamai live streaming infrastructure

A

A

A

A

A

A

Reflectors Edge servers

Source

Trace format[Client IP, Player ID, Stream URL, Session start time, Session duration, Mean receiving bandwidth]

Extensive traces (2004)

~ 1,000,000 daily requests

~ 200,000 daily client IP addresses from over 200 countries

~ 1,000 daily streams

~ 1,000 edge servers

~ Everyday, over a 3-month period

Audio vs. video

Unknown 22%

Audio 71%

Video 7%

Most streams are audio.

Stream types• Non-stop (76%) vs. short duration (24%)

– All video streams have short duration

• Smooth arrivals (50%) vs. sudden onset (50%)– Flash crowds are common

Obama Inauguration (2009)

• 7 million simultaneous data streams, most carrying live video

• total traffic peaking at 2 terabits per second

• requests per minute fifth place historically, behind election night in November and three sports events (a World Cup soccer playoff and two NCAA basketball playoff games)



Just after – the Internet’s performance starts to degrade massively



And later is mostly better but not great

How Akamai Handles Large Events Bruce Maggs Carnegie Mellon Duke Akamai Technologies.

Documents

Transcript of How Akamai Handles Large Events Bruce Maggs Carnegie Mellon Duke Akamai Technologies.