An Analysis of Facebook Photo Caching

Qi Huang, Ken Birman, Robbert van Renesse (Cornell), Wyatt Lloyd (Princeton, Facebook),

Sanjeev Kumar, Harry C. Li (Facebook)

An Analysis ofFacebook Photo Caching

2

250 Billion* Photos on Facebook

Profile

Feed

Album

* Internet.org, Sept., 2013

StorageBackend

CacheLayers

Full-stackStudy

3

Preview of Results

Current Stack Performance

Opportunities for Improvement

• Browser cache is important (reduces 65+% request )

• Photo popularity distribution shifts across layers

• Smarter algorithms can do much better (S4LRU)

• Collaborative geo-distributed cache worth trying

4

Client

Facebook Photo-Serving Stack

5

Client-based Browser CacheClient

Browser

Cache

Client

LocalFetch

6

Client-based Browser CacheClient

Browser

Cache(Millions)

Client

7

Stack Choice

Browser

Cache

Client

CachesStorage

Facebook Stack

Akamai

Content Distribution Network(CDN)

• Focus: Facebook stack

(Millions)

8

Geo-distributed Edge Cache (FIFO)

Edge Cache

(Tens)

Browser

Cache

Client

PoP

(Millions)

9


Edge Cache

(Tens)

Browser

Cache

Client

PoP

(Millions)

Purpose

1. Reduce cross-country latency

2. Reduce Data Center bandwidth

10


Edge Cache

(Tens)

Browser

Cache

Client

PoP

(Millions)

11


Edge Cache

(Tens)

Browser

Cache

Client

PoP

(Millions)

12

Single Global Origin Cache (FIFO)

Browser

Cache

Edge Cache

OriginCache

PoPClient

Data Center

(Tens)(Millions) (Four)

13


Browser

Cache

Edge Cache

OriginCache

PoPClient

Data Center


Purpose

1. Minimize I/O-bound operations

14


Browser

Cache

Edge Cache

OriginCache

PoPClient

Data Center


Hash(url)

15


Browser

Cache

Edge Cache

OriginCache

PoPClient

Data Center


16

Haystack Backend

Backend (Haystack

)

Browser

Cache

Edge Cache

OriginCache

PoPClient

Data Center


17

How did we collect the trace?

18

Trace Collection

Instrumentation Scope

Backend (Haystack

)

Browser

Cache

Edge Cache

OriginCache

PoPClient

Data Center

(Object-based sampling)

• Request-based: collect X% of requests• Object-based: collect reqs for X% objects

19

Sampling on Power-law

Object rank

20


• Req-based: bias on popular content, inflate cache perf

Object rank

Req-based

21


Object-based

Object rank

• Object-based: fair coverage of unpopular content

22


• Object-based: fair coverage of unpopular content

Object rank

Object-based

23

Trace Collection

77.2M reqs

(Desktop)

12.3MBrowser

s

Instrumentation Scope1.4M photos, all reqs for each

Backend (Haystack

)

Browser

Cache

Edge Cache

OriginCache

PoPClient

Data Center

Resizer

R

2.6M photo objects, all reqs for each

12.3KServers

24

Analysis

• Traffic sheltering effects of caches

• Photo popularity distribution

• Size, algorithm, collaborative Edge

• In paper– Stack performance as a function of photo age– Stack performance as a function of social

connectivity– Geographical traffic flow

25

Traffic Sheltering

77.2M

26.6M11.2M

7.6M

Backend (Haystack

)

Browser

Cache

Edge Cache

OriginCache

PoPClient

Data Center

65.5%58.0%

31.8%

R

Traffic Share

65.5% 20.0% 4.6% 9.9%

26

Photo popularity and its cache impact

27

Popularity Distribution

• Browser resembles a power-law distribution

2%

28


• “Viral” photos becomes the head for Edge

29


• Skewness is reduced after layers of cache

30


• Backend resembles a stretched exponential dist.

31

Popularity with Absolute Traffic

• Storage/cache designers: pick a layer

32

Popularity Impact on Caches

High Low M

Lowest

Each has 25% requests

33


• Browser traffic share decreases gradually

34


• Edge serves consistent share except for the tail

7.8%

22~23%

35


• Origin contributes most for “low” group

9.3%

36


• Backend serves the tail

70% Haystack

37

Can we make the cache better?

38

Simulation

• Replay the trace (25% warm up)

• Estimate the base cache size

• Evaluate two hit-ratios (object-wise, byte-wise)

39

Edge Cache with Different Sizes

• Picked San Jose edge (high traffic, median hit ratio)

59%

40


• “x” estimates current deployment size (59% hit ratio)

65%68%

59%

41


• “Infinite” size ratio needs 45x of current capacity

Infinite Cache

65%68%

59%

42

Edge Cache with Different Algos

• Both LRU and LFU outperforms FIFO slightly

Infinite Cache

43

S4LRU

Cache Space

More Recent

L3

L2

L1

L0

44

S4LRU

Cache Space

L3

L2

L1

L0Missed Object

More Recent

45

S4LRU

Cache Space

L3

L2

L1

L0

HitMore Recent

46

S4LRU

Cache Space

L3

L2

L1

L0

Evict

More Recent

47


• S4LRU improves the most

68%

1/3x

Infinite Cache

59%

48


• Clairvoyant (Bélády) shows much improvement space

Infinite Cache

49

Origin Cache

• S4LRU improves Origin more than Edge

14%

Infinite Cache

50

Which Photo to Cache

• Recency & frequency leads S4LRU

• Does age, social factors also play a role?

51

Collaborative cache on the Edge

52

Geographic Coverage of Edge

Small working set

53


9 Edges with high-volume traffic

54


Do clients stay with local Edge?

55


Atlanta

56


Atlanta

20% local

5% Dallas

35% D.C.

5% NYC

20% Miami

5% California

10% Chicago

• Atlanta has 80% requests served by remote Edges

57


Atlanta

20% local

Miami

35% localDall

as50% local

Chicago

60% local

LA 18% local

NYC 35% local

• Substantial remote traffic is normal

58


Amplified working set

59

Collaborative Edge

60

Collaborative Edge

• “Independent” aggregates all high-volume Edges

61

Collaborative Edge

• “Collaborative” Edge increases hit ratio by 18%

18%

Collaborative

62

Related WorkStorage Analysis

Content Distribution Analysis

Web Analysis

BSD file system (SOSP ’85), Sprite ( SOSP ’91), NT (SOSP ’99),NetApp (SOSP ’11), iBench (SOSP ’11)

Cooperative caching (SOSP ’99), CDN vs. P2P (OSDI ’02),P2P (SOSP ’03), CoralCDN (NSDI ’10), Flash crowds (IMC ’11)

Zipfian (INFOCOM ’00), Flash crowds (WWW ’02),Modern web traffic (IMC ’11)

63

Conclusion

• Quantify caching performance

• Quantify popularity changes across layers of caches

• Recency, frequency, age, social factors impact cache

• Outline potential gain of collaborative caching

An Analysis of Facebook Photo Caching

Documents

Transcript of An Analysis of Facebook Photo Caching