Exploiting Locality of Interest in Online Social Networks

Mike P. Wittie, Veljko Pejovic, Lara Deek, Kevin C. Almeroth, Ben Y. ZhaoDepartment of Computer Science

University of California, Santa Barbara{mwittie, veljko, laradeek, almeroth, ravenben}@cs.ucsb.edu

Paper #155, 14 pages

ABSTRACTOnline Social Networks (OSN) are fun, popular, and sociallysignificant. An integral part of their success is the immensesize of their global user base. To provide a consistent serviceto all users, Facebook, the world’s largest OSN, is heavilydependent on centralized U.S. data centers, which rendersservice outside of the U.S. sluggish and wasteful of Internetbandwidth. In this paper, we investigate the detailed causesof these two problems and identify mitigation opportunities.Because details of Facebook’s service remain proprietary,we treat the OSN as a black box and reverse engineer its op-eration from publicly available traces. We find that contraryto current wisdom, OSN state is amenable to partitioning andthat its fine grained distribution and processing can signifi-cantly improve performance without loss in service consis-tency. Through simulations of reconstructed Facebook traf-fic over measured Internet paths, we show that user requestscan be processed 79% faster and use 91% less bandwidth.We conclude that the partitioning of OSN state is an attrac-tive scaling strategy for Facebook and other OSN services.

1. INTRODUCTIONOnline Social Networks (OSN) are fun, popular, and

socially significant. Their core functionality is the archi-val and delivery of communications along social linksbetween users. Integral to OSNs’ success is the size oftheir global user base: Facebook has more than 400 mil-lion users, over 70% of whom are outside the U.S. [10].To provide a consistent view of the social network toevery user, Facebook relies on large data centers in theU.S. to host full replicas of the OSN’s state. As a result,requests from users outside the U.S. suffer from slow re-sponse time, or lag, and put high load on the Internet

.

backbone. In scaling its infrastructure by replication,Facebook has put an emphasis on consistency over per-formance. Could Facebook have achieved both goals allalong?

In this paper, we investigate the causes of Facebook’spoor performance and the opportunities for improve-ment. Judging by their non-trivial efforts to optimizecommunication protocols, Facebook is keenly interestedin reducing delay and hosting costs [20, 26]. However,many details of Facebook’s service remain proprietary,and so we treat the OSN as a black box and reverse en-gineer its traffic and operations from publicly availabletraces and custom network measurements.

Our analysis lets us make two observations. First, de-lay experienced by users outside the U.S. is due to theinterplay between the Facebook communication proto-col and Internet path characteristics. While the volumeof an individual Facebook request is low, the numberof involved round trips is high. Internet paths betweenhosts outside the U.S. and Facebook data centers arecharacterized by high latency and high loss. The com-bination of high latency and many round trips causeslong request delay, further increased by retransmissions.

Second, the centralization of OSN servers in the U.S.means Internet bandwidth is wasted when many usersin a region outside the U.S. request the same data.While Akamai solves that problem for static content, asignificant share of regional traffic is dynamic and shut-tled to the U.S., thereby wasting Internet bandwidth.

To mitigate the problem of high latency and highloss paths, we observe that these characteristics can beisolated by a well-connected server in each region, sug-gesting the use of TCP proxies, which have been shownto reduce TCP transfer duration in similar scenarios [4,24]. To reduce the load of OSN traffic in the Internet,we observe that the majority of communications arebetween users within the same geographic region. Thishigh locality of interest suggests that partitioning anddistribution of the social graph required for processingand caching of regionally generated content could allowmany requests to be processed in the region, therebyreducing the volume of OSN traffic in the Internet.

Based on these observations, we believe that a distri-bution of Facebook’s architecture at a finer granularityhas the potential to improve OSN service responsivenessand efficiency from the network perspective.

TCP proxies and caching are well understood tech-niques, however, their effectiveness depends on specifictraffic and network characteristics. While it is intuitivethat these solutions should be effective at improvingOSN service responsiveness and efficiency, we quantifytheir effectiveness by simulating TCP connections ofFacebook traffic reconstructed from their social graphand history of communications within a number of fastgrowing regional networks. This approach is novel inits focus on the interplay between user behavior, mech-anisms of a black box OSN, and network characteristics.

We show that regional servers improve service qual-ity and reduce the mean user request delay in Russia,for example, by 79% from 3.4 to 0.7 seconds. We alsoshow that regional servers improve service efficiency andreduce data center load from Sweden, for example, by91% from 568 Mbps to 49 Mbps and network load forthe same interactions by 83%. These reductions in re-quest delay and traffic load are statistically significantand meaningful improvements for Facebook users. Wealso show these gains can be achieved without sacri-ficing Facebook service consistency. Although the de-ployment of regional servers represents additional cost,their benefits and higher efficiency have the potential tooffset the initial investment and indeed present a morecost effective scaling strategy than wholesale data cen-ter replication.

The rest of this paper is organized as follows. In thenext section we describe the Facebook service through areverse engineering effort. Section 3 describes the imple-mentation of TCP proxies and OSN caches. Section 4details the evaluation methodology we use to quantifythe proposed architectural changes in Section 5. Wesummarize related research in Section 6 and concludein Section 7.

2. REVERSE ENGINEERING FACEBOOKMotivated by the size and social significance of Face-

book we are interested in understanding the factorsbehind its performance shortcomings outside the U.S..However, few details about the OSN are available pub-licly, and so we set out to reverse engineer Facebookfrom Web crawls, packet captures, and network mea-surements [11, 23, 30]. We examine the extent to whichnetwork characteristics on paths between users and Face-book can affect performance, and so we locate infras-tructure endpoints and measure paths that connect themto users. We also want to understand how often thesame data is fetched, and so we capture packets andelicit rules of interaction and analyze information flow.

2.1 Overview of The Facebook ServiceOur first step is to describe the relevant details of the

Facebook service as a representative example of a com-mercial OSN, or more broadly, a hosted application thatattracts users with a set of features and attracts adver-tisers, who pay for the privilege of displaying ads tar-geted to these users. OSNs interconnect users throughfriendship relations and allow for asynchronous commu-nications of user generated content, such as text, pho-tos, videos, or third party OSN application updates overa social graph. These communications form the bulk ofOSN traffic and the core of OSN functionality.

To simplify the problem and yet accurately representOSN traffic patterns, we consider three built-in Face-book modes of interaction: wall posts, comments, andlike tags. Wall posts or status updates allow users toassociate text, photos, or videos with their own or theirfriends’ profiles. These posts are displayed in chrono-logical order on a user’s profile and disseminated to thehome page news feeds of the user’s friends. Commentsand like tags (likes) are follow-ups to existing posts.These interactions are delivered to the intended users,their friends, as well as previous participants of the in-teraction chain.

Each user interaction is implemented through one ormore connections with Facebook servers. For example,during a wall post a user opens a TCP connection withFacebook, sends the new content, and receives HTMLmarkup to display the post in the browser. Immedi-ately, the intended addressee user is notified of the newpost on their wall by a push notification through a per-sistent TCP connection that also supports Facebookchat. Other users discover new posts by periodicallypolling Facebook servers. In either case, a new connec-tion is established to request post HTML markup andan additional connection may be required to downloadembedded static content such as photos or video.

To support this traffic, Facebook maintains three datacenters located in Santa Clara, CA, Palo Alto, CA, andAshburn, VA [12]. Ashburn servers reduce service la-tency for East Coast and European users, but the geo-graphic separation poses challenges for Facebook’s stateconsistency. The Ashburn database is a slave to the onein California, and during peak hours, the state of thetwo can diverge by as much as twenty seconds [22]. Toprevent memcached servers from serving stale data, allread requests for posts less than twenty seconds old areserved from California. The California data centers alsohandle write requests for all new posts. To host staticcontent such as photos and videos, Facebook uses Aka-mai’s Content Distribution Network (CDN).

2.2 Social Graph and TrafficTo understand the performance issues of Facebook’s

infrastructure, we first characterize the user social graph

Table 1: Characteristics of regional social graphs and history of posts.Social graph Interactions per day

Users Node degree Wall Comment Like Photo

RegionLocal Total To local users To all users

Num.Vol.

Num.Vol.

Num. Num.Vol.

(mil.) (mil.) md. mn. s.d. md. mn. s.d. (MB) (MB) (GB)

Russia 0.22 0.26 2 5.4 12.3 2 21.6 71.8 6996 1.62 5495 0.54 176 1328 173.2Egypt 1.05 3.86 5 18.4 48.0 5 29.9 94.2 32587 7.45 15200 1.93 126 2004 817.96Sweden 2.30 8.59 11 35.0 60.1 12 48.1 94.3 153015 37.5 18518 1.36 353 25910 3360.5NYC 2.12 2.95 8 24.4 47.6 14 71.0 164.5 415655 71.3 21057 1.30 1875 46685 6055.1LA 1.56 2.25 6 22.3 46.3 9 62.0 158.6 165264 30.9 18481 30.9 1031 20448 2652.2

and its traffic. We analyze the social graphs and userinteractions distilled from publicly available Facebookcrawls of user profiles in a number of regional networks,selected for their size, geographic location, and avail-ability of crawls. Table 1 shows network characteristicsgermane to our problem collected over 30 days startingon 1-Jun-2009, using techniques of Wilson et al. [30].

The left half of the Table 1 shows the characteris-tics of the social graph. The social graph of a regionalnetwork is composed of links between users in that net-work, or local users, and all their friends, those in andout of the regional network. Column two and threeshow the number of local users and all users in the so-cial graph of each regional network. We observe thatnetworks vary by size as well as the ratio of local andnon-local users in their social graphs.

Further, we characterize node degree of local usersto other local users and to all users in the social graphin columns four through nine. Because the number offriends varies substantially between users, we charac-terize the distribution of node degree by median (md.),mean (mn.), and standard deviation (s.d.).

The small difference between in-network and totalnode degree in the social graphs of the first three net-works suggests that their users are mostly friends withother users in the network. In comparison, users in thetwo U.S. city networks have a higher percentage of out-of-network, or remote, friends, which when comparedagainst the high percentage of in-network, or local, usersin the social graphs of these networks, suggest high re-latedness between remote user sets. This observationwill help us interpret results in Section 5.

Also helpful in interpretation of results will be a sum-mary of the number and volume of posts made by usersin each region in columns 10 through 16. The number ofposts and volume of their content are averaged over 24hours. We observe that in each network, wall posts arethe most popular, followed by comments, photo posts,and like tags. Photo posts generate the most uploadnetwork traffic, because of the size of embedded content.

2.3 Transaction AnalysisTo understand network load, we need to understand

how posts relate to network packet traffic. Tho this end,we collect pcap network traces of packets exchanged ineach type of user post between our lab and Facebook

TRANSFER FROM TO BYTES COMMENT1 user CA 1510 #upload script request2 CA user 5703 #upload script3 user CA 136802* #photo and post text4 CA user 7168 #display markup5 user CDN 495 #display image request6 CDN user 3819* #display jpeg

Figure 1: Transfers of a photo post.

servers. The network traffic signature of each trans-action varies with network conditions, so we extractHTTP requests for each transaction. For example, Fig-ure 1 shows a photo wall post (image plus text) as aseries of HTTP transfers between the user and sym-bolic infrastructure endpoints. The number of bytes intransfers marked with a ‘*’ signifies that transfer sizedepends on the actual size of the photo and length oftext being uploaded.

We have also captured packets of push and poll trans-actions and each type of interaction they deliver. Auser’s browser periodically polls Facebook at approxi-mately 55 second intervals. Replies to a poll requestreturns whatever posts have been made since the lastpoll. The data required to display these interactionsis concatenated to make the transfers more efficient,though embedded static content requires separate com-munications with the CDN.

In summary, we make three observations. First, dis-play markup and upload scripts (for static content) cansignificantly inflate network traffic beyond the post con-tent volume in Table 1. Second, while the network traf-fic involved in each post is small on its own, the numberof posts, multiplied by their delivery to multiple friends,can result in substantial server load, an effect we inves-tigate in Section 5.2.1. Finally, many interactions, espe-cially those that include static content or concatenatemultiple posts, may be comprised of a large numberof network round trips that can add up to substantialdelays depending on network characteristics.

2.4 Internet Path AnalysisTo understand the behavior of Facebook’s traffic in

the network, we need to understand the characteristicsof the Internet paths between users and the OSN infras-tructure. First, we discover the locations of endpointsthat service interactions in different regions. Second, wemeasure latency, loss, and capacity on paths between

Figure 2: Region to CDN server DNS map.

these endpoints and user hosts in each region. Our goalis to gauge the effect network characteristics can haveon OSN traffic.

While the identity and location of Facebook U.S. datacenters are well known, the CDN server used in each in-teraction is determined dynamically through DNS redi-rection. To discover the set of CDN servers used byFacebook requests we run the following experiment. Weiterate over a list of globally distributed DNS servers1,setting each as our nameserver and issuing lookups forCDN hosted Facebook objects. Our approach is similarto the one take by Su et al. [23], but instead of issu-ing requests for CDN objects from PlanetLab nodes,we contact geographically distributed DNS servers di-rectly. We record the Akamai IP addresses returned byeach DNS query and create a DNS-CDN server map-ping. We then query the IP2Location2 service for thegeographic coordinates of DNS and CDN endpoints andplot the set of unique links in Figure 2. While Su et al.discover that a large number of Akamai servers can beused to service requests from a single host, we observethat many of the servers handling Facebook objects areco-located geographically. Our conclusion is that theCDN server handling the request may not be the closestone, for example, when requests issued from the U.S. aresometimes redirected to Europe, and so, the network-induced delay between hosts in Russia, or Egypt, forexample, and the CDN server located in Western Eu-rope may be significant. We include the impact of user-CDN traffic in our evaluation of the overall duration ofFacebook transactions.

Next, we perform measurements of network latency,loss, and bandwidth to characterize network performanceon Internet paths between user hosts and the OSN and

1http://dnsserverlist.org/2http://ip2location.com/

CDN endpoints. The performance of “last-mile” accessnetworks can vary between regions and even within re-gions. To adequately represent this variation, we mea-sure Internet paths to IP hosts placed in each regionthough IP block country assignments3 and through theIP2Location service. However, the number of users ineach region is larger than the number of host paths wecan reasonably measure, and so we characterize the con-nectivity within each region as a distribution based onlast mile path measurements. We found that measure-ments to around 500 random IP hosts in each regionstabilize the shape of that distribution.

Because we do not have access to the actual infras-tructure endpoints or the user hosts themselves, for ourmeasurements we rely on PlanetLab4 nodes in close net-work proximity to Facebook and CDN data centers. Ta-ble 2 lists the infrastructure endpoints and the Planet-Lab nodes we have selected as substitutes. The latenciesseparating Facebook infrastructure endpoints and Plan-etLab nodes, shown in the last column, are negligible,and the PlanetLab nodes, which reside in university net-works, have high capacity Internet connections. Whileour approach ignores queuing delays at OSN servers,these are the same for all users regardless of their geo-graphic location and represent a separate problem [18].For these reasons, we believe that these substitutions donot significantly distort measurements performed fromthese nodes to user hosts. From each PlanetLab node,we measure latency and packet loss to user hosts withineach country using a large number of ping requests.

Accurate available bandwidth and network capacitymeasurements without the control of both path end-points is a difficult problem [17]. We make use of anexisting measurement effort that estimates last mile ca-pacity from logs of BitTorrent TCP transfers in differ-ent regions [11]. We adopt an assumption made in thatstudy that network capacity to hosts within the same/24 subnets is similar and that the last mile capacityis the path bottleneck. Because individual transfers in-volved in Facebook interactions are small, our evalua-tion models only consider network capacity, which gov-erns packet serialization delay. Subsequently, we ignorediurnal variations of available bandwidth and asymmet-ric provisioning of last mile links that affect to a lesserextent “mouse” transfers that comprise OSN traffic.

Table 3 shows means of network measurements ineach region. The direct (dir.) connections are betweenuser hosts and Palo Alto, CA. The proxied (prox.) con-nections show two numbers: first for paths between CAand a PlanetLab node in each region, and second be-tween the PlanetLab node and regional hosts.

We make two observations. First, direct paths be-tween Facebook and regional users are characterized by

3http://countryipblocks.net/4http://planet-lab.org/

Table 2: Infrastructure endpoint substitutions.Region Endpoint Location PlanetLab node Location Separating latency

Facebook CA Palo Alto, CA planet1.scs.stanford.edu Palo Alto, CA 2.984 msVA Ashburn, VA planetlab3.cnds.jhu.edu Baltimore, MD 4.228 ms

Russia CDN UK planetlab2.cs.ucl.ac.uk London, UK 2.217 msEgypt CDN UK planetlab2.cs.ucl.ac.uk London, UK 2.731 msSweden CDN Sweden planetlab2.sics.se Stockholm, Sweden 1.183 msNYC CDN Ashburn, VA planetlab3.cnds.jhu.edu Baltimore, MD 4.228 msLA CDN Palo Alto, CA planet1.scs.stanford.edu Palo Alto, CA 2.984 ms

Table 3: Network measurement summary.Latency (ms) Loss (%) Capacity (Mbps)

Region dir. prox. dir. prox. dir. prox.

Russia 148 115 31 6.1 0 1.8 29.6 367 29.6Egypt 164 176 67 5.8 0 5.8 0.92 736 0.92Sweden 104 95 14 0.32 0 2.9 9.47 188 9.47NYC 74 43 33 0.75 0 0.6 9.51 99 9.51LA 27 9.1 18 0.50 0 0.4 2.02 228 2.02

high latency and high loss rates diminishing with prox-imity to CA. Second, connections tunneled through aregional PlanetLab node isolate most of the path la-tency on the link to CA and all of the loss onto theregional link. Based on these observations, we believethat TCP proxies deployed at network locations similarto regional PlanetLab nodes would improve TCP per-formance on paths between Facebook data centers andregional users, and so reduce user request delay.

2.5 Locality of Interest AnalysisWhile TCP proxies can reduce delay, the other prob-

lem is load. Data can be fetched to the region multipletimes wasting ISP bandwidth. Our goal is to charac-terize locality of interest to understand the extent towhich this is a problem.

To characterize user communication patterns withineach region, we first address a shortcoming of the crawlprocess: posts not addressed to users within the crawledregion cannot be observed in the user news feeds scannedby the crawl. Local-to-local (LL) and remote-to-local (RL)user posts can be observed by scanning news feeds oflocal users. Local-to-remote (LR) interactions can onlybe observed on the profiles of remote users, which arenot represented in the crawls. To estimate the num-ber of LR interactions, we first calculate for each lo-cal user u the reciprocity factor for interaction type t

as: rtu = 1

|Fu|∑

v∈Fu,v∈L|Pt

u,v||Pt

v,u|, where Fu is the set of

friends of u, L is the set of local users, and Ptu,v is the

set of posts of type t made from u to v. We estimatethe number of LR interactions as rt

u

∑v∈Fu,v /∈L |Pt

u,v|.Figure 3 shows the breakdown of interactions in each

region by user location, marked as a percentage on they-axis. We observe that posts made by local users, LLand LR, comprise at least half of all posts in each re-gional network. If posts by local users are delivered tomany users within the region, or there exists a high lo-

cality of interest, then handling this traffic within theregion could reduce OSN service latency.

Delivery of posts is determined by the social graph. Apost is delivered to users who are friends with both theposter and the addressee. Figure 4 shows the deliveryratio of wall posts, where the y-axis shows the num-ber of users within the region that receive each post.LL posts result in the largest read-to-write ratio sincethe sets of poster’s and addressee’s friends tend to havehigh overlap within the region. LR and RL interactionshave much lower delivery ratio within the region, be-cause friends of the local poster tend to have differentsets of out-of-network friends and because these updatesdo not include self-posts delivered to all the poster’sfriends.

The delivery mechanisms for comments and likes areslightly different. After a user makes a comment or alike post, subsequent posts are delivered to that user asnotifications irregardless of whether the user is friendswith the posting users. This cascade of updates candrastically increase the read-to-write ratio of subsequentposts, and so for clarity, we separate the delivery ratioof these posts in Figure 5. Both LL and RL commentsand likes can refer to the same news feed item and soconsecutive posts quickly inflate the number of readersfor each type of interaction. Note that observed ratioof reads to writes would significantly inflate the numberof transactions over the writes shown in Table 1.

The cascade of local reads for LR comments or likesdepends on the number of RR posts to the same re-mote user. The RR interactions are not observed bythe crawls of local users, and indeed, collecting theirset for every remote user might require the knowledgeof the entire Facebook state. Subsequently, the numberof comment and like updates delivered to local users (forthe LR category only) is under-represented in our anal-ysis, and so conclusions we offer later pertain only tothe subset of traffic we were able to observe. However,we expect the read-to-write ratio for LR interactions tobe comparable to the LL ratio under the assumptionthat the number of comments made to an LL news feeditem is comparable to the number of follow-ups on anRR item. The exclusion of some comment deliveriesfrom our evaluation results in under-estimation of localtraffic, but does not substantively impact our conclu-sions.

Russia Egypt Sweden NYC LA

Inte

ract

ion

perc

enta

ge0

2040

6080

100 LL LR RL

Figure 3: Wall posts by end-point location.

Russia Egypt Sweden NYC LA

Rea

ds p

er w

rite

020

4060

80

LL LR RL

Figure 4: Delivery ratio of wallposts.

Russia Egypt Sweden NYC LA

Rea

ds p

er w

rite

010

030

050

0

LL LR RL

Figure 5: Delivery ratio of com-ment and like posts.

The analysis of regional interaction patterns showsthat traffic local to a region is produced and consumedin significant volumes within the same region. Such ahigh locality of interest suggests that caching of postsoriginating in the region would reduce request delay.We believe this shows promise for a redesign of Face-book’s infrastructure to cater to local interactions and,by doing so, improve user perceived responsiveness.

2.6 DiscussionBased on reverse engineering Facebook from crawls,

packet captures, and network measurement, we haveidentified causes of lag and wastage of Internet band-width. The lag experienced by users interacting withFacebook from outside the U.S. is created when manyround trips required in Facebook’s communication pro-tocol traverse high latency and loss Internet paths. Fur-ther, the high locality of interest means that users inthe same geographic region read the same updates cre-ated in the region itself. The Facebook infrastructure iswasteful of Internet bandwidth as it delivers the samecontent, not counting static content delivered by Aka-mai, many times to a geographic region.

We believe these shortcomings are fundamental toFacebook’s design approach, which values consistencyof service over performance, and so relies on central-ized data centers. The observed shortcomings can beaddressed at their core by straightforward and well-understood techniques: TCP proxies and caching. Inthe next section we offer a proposal to incorporate thesetechniques in a distributed architecture. At this point,we are interested in quantifying the performance gainthat could come from such more distributed architec-tures and leave optimizations to future work. In Sec-tion 5 we will show that performance gains can be achievedwithout endangering service consistency.

It is germane to relate OSN traffic characteristics tothose of other distributed communication applications,such as chat, or email. Unlike chat or email traffic,OSN communications are routinely addressed to hun-dreds of users, and so OSN traffic is more multicastin nature. Unlike chat, but similarly to email, not allof the addressees may be online when an OSN post ismade, and so archival of old messages is required. Un-like email, but similarly to chat, OSN user addresses do

not rely on DNS resolution, and so message forward-ing is done within the service infrastructure. Finally,similarly to chat and email, OSN users have an expec-tation of timely delivery of their communications. Thecombination of multicast traffic patterns, need for per-sistent storage, timeliness of communications, and lackof a distributed addressing scheme differentiate OSNsenough that it is not clear if such services would benefitfrom distributed architectures in the same way chat andemail systems do [8].

3. DISTRIBUTED FACEBOOK STATEThrough measurement and analysis, we have quanti-

fied characteristics of Facebook’s service that suggest adistributed infrastructure would improve its responsive-ness and efficiency. We now describe two alternative ar-chitectures. Both alternatives are based on the deploy-ment of regional servers, but with different functional-ity. We describe the current Facebook architecture inFigure 6 and the alternatives in the following sections.

The example in Figure 6 shows the geographic lo-cations of endpoints involved in processing Facebookrequests of users in Russia. Referring back to Figure 1,which lists the transfers of a photo post transaction,the path labels in Figure 6 specify transfers that tra-verse these paths. Transfers 1 through 4 traverse thehigh latency, high loss path between users and U.S. datacenters and do not take advantage of either TCP prox-ies, or the locality of interest.

3.1 TCP ProxiesThe long latency and high loss on paths between users

and the OSN’s servers, and the fact the these path prop-erties can be isolated by regional servers, suggest thatTCP proxies would reduce transfer delays [4, 24]. Be-cause the round-trip time between a user host to the re-gional server (REG) is shorter than to U.S. data centers,the TCP congestion window can expand more quicklyand reduce transfer time, especially for the short trans-fers prevalent in Facebook traffic. At the same time,the TCP connection on the other leg of the path can beshared between transfers, and so use a congestion win-dow expanded in accordance with available link band-width. Finally, when losses do occur on a split TCP con-

Figure 6: Photo post transfers in Facebook.

Figure 7: Photo post transfers with TCP proxy.

Figure 8: Photo post transfers with OSN cache.

nection, packets are retransmitted only over the shortersub-paths and incur less delay.

Figure 7 shows the transfer of a photo post forwardedvia a TCP proxy regional server. The user host connectsto the regional server, which forwards the request to theappropriate infrastructure endpoints. While TCP prox-ies do not reduce the number of transfers, as comparedwith Figure 6, these transfers can achieve lower delayover sub-paths connected at the regional server. Ourapproach is different from one-hop routing [23], becausewe proxy the connection and consider path loss rate aswell as latency. The regional server separates high lossfrom high latency of a path because loss is found mainlyon the last mile links within the region.

3.2 Regional OSN CacheThe high locality of interest of users in each observed

region suggests the use of OSN caches at the regionallevel. Caches allow post delivery to avoid the high la-tency to U.S. data centers and to reduce network loadon these paths. We propose to extend the functionalityof regional servers to cache posts and the social graphrequired for their delivery within a region. Because stor-ing these updates indefinitely would require an amountof storage comparable to that of the U.S. data centers,we time limit the regional cache to the user poll intervalof 55 seconds.

Figure 8 shows a photo post processed through a re-gional cache. As in the TCP proxy solution, the userhost communicates with the regional server. However,the upload scripts and HTML markup can be returnedby the cache server without contacting U.S. data cen-

Figure 9: OSN simulation schematic.

ters, which requires only the content of Transfer 3 tobe sent to the U.S. to maintain global OSN state con-sistency. Additionally, many read transactions couldbe served directly form the regional server due to thehigh locality of interest. This change would eliminatemany requests to U.S. data centers altogether. Reduc-tions in long range traffic can save money for InternetService Providers (ISPs) that can now keep more trafficwithin their own networks and allow Facebook to nego-tiate lower Internet access costs to the regional servers.Thus, OSN caches have the potential to translate intoa long-term service cost reduction in spite of the initialinvestment in regional infrastructure.

We note that a practical deployment of regional cacheswould also require the implementation of ad placement.Regional cache servers send page display markup, andso, embedded ads, which could be placed by runningregional algorithms, or in coordination with U.S. in-frastructure. In this work, we deal only with automaticdelivery of posts through the push and poll mechanisms,which do not carry ad content themselves.

4. EVALUATION METHODOLOGYTo evaluate the degree to which our proposed changes

are useful, we quantify the performance of Facebookunder three alternatives: the current Facebook archi-tecture, the regional TCP proxy architecture, and theregional OSN cache architecture. To this end, we im-plement an event-driven simulator that recreates andmodels OSN traffic over measured Internet paths.

Figure 9 illustrates the simulator components and thestages of information flow between them. As input, oursimulator takes user post history, post HTTP traces,and Internet path measurement, described in Section 2.To reconstruct Facebook traffic, the simulator performstwo preprocessing steps. First, it assigns users with-out known location into regions represented in the so-cial graph. We begin with a distribution of users inthe known networks. We then assign users to regionsbased on a probability equal to the fraction of totalusers in each region. While this method can misplacesome users, the overall distribution of users, locality of

−1.5 −1.0 −0.5 0.0

0.0

0.2

0.4

0.6

0.8

1.0

TCP proxy minus direct transfer time (s)

CD

FRussiaEgyptSwedenNYCLA

Figure 10: CDF of TCP proxy time saved.

interest, and interaction consumption ratios remain thesame. In the second preprocessing step our simulatorgenerates LR interactions through the process describedin Section 2.5.

Preprocessing leads to the simulator core, which iter-ates through the history of user interactions. Dependingon the type of interaction, our simulator reconstructstraffic of each post and its delivery through the pushor poll mechanisms. The network traffic is replayedover TCP connections modeled on the measured Inter-net paths to collect a set of delay and load metrics thatdescribe OSN performance. These results become thebasis for our conclusions.

In more detail, process_write() handles individualwrite interactions. TCP transfers are simulated overmeasured paths to calculate interaction delay and set awrite’s commit_time to the California databases. Sortedby commit_time processed writes comprise the simulatedOSN state. process_reads() handles the delivery ofwrites to active users. We assume that users are activefor half an hour at a time [31] and that user activity cor-responds with their posts. Writes are delivered as readsonly to users active at a write’s commit_time throughpush or poll mechanisms. Push message are deliveredimmediately as a notification to an active user. Oth-erwise, active users poll every 55 seconds and receivewrites that have been committed since the last poll.

4.1 Regional Server FunctionalityThe differences between the current Facebook archi-

tecture and the proposed alternatives are implementedin process_interactions(), which simulates the seriesof TCP transfers required for each interaction in Fig-ures 6 through 8. As illustrated in Figure 7 transfersbetween an infrastructure endpoint and a user are prox-ied at the regional server. Data is forwarded on pathsbetween the infrastructure endpoint and the regionalserver and between the regional server and the user. Incalculating the delay of proxied transfers we accountfor the fact that both TCP connections are establishedsequentially, but can proceed simultaneously.

Figure 10 shows a CDF of the time differences be-tween proxied and direct 8 KB transfers to Palo Alto, CAfrom hosts within each studied region. All of the mea-sured hosts in regions separated from the US by transat-lantic links, and some hosts in US cities, benefit fromtunneling as indicated by the negative difference be-tween transfer times. To avoid penalizing well-connectedclients in regions close to Facebook data centers, TCPproxies are used only for hosts that can benefit basedon their network connectivity.

To implement the regional cache architecture in oursimulator, two extensions are needed. First, we asso-ciate cached_time with each write to indicate when apost becomes available for push or poll requests. Sec-ond, LL and LR writes are sent to REG, cached thereon arrival, and forwarded to US for global OSN con-sistency. The regional cache can service push and pollrequests pertaining to LL and LR writes as soon asthese are cached. Push and poll requests for RL writesrequire these posts to be fetched first from US data cen-ters, and so each poll query sent to REG is forwarded toUS to check for such posts. Once fetched to the regionalcache, cached_time of RL writes is set, they are pushedto the addressee, and become available for delivery insubsequent poll queries.

5. RESULTSWe evaluate the degree to which the proposed archi-

tecture alternatives improve Facebook responsivenessand reduce resource use. We characterize service re-sponsiveness in terms of interaction delay and OSNstate drift, defined as the difference between the timea post is made and the time it is delivered to the ad-dressee. We show that regional servers reduce inter-action delay and OSN state drift, which defines userperception of the real-time feel of the system.

We characterize resource use in terms of server andnetwork traffic load, which are tied to hosting costs,and in terms of the cache size requirements of regionalservers, which determine their size and initial cost ofdeployment.

We show that regional servers can lower hosting costsby reducing U.S. data center load and by shuttling dataover shorter distances, thereby preserving network re-sources. We also demonstrate that regional servers re-quire very modest resources, and therefore pose an at-tractive alternative for infrastructure expansion.

5.1 OSN Service ResponsivenessWe characterize service responsiveness in terms of in-

teraction delay and OSN state drift.

5.1.1 Interaction DelayFigure 11 shows the mean delay of local write inter-

actions (LL and LR) within studied regions. The y-axis

Russia Egypt Sweden NYC LA

Del

ay (

s)0.

01.

02.

03.

0Current Proxy Cache

Figure 11: Mean write delay.Russia Egypt Sweden NYC LA

Del

ay (

s)0.

00.

51.

01.

52.

0 Current Proxy Cache

Figure 12: Mean read delay.Russia Egypt Sweden NYC LA

Sta

te d

rift (

s)0

24

68

Current Proxy Cache

Figure 13: Mean state drift.

marks the mean write delay and 97% confidence inter-vals separated on the x-axis by region and architecture.Generally, mean write delay decreases in Figure 11 fromleft to right for regions closer to CA data centers. Weobserve that for all regions, the current architecture hasthe highest interaction delay. This result is due to thelong latencies and high loss rates on paths between userhosts and U.S. data centers and poor TCP performanceon these paths.

Write delay in the TCP proxy architecture is shownin the middle column of each region. Proxied TCP con-nections reduce write delay by almost 75% in Russiaand to a lesser degree in regions located closer to CAas predicted in Figure 10.

The third column in each region shows the write de-lay when regional caches are present. Caches reducemean write delay because users communicate with theregional server over low latency links and do not waitfor post data to propagate to CA. We observe that theeffect of the cache solution as compared to the tunnelsolution is most pronounced in Sweden, which has thehighest percentage of writes originating in the region asshown in Figure 3.

Figure 12 shows the mean read delay of push and pollrequests. The y-axis marks interaction delay separatedon the x-axis by region and architecture. Similar, towrite delay, read delay decreases the closer the region isto U.S. data centers. While Russia and Egypt are simi-larly far from U.S. data centers in terms of network la-tency, the delay of read requests in Egypt is higher. Thereason for this difference is that both Russia and Egyptconnect to CDN servers in Western Europe, which arerelatively farther away for users in Egypt, and so readrequests from Egypt take proportionally longer to com-plete.

As was the case for write interactions, read delayis also shortened by the use of tunneled TCP connec-tions, as seen by the difference between the first andsecond column for each region. The delay reductiondue to TCP proxies is less pronounced for reads thanfor writes. Write transactions may contain static con-tent such as photos or videos that take a long time toupload to CA. Reads on the other hand, deliver muchsmaller versions of the same content, compressed fordisplay, and are served from CDN servers, which are

much closer to users than CA. As a result, the tunnel-ing and cache mechanism are more effective at reducingmean delay for the larger write transactions that takeplace over longer and more lossy paths

Finally, the third column of each region in Figure 12shows read delay when regional OSN caches are used.While caches can eliminate transfers between regionalservers and U.S. datacenters, these transfers take placeover high data rate paths, and so the reduction to readdelay of these TCP transfers is moderate.

5.1.2 OSN State DriftRead and write delay reflect the responsiveness of the

OSN service to user prompts. Another measure of ser-vice responsiveness is the time it takes for informationto propagate between users. We measure this propaga-tion delay as OSN state drift defined as the differencebetween the time of the post and the time that the postis delivered to interested users. The reduction in statedrift is important because it improves the interactivefeel between users and avoids “Can you see it yet?” sit-uations.

Figure 13 shows the mean OSN state drift for eachstudied region. The y-axis marks state drift in secondsseparated on the x-axis by region and the traffic han-dling method. In general, we notice that OSN statedrift is higher for NYC and LA than for the transat-lantic regions. This result may seem counterintuitive,since NYC and LA are much closer to Facebook datacenters and indeed read delay is lowest for these tworegions. However, state drift is affected not only byinteraction delay, but in the case of poll transactions,also by poll interval. The high node degree of the so-cial graph in the U.S. and the high node degree of themost active users means that proportionally more inter-actions in NYC and LA are delivered through the pollmechanism, which inflates state drift.

Comparing the first and second columns in each re-gion, we observe that TCP proxies reduce drift moreeffectively in regions that are farther away from U.S.data centers. We also observe that serving read re-quests from a local cache offers a further reduction inOSN state drift, especially in NYC and LA. In theseregions, poll requests dominate read delay, and so onlythe first poll incurs the delay of fetching post contentand subsequent polls simply access the cache.

Overall, tunneled TCP connections and regional OSNcaches are effective techniques for improving the respon-siveness of an OSN service. Tunneling reduces the im-pact of long network latencies and high loss rates oflast-mile links. Caching offers additional delay reduc-tions over tunneling and is especially effective at reduc-ing state drift. Taken together, these techniques havethe potential to improve the real-time feel of an OSN byreducing interaction delay in some regions from a no-ticeable three seconds to a barely perceivable fractionof a second. Because post, push, and poll interactionsrepresent core OSN functionality, we expect our resultsto translate to all OSN traffic that rely on these mech-anisms.

It is important to notice that the consistency of theFacebook service is not violated. First, posts traversethe regional servers to the U.S., and so the the deploy-ment of regional servers does not change the state inU.S. datacenters. Second, user interactions outside theU.S. experience on average less lag and are deliveredsooner. Poor performance is no longer the norm, butthe exception – we believe that does not constitute alapse in consistency of the service.

5.2 Resource UtilizationFacebook has demonstrated interest in reducing host-

ing costs, which are in turn tied to resource use. Wecharacterize resource use in terms of server and networktraffic load, and in terms of cache size requirements ofregional servers.

5.2.1 Traffic LoadFigure 14 shows the traffic load at each infrastruc-

ture endpoint. Each column is a stack of load measure-ments for each endpoint marked in Mbps on the y-axis.Stacked load measurements are grouped on the x-axisby region and traffic handling method. Because the to-tal volume of traffic is different in each region, we plotthe load of each network on its own scale.

In the current architecture, CA data centers handleall the writes, as well as reads, for posts that are lessthan twenty-seconds-old, which constitutes most of theupdates requested by push and poll mechanisms. TheCA data center also receives the heavy volume of staticcontent upload traffic. Reads older than twenty secondsare serviced from VA.

The volume of CDN traffic is dwarfed by the previoustwo categories. Considering the high volume to staticcontent shown in Table 1, this result is unexpected andneeds to be interpreted carefully. First, traffic we markas CDN’s includes only downloads, as uploads are sentto CA. Second, the size and resolution of displayedstatic content can be reduced by as much as a factor of4 or more from the uploaded image; for example, noticethe difference in uploaded and downloaded image size

in Figure 1. Finally, our analysis only includes the au-tomatic delivery of information onto users news feed bythe push and poll mechanism. User browsing behavior,not included in our traffic, would rely much more heav-ily on the CDN. In this last point we acknowledge thatCDN load in Figure 14 represents only a small fractionof total CDN traffic. Nevertheless our characterizationof CA and VA traffic, and CDN traffic for push and pollmechanisms remains valid.

The introduction of the TCP proxies does not reducetraffic to U.S. data centers or the CDN. In fact, the to-tal amount of traffic handled by Facebook increases asthe proxy needs to service the incoming and outgoingtraffic on the paths from the regional servers to usersand to infrastructure endpoints. Notice, however, thatproxy traffic on the regional server is less than twice thetraffic at other endpoints because lower number of re-transmissions on the more reliable paths on either sideof the TCP proxy (see Table 3). Finally, we observerelatively lower regional server load for U.S. city net-works. The reason for this can be seen in Figure 10,which shows that only a portion of clients in those net-works benefit and use proxied TCP connections.

Traffic reduction to U.S. data centers is only achievedwith the introduction of regional OSN state caches,which reduce write traffic from regional servers to in-frastructure endpoints and satisfy many reads entirelywithin the regional network. The implication of theseobservations is that while TCP proxies may increasenetwork costs for Facebook, a partnership between Face-book and regional ISPs could benefit both parties. Asimilar tradeoff has been used by Akamai, who placestheir servers for free in ISP networks [23]. Likewise,OSN caches can keep traffic within the region and re-duce ISP peering costs. ISPs will pass on at least partof these savings to Facebook, which may want to realizethe benefits of regional servers, but may be reluctant toincur extra traffic costs at those servers.

The forwarding of traffic over long distances consumesadditional resources that need to be allocated throughinfrastructure deployment within an ISP or paid for viapeering agreements between ISPs. The exact cost ofnetwork use depends on network topology, network con-gestion, and the nature of peering agreements betweenlocal and higher tier ISPs.

To simplify the problem, we consider the measureof bytes transmitted multiplied by the distance trav-eled along the curvature of the Earth, also known asthe great circle distance. We calculate great circle dis-tance between server geographic coordinates obtainedfrom the IP2Location service. Although the great cir-cle distance is independent of actual Internet topology,considering the large distances between endpoints it stillallows for valid conclusions.

Cur

rent

Pro

xyC

ache

Ser

ver

load

(M

bps)

020

4060

Russia

Cur

rent

Pro

xyC

ache

Ser

ver

load

(M

bps)

010

020

030

0

EgyptC

urre

ntP

roxy

Cac

he

Ser

ver

load

(M

bps)

040

080

014

00

Sweden

Cur

rent

Pro

xyC

ache

Ser

ver

load

(M

bps)

050

010

00

NYC

Cur

rent

Pro

xyC

ache

Ser

ver

load

(M

bps)

020

040

0

LA

CA VA CDN REG

Figure 14: Server endpoint load.Russia Egypt Sweden NYC LAD

ata

times

dis

tanc

e (M

bm)

0e+

002e

+05

4e+

05

Current Proxy Cache

Figure 15: Network load.Russia Egypt Sweden NYC LA

Cac

he s

ize

(MB

)0

1020

3040

Figure 16: Cache size.

Load multiplied by distance allows us to estimateglobal impact on network resources consumed by a trans-fer, as well as the number and size of transfers routedon longer versus shorter paths. The implication is thatlower network impact for the same number of interac-tions makes more efficient use of resources, which shouldtranslate to savings for ISPs, and so lower network costsfor large customers such as Facebook.

Figure 15 shows the mean network traffic load multi-plied by distance in Mbm (Mega-bit-meters) for all in-teractions. For the European networks the use of TCPproxies slightly decreases data times distance due tofewer retransmissions on the more reliable sub-paths.For the U.S. networks data times distance is slightlyhigher for the proxied connections, because the effect offewer retransmissions is lower in the more reliable U.S.last mile networks, and the effect of triangle inequal-ity increases total path distance. The use of regionalcaches, however, prevents the traversal of the long pathsto Facebook data centers and significantly reduces thedistance over which data is sent. This effect is mostpronounced in regions distant from CA data centers.

5.2.2 Cache UseImprovement in Facebook service responsiveness and

reduction in resource use depend on the deployment ofregional caches. Besides the already investigated net-work load of these endpoints, the other component oftheir cost is the amount of storage required for thecached OSN state.

We calculate cache storage use, or required cache size,as the sum of all the posts stored on the regional cache,where the size of each post is its meta data: ID ofposter (8B), ID of addressee (8B), time stamp (8B),target (4B), and item (8B). The target and item fieldsare used to reconstruct conversation chains. The cachealso stores the text of each post, but not the static con-tent such as photos or videos, served by the CDN.

Figure 16 shows the maximum cache storage used atany point during the simulation marked on the y-axisand separated by region on the x-axis. Maximum cacheusage depends on the number and size of stored postswithin the 55 second poll interval, and so depends onpeak load in any such interval. The links of maximumcache usage to peak load makes it hard to correlate

cache usage to aggregate regional network propertiespresented throughout this paper. However, the mod-erate cache size required to handle peak load in allnetworks shows that regional caches we propose couldbe implemented on inexpensive commodity hardware.This result also implies that for the evaluated mecha-nisms, the major resource requirements are the queriesand storage of the social graph. NYC has the largestone at 3.5 GB and could easily be handled by commod-ity off-the-shelf hardware.

6. RELATED WORKOSNs consist of social structures and communica-

tions, and have been studied from these two perspec-tives. The structure of the social graph and it’s evo-lution has been studied for a number of the most pop-ular OSNs [2, 3, 13, 15]. At a high level, studies ofOSN interactions can be characterized through modelsof information diffusion [7, 25]. Another approach isto focus on patterns of communications with respect tothe social graph by defining a weighted communicationgraph [30, 9]. Later studies have observed that the com-munication graph changes over time and so studied itstemporal aspects [27, 29]. Finally, recent studies havelooked at the interactions of users with OSN servicesthrough click-streams collected at ISPs [21, 5].

In our work, we consider network traffic of user in-teractions within regional networks. Most related arethe studies by Nazir et al., who have looked at the net-work traffic of third-party applications in Facebook, byLiben-Nowell et al., who considered the role geographicproximity of users plays in friendship link formation,and by Carrasco et al., who investigated the role of prox-imity in email communications [16, 14, 6].

Related to the distributed OSN proposed in this pa-per is research on optimizing placement data acrossdata centers with respect to user demand [1, 32] and onbalancing traffic demands with economic costs of datacenters [19, 28]. A recent work by Pujol et al. uses socialgraph structure to more efficiently distribute OSN statewithin a data center [18]. Our work is different in thatwe propose distribution based on locality of interest anduse short caching time.

Beyond data center based OSN design, a number ofprojects have introduced interfaces and peer-to-peer de-

signs.5 Our view is that reliable OSN service is bestprovided by commercial ventures and single adminis-trative domains, and so we attempt to introduce thebenefits of infrastructure distribution to that setting.

7. CONCLUSIONSThe deployment of regional servers to tunnel TCP

connections and cache regional OSN state is effective atimproving the responsiveness and efficiency of an OSNservice. Because of the long network latencies and highloss rates on Internet paths between U.S. data centersand other regions, as well as a high locality of interestin these regions, regional servers are an attractive alter-native for the expansion of Facebook’s infrastructure.Combining the reduction of traffic to U.S. data centers,the potential to decrease the cost of that traffic, and thevery modest storage requirements, cost effective deploy-ment of regional servers could follow a model similar tothat used by Akamai, which embeds their servers withISPs. Finally, judging by the similarity and predictabil-ity of performance gains between Egypt, Russia, andSweden, we believe that the benefits of regional serverswill be at least as effective in regions even more dis-tant from U.S. data centers. Further studies might beneeded if a precise quantification of results is deemedworthwhile.

8. ACKNOWLEDGEMENTSWe would like to thank Christo Wilson and Krishna

Puttaswamy for making available their Facebook crawls.

9. REFERENCES[1] S. Agarwal et al. Volley: Automated data placement

for geo-distributed cloud services. In Usenix NSDI,April 2010.

[2] Y.-Y. Ahn et al. Analysis of topological characteristicsof huge online social networking services. In WorldWide Web Conference (WWW), May 2007.

[3] L. Backstrom et al. Group formation in large socialnetworks: membership, growth, and evolution. InACM KDD, August 2006.

[4] A. Bakre and B. R. Badrinath. I-TCP: Indirect TCPfor mobile hosts. In International Conference onDistributed Computing Systems (ICDCS), May 1995.

[5] F. Benevenuto et al. Characterizing user behavior inonline social networks. In ACM IMC, November 2009.

[6] J. Carrasco et al. Agency in social activityinteractions: The role of social networks in time andspace. Journal of Economic and Social Geography,99(5):562–583, December 2008.

[7] M. Cha et al. Characterizing social cascades in Flickr.In Sigcomm Workshop on Online Social Networks(WOSN), August 2008.

[8] G. Chen et al. Energy-aware server provisioning andload dispatching for connection-intensive internetservices. In Usenix NSDI, April 2008.

5http://diso-project.org/, http://daisycha.in/,http://appleseedproject.org/

[9] H. Chun et al. Comparison of online social relations involume vs interaction: a case study of CyWorld. InACM IMC, October 2008.

[10] Facebook. Statistics. http://www.facebook.com/press/info.php?statistics, 2010.

[11] T. Isdal et al. Leveraging BitTorrent for end hostmeasurements. In Passive and Active NetworkMeasurement (PAM), April 2007.

[12] N. Kennedy. Facebook’s growing infrastructurespending. http://www.niallkennedy.com/blog/2009/03/facebook-infrastructure-financing.html,March 2009.

[13] R. Kumar et al. Structure and evolution of onlinesocial networks. In ACM KDD, August 2006.

[14] D. Liben-Nowell et al. Geographic routing in socialnetworks. Proceedings of the National Academy ofSciences, 102(33):11623–11628, August 2005.

[15] A. Mislove et al. Measurement and analysis of onlinesocial networks. In ACM IMC, October 2007.

[16] A. Nazir et al. Network level footprints of Facebookapplications. In ACM IMC, November 2009.

[17] R. Prasad et al. Bandwidth estimation: metrics,measurement techniques, and tools. IEEE Network,17(6):27–35, November 2003.

[18] J. M. Pujol et al. The little engine(s) that could:scaling online social networks. In Sigcomm, August2010.

[19] A. Qureshi et al. Cutting the electric bill forinternet-scale systems. In Sigcomm, August 2009.

[20] D. Schafer. Reducing markup size. http://www.facebook.com/note.php?note_id=125015758919,September 2009.

[21] F. Schneider et al. Understanding online socialnetwork usage from a network perspective. In ACMIMC, November 2009.

[22] J. Sobel. Scaling out. http://www.facebook.com/note.php?note_id=23844338919, August 2008.

[23] A. Su et al. Drafting behind Akamai (travelocity-baseddetouring). Sigcomm Computer CommunicationsReview, 36(4):435–446, October 2006.

[24] D. M. Swany and R. Wolski. Data logistics in networkcomputing: The logistical session layer. In IEEENetwork Computing and Applications, October 2001.

[25] J. Tang et al. Temporal distance metrics for socialnetwork analysis. In Sigcomm Workshop on OnlineSocial Networks (WOSN), August 2009.

[26] P. Vajgel. Needle in a haystack: efficient storage ofbillions of photos. http://www.facebook.com/note.php?note_id=76191543919, April 2009.

[27] M. Valafar et al. Beyond friendship graphs: a study ofuser interactions in Flickr. In Sigcomm Workshop onSocial Networks (WOSN), August 2009.

[28] V. Valancius et al. Greening the internet with nanodata centers. In ACM CoNEXT, December 2009.

[29] B. Viswanath et al. On the evolution of userinteraction in Facebook. In Sigcomm Workshop onOnline Social Networks (WOSN), August 2009.

[30] C. Wilson et al. User interactions in social networksand their implications. In European Conference onComputer Systems (EuroSys), April 2009.

[31] Z. Yang. Every millisecond counts. http://www.facebook.com/note.php?note_id=122869103919,August 2009.

[32] Z. Zhang et al. Optimizing cost and performance inonline service provider networks. In Usenix NSDI,April 2010.