Measuringtheimpact:

StephenMerity/smerity.com @smerity

Smerity@CommonCrawl

ContinuingthecrawlDocumentingbestpractices

GuidesfornewcomerstoCommonCrawl+bigdataReferenceforseasonedveterans

Spendingmanyhoursblessingand/orcursingHadoop

Before:UniversityofSydney'11,Harvard'14

GoogleSydney,Freelancer.com,GrokLearning

banned@slashdot.org

Iwashopingoncreatingatoolthatwillautomaticallyextractsomeofthemostcommonmemes("ButdoesitrunLinux?"and

"InSovietRussia..."stylejokesetc)andIneededacorpus-

.Idointenselyapologise.

Iwroteaprimitive(threaded:S)webcrawlerandstarteditbeforeI

consideredrobots.txt

--PastSmerity(16/12/2007)

WheredidalltheHTTPreferrersgo?

Referrers:leakingbrowsinghistory

Ifyouclickfrom

to

http://www.reddit.com/r/sanfrancisco

http://www.sfbike.org/news/protected-bikeways-planned-for-the-embarcadero/

thenSFBikeknowsyoucamefromReddit

1)HowmanywebsitesisGoogleAnalytics(GA)on?

2)Howmuchofauser'sbrowsinghistorydoesGAcapture?

Top10kdomains:65.7%

Top100kdomains:64.2%

Topmilliondomains:50.8%

Itkeepsdroppingoff,butbyhowmuch..?

Estimateofcapturedbrowsinghistory...

?

ReferrersalloweasywebtrackingwhendoneatGoogle'sscale!

Noinformation!GA→!GA

Fullinformation!GA→GA

GA→!GA→GAGA→!GA→GA→!GA→GA→!GA→GA→!GA→GA

Keyinsight:leakedbrowsinghistory

GoogleonlyneedsoneineverytwolinkstohaveGAinordertohaveyourfullbrowsingpath*

*possiblylessiflinkgraph+clicktiming+machinelearningused

Estimatingleakedbrowserhistory

foreach :link={pageA}→{pageB}total_links+=1if{pageA}or{pageB}hasGA:

total_leaked+=1

Estimateofleakedbrowserhistoryissimply:total_leaked/total_links

JointprojectwithChadHornbaker*atHarvardIACS

*Bestfullnameever:CaptainCharlesLafforestHornbakerII

Thetask

GoogleAnalyticscount:" "

Generatelinkgraph

Mergelinkgraph&GAcount

.google-analytics.com/ga.jswww.winradio.net.auNoGA1www.winrar.com.cnGA6www.winratzart.comGA1www.winrenner.chGA244

domainA.com->domainB.com<totaltimes>

cnet-cnec-driver.softutopia.com->www.softutopia.com24

Excitingageofopendata

Opendata+

Opentools+

Cloudcomputing

WARCrawwebdata

WATmetadata(links,title,...)foreachpage

WETextractedtext

WARC=GAusagerawwebdata

WAT=hyperlinkgraphmetadata(links,title,...)foreachpage

Estimatingthetask'ssize

Pagelevel( ):http://en.wikipedia.org/3.5billionnodes,128billionedges,331GBcompressed

Subdomainlevel( ):101millionnodes,2billionedges,9.2GBcompressed

Decidedonusingsubdomainsinsteadofpagelevel

http:// /

Engineeringforscale

✓Usetheframeworkthatmatchesbest

✓Debuglocally

✓StandardHadoopoptimizations(combiner,compression,re-useJVMs...)

✓Manysmalljobs≫onebigjob

✓Gangliaformetrics&monitoring

Hadoop:'(

Hadoop:'(

Monitoring&metricswithGanglia

Engineeringforcost

✓AvoidHadoopifit'ssimpleenough✓Usespotinstanceseverywhere*✖UseEMRifhighlycostsensitive

(ElasticMapReduce=hostedHadoop)

*Everywherebutthemasternode!

Jugglingspotinstances

c1.xlargegoesfrom$0.58p/hto$0.064p/h

EMR:Thegood,thebad,theugly

significantlyeasier,oneclicksetup

priceisinsanewhenusingspotinstances(spot=$0.075withEMR=$0.12)

Guesshowmanylogfilesfora100nodecluster?

584,764+logfiles.

Ouch.

Costprojection

BestoptimizedsmallHadoopjob:1/177ththedatasetin23minutes(12c1.xlargemachines+Hadoopmaster)

Estimatedfulldatasetjob:~210TBforwebdata+~90TBforlinkdata~$60inEC2costs(177hoursofspotinstances)~$100inEMRcosts(avoidEMRforcost!)

Finalresults

29.96%of48milliondomainshaveGA(topmilliondomainswas50.8%)

Thatmeansthat

oneineverytwohyperlinkswillleakinformationtoGoogle

Thewiderimpact

WantBigOpenData?

WebData

Coverseverythingatscale!Languages...

Topics...Demographics...

Processingthewebisfeasible

Downloadingitisapain!CommonCrawldoesthatforyou

Processingitisscary!Bigdataframeworksexistandare(relatively)painless

Theseexperimentsaretooexpensive!Cloudcomputingmeansexperimentscanbejustafewdollars

Getstartednow..!

Wantrawwebdata?CommonCrawl.org

Wanthyperlinkgraph/webtables/RDFa?WebDataCommons.org

Wantexamplecodetogetyoustarted?https://github.com/Smerity/cc-warc-examples

Measuringtheimpact:

Fullwrite-up:http://smerity.com/cs205_ga/

StephenMerity/smerity.com @smerity