CS 349: WebBase 1

What the WebBase

can and can’t do?

Summary

• What is in the WebBase

• Performance Considerations

• Reliability Considerations

• Other sources of data

WebBase Repository

• 25 million web pages

• 150 GB (50 GB compressed)

• spread across roughly 30 disks

What kind of web pages?

• Everything you can imagine

• Infinite Web Pages (truncated at 100K)

• 404 Errors

• Very little correct HTML.

Duplicate Web Pages

• Duplicate Sites– Crawl Root Pages first– Find duplicates and assume same for

remainder of crawl

• Duplicate Hierarchies off of main page– Mirror Sites

• Duplicate Pages• Near Duplicate Pages

Shiva’s Test Results

• 36% Duplicates• 48% Near Duplicates• Largest Sets of Duplicates:

– TUCOWS (100)– MS IE Server Manuals (90)– Unix Help Pages (75)– RedHat Linux Manual (55)– Java API Doc (50)

Order of Web Pages

• First half million are root pages

• After that, pages in PageRank order

• Roughly by importance

Structure of Data

• magic number (4 bytes), packet length (4 bytes), packet (~2K bytes)

• packet is compressed

• packet contains: docID, URL, HTTP Headers, HTML data

Performance Issues: An Example

• One Disk Seek Per Document:– 10 ms seek latency +10 ms rotational latency– x ms read latency + x ms OS overhead– y ms processing

• Realistically 50 ms per document = 20 docs per second

• 25 million / 20 docs per second = 1,250,000 seconds = 2 weeks (too slow)

How fast does it have to be?

• Answer: 4ms per doc = 250 docs per second

• 25 million / 250 = 100,000 seconds = 1.2 days

• Reading + Uncompressing + Parsing == ~3 to 4 ms per document

• So there is not much room left for processing

How can you do something complicated?

• Really fast processing to generate some intermediate smaller results.

• Run complex processing over smaller results.

• Example: Duplicate Detection– Compute shingles from all documents– Find pairs of documents that share

shingles

Bulk Processing of Large Result Sets

• Example: Resolving Anchors

• Resolve URLs and save from - to in ASCII

• Compute 64 bit checksum of “To” URLs

• Bulk Merge against checksum - docid table

Reliability - Potential Sources of Problems

• Source code bug.

• Hardware failure

• OS failure

• Out of resources

Software Engineering Guidelines

• Number of bugs seen ~ log(size of dataset)

• Not just your bugs– OS bugs– Disk OS bugs

• Generate incremental results

Other Available Data

• Link Graph of the Web

• List of PageRanks

• List of URLs

Link Graph of the Web

• From DocID : To DocID

• Try red bars on Google to find backlinks

• Interesting Information

What is PageRank

• Measure of “importance”

• You are important if important things point to you

• Random surfer model

Uncrawled URLs

• Image links

• MailTo links

• CGI links

• Plain uncrawled HTML links

Summary

• WebBase has lots of web pages– very heterogeneous and weird

• Performance Considerations– code should be very very fast– use bulk processing

• Reliability Considerations– write out intermediate results

• Auxiliary data