Web Search Engines

Brief History of Search Engines

Anatomy of a Search Engine Result Page

Anatomy of a Search Engine Result Page

Actors in Web Search

What Makes Web Search Difficult?

What Makes Web Search Difficult?

Expectations from a Search Engine

Web Growth

Search Data Centers

Cost of Data Centers

●  Data center facilities are heavy consumers of energy, accounting for between 1.1% and 1.5% of the world’s total energy use in 2010.

●  depreciation: old hardware need to be replaced ●  maintenance: failures need to be handled ●  operational: energy spending need to be

reduced

Major Components in a Web Search Engine

Web Crawling

•  Web  crawling  is  the  process  of  loca4ng,  fetching,  and  storing  the  pages  available  in  the  Web    •  Computer  programs  that  perform  this  task  are  referred  to  as  -­‐  crawlers  -­‐  spider  -­‐  harvesters    

Web Graph

Web Crawling Process

Issues in Web Crawling

•  Dynamics  of  the  Web  –  Web  growth  –  content  change    •  Malicious  intent  –  hos4le  sites  (e.g.,  spider  traps,  infinite  domain  name  generators)  –  spam  sites  (e.g.,  link  farms)    •  Web  site  proper4es  –  sites  with  restricted  content  (e.g.,  robot  exclusion),    –  unstable  sites  (e.g.,  variable  host  performance,  unreliable  networks)  

Robot Exclusion Protocol

Published Web Crawler Architectures

Open Source Web Crawlers

Nodejs crawler

Indexing

PageRank

Query Processing

Query Processing

Metrics

References

1. nginx.org/en/docs/http/load_balancing.html

2. http://www2014.kr/asset/slide/Scalability+and+Efficiency.pdf

3. http://www2014.kr/asset/slide/Social+Spam,+Campaigns,+Misinformation+and+Crowdturfing-www2014-tutorial.pdf

4. http://www.slideshare.net/freekbijl/web-30-explained-with-a-stamp?from=ss_embed