1Internet search engines:
Fluctuations in document accessibility
• Wouter Mettrop
CWI, Amsterdam, The Netherlands
• Paul Nieuwenhuysen
Vrije Universiteit Brussel, and Universitaire Instelling Antwerpen, Belgium
http://www.cwi.nl/cwi/projects/IRT
Presented at NOM 2000 New York Hilton May 16-18, 2000
3
WWW
WWW: growing number of WWW servers
0
1000000
2000000
3000000
4000000
5000000
6000000
7000000
1993 1994 1995 1996 1997 1998 1999 2000
4
Internet based information sources: how many? how much?
In 2000:
• about 1 billion = 1000 million unique URLs in the total Internet
• about 10 terabyte (= 10 000 gigabyte) of text data
5
Internet information retrieval systems in 2000
• Several types of systems exist to retrieve information:
»Directories of selected sources categorised by subject, made by humans, mainly for browsing.
»Search systems, based on databases with machine made indexes, for word-based searching!
»“Meta-search” or “multi-threaded” search systems.
• We have studied and compared several well-known international (and a few national) word-based Internet search engines.
6
Internet information retrieval systems: evaluation criteria
• Many aspects/criteria can be considered in the evaluation of an Internet search engine, including
»coverage of documents present on WWW (studies exist);
»number of elements of a document, that are indexed to make them usable for retrieval = “depth of indexing”;…
• We started to study the depth of indexing and we were soon confronted with the fluctuations in the performance that do exist.
• We think that these fluctations are another important aspect of performance.
7
Internet information retrieval systems: our research group
The following persons have been involved in the research: • Louise Beijer (Hogeschool van Amsterdam, The Netherlands)
• Hans de Bruin (Unilever Research Laboratorium, Vlaardingen, The Netherlands)
• Hans de Man (JdM Documentaire Informatie, Vlaardingen, The Netherlands)
• Rudy Dokter (PNO Consultants, Hengelo, The Netherlands)
• Marten Hofstede ( Rijksuniversiteit Leiden, The Netherlands)
• Wouter Mettrop (CWI, Amsterdam, The Netherlands)
• Paul Nieuwenhuysen (Vrije Universiteit Brussel, Belgium)
• Eric Sieverts (Hogeschool van Amsterdam, and RUU, The Netherlands)
• Hanneke Smulders (Infomare, Terneuzen, The Netherlands)
• Hans van der Laan (Consultant, Leiderdorp, The Netherlands)
• Ditmer Weertman (ADLIB, Utrecht, The Netherlands)
8
Internet search engines: research on indexing functionality
• Our method to assess the indexing functionality of search engines:
»A “rich” test document with many element types has been created
»Identical test documents were placed at 8 sites in 2 countries
»A procedure was set up to assess retrieval
—in an automatic way
—with regular intervals
9
0 8 16
Number of our test documents thatwere retrieved at least once during theinvestigation period
Number of our test documents that were retrieved
10
Internet search engines : reachability
• 14 528 queries were sent to 13 search engines.
• Search engines were 721 times unreachable.
• The percentage of unreachability varies from nearly 0% to nearly 15%.
• The studied search engines were reachable for 95% of the queries.
11
Internet search engines: elements of test document studied
• title tag
• META-tags: keywords, description and author
• comment tag
• ALT tag
• text/URL of a link to a document
• H3 tag
• table header
• text of: an internal link, a reference anchor, a link to a sound file
• name of a sound file (au/wav/aiff/ra)
• text of a link to an image
• name of an image file (gif or jpg; inline or linked to)
• name of a Java applet (with or without extension class)
• terms after the first 100 lines in a document (200/…/700)
• the URL of a document
12
0 5 10 15 20 25
Number of studieddocument elementsthat were indexedat least once duringthe observationperiod
Number of the studied document elements that were indexed
13
Search engine indexing functionality: conclusions
• Considerable differences among search engines exist in their depth of indexing!
• Not “all of the static web” is indexed.
»Not each of our test documents/pages.
»Not all HTML elements of our test document/page.
• Some of the studied search engines showed changes in the indexing policy during the experiment fluctuations…
14
Internet search engines: fluctuations - definition
• A fluctuation appears when the result set of an observation
- i.e.
» one query or
» set of queries
misses documents with respect to a frame of reference
- i.e.
» other observations and
» knowledge about Web reality
15
Internet search engines: detecting fluctuations
• Through time: comparing result sets of 1 observation repeatedly performed
» Observation = one query or set of queries
» Frame of reference = other observations & web-knowledge
• One moment: consistency of result sets
» Observation = one query in set of queries
» Frame of reference = other observations
16
Internet search engines: types of fluctuations
• Through time: comparing result sets of 1 observation repeatedly performed
» “Document fluctuations”
» “Indexing fluctuations”
• One moment: consistency of result sets
» “Element fluctuations”
20
0 10 20 30 40 50 60 70 80 90 100
AltaVistaEuroferret
Excite
HotBot
Ilse
Infoseek Lycos
MSNNorthernLight
Search.nl
Snap
VindexWebcrawler
Average percentage offorgotten documents perround
Percentage of roundswith one or moreforgotten documents
Document fluctuations: experimental results
22
0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2
Average percentageof missed documentsper result set =Percentage of resultsets with missingdocuments
Indexing fluctuations:experimental results
24
Element fluctuations: example
0
1
2
3
4
5
6
7
8
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32
Number of documents retrieved by HotBot in every query in observation set 23
25
0 10 20 30 40 50 60 70 80 90 100
Average percentage ofmissed documents perresult-set
Percentage of result-setsthat were incomplete
Element fluctuations: experimental results
26
0 10 20 30 40 50
Lost by elementfluctuations
Lost by documentfluctuations
Lost by indexingfluctuations
Percentage of documents missed due to fluctuations
28
Fluctuations: remarks on “correctness”
• Fluctuations can be seen as “correct”, if they are reflections of alterations in:
»(web-) reality
— then document, indexing and element fluctuations are incorrect
»the indexed database of a search engine
— then only element fluctuations are incorrect
• Users do not care; they miss documents
29
Fluctuations:remarks on “size”
• No relation document / element fluctuations < ===== > “size”
• Percentage missed documents determines (with other reducing effects, such as depth of indexing) the effective size of an engine
30
Fluctuations:remarks on “importance”
• Users of information
»should be aware of the existence of fluctuations
»should observe them systematically
• Providers of information
»should be aware of the existence of fluctuations
• Quantitative analyses of the web are hindered by fluctuations
»scientometrics; citation analysis
»fluctuations lower the effective size of an index
31
Internet search engines: conclusions of our research
• Search engines differ in depth of indexing documents.
• Search engines make mistakes:
»They are subject to changes in indexing policy.(“indexing fluctuations”)
»They forget documents completely (“document fluctuations”)
»They miss documents in their result sets (“element fluctuations”).
• Considerable differences exist among search engines regarding these fluctuations.
32
Internet search engines: recommendations related to fluctuations
• Fluctuations are “normal”; do not be surprised; do not worry.
• Do not try to find a simple explanation to fully understand what happens.
• Known item searchers should repeat the search
»when using an engine with many element fluctuations; use other search terms;
»when using an engine with many document fluctuations: repeat later.
Top Related