Search

Information retrieval

Information and scientific progress

Let’s go back into time with this video:

The American Engineer

Information retrieval—and information science—emerge in a post-WW2 atmosphere where scientific and technical advance is key to social progress. Information serves this goal.

The basic IR task model

The goal of information retrieval is often described something like this:

Given a user information need (typically expressed as a search phrase or query), provide relevant documents from the collection.

Note that key term relevance. We’ll come back to it.

A lovely diagram of the IR process

Diagram from Jaime Arguello, UNC-Chapel Hill

Background: indexing

Indexing is the process of representing the subject of documents. An index can comprise terms extracted from the document itself or terms from another source.

An index language specifies the concepts used for indexing. An index language includes terms to represent concepts and a syntax for assigning (and perhaps combining, etc) terms.

A thesaurus is a type of index language in which terms are controlled and relationships between terms established.

Background: indexing

Precoordinate terms combine concepts at the point of indexing: foraging for wild mushrooms is a precoordinate term.

Postcoordinate terms are combined only at the time of search. Foraging and wild mushrooms are postcoordinate terms. (The searcher would input “foraging for wild mushrooms.”)

Background: indexing

A classification (in this context) is a subject scheme that expresses complex concepts by means of a notation; a document is placed in the single class that best expresses its subject. Most classifications are enumerative; individual classes are defined in advance. Some classifications have synthetic elements (ways to customize classes for particular geographic locations, for example).

A faceted classification is a synthetic scheme in which various elements of the subject are combined at the time of indexing (via a syntax) to create classes.

Background: indexing

Exhaustivity is an indexing principle that describes how much of a document’s content is indexed. (The “main” subject? The first three “important” subjects? Everything mentioned in the document? )

Specificity is an indexing principle that describes the level of abstraction at which a document’s content is indexed. (Is the subject information retrieval or information retrieval evaluation methods or precision and recall measures?)

Background: indexing

The Cranfield experiments test the relative effectiveness of three types of indexing languages: • Single words extracted from the text itself. • Concepts (phrases) extracted from the text itself. • Terms drawn from a controlled vocabulary.

Multiple variations of these three were also created (single words plus synonyms, controlled terms plus broader and narrower concepts).

Cranfield experiments

Although the Cranfield tests did not involve computers (!), they illustrate the basis from which information retrieval evaluations are still conducted, both in the conceptualization of the retrieval task and in the methods employed to measure it.

Also, the results were SHOCKING.

Again: social context of IR beginnings

Cleverdon: librarian at a college of aeronautics. Interested in new indexing technologies. Wants to systematically test them.

The unspoken retrieval context: Facilitate the efficient progress of technical specialists working on specific problems.

Cranfield protocol

1. Assemble a test collection of 1400 research papers, primarily in aerodynamics.

2. Index each document with test languages.

3. Develop test queries.

4. Determine relevance of each document to each query, using 1-4 scale.

5. Search collection using a specified query, index language, relevance level, and search rule (number of matching terms, for example).

6. Determine which of the retrieved documents are relevant to the query.

Cranfield protocol

Where do the test queries come from?

Cleverdon and colleagues asked the authors of scientific papers to state in the form of a question “the problem to which their paper was addressed.”)

(The example given is “small deflection theory of simple supported cylinders.”)

In using this to approximate an information seeker, Cranfield focuses on specialists who have an extremely good idea of what they are looking for.

Cranfield protocol

Where do the relevance judgments come from?

Students do the first pass, and the original document authors confirm.

Again, this may be an excellent approximation of “the real thing,” but it is a certain sort of approximation.

Cranfield protocol

Search results were then described according to the precision and recall measures.

Precision: Percentage of documents retrieved that are relevant.

Recall: Percentage of relevant documents retrieved out of all possible relevant documents.

Precision and recall

Say I have a collection of 10 documents. For a query X, documents 2, 7, and 8 are relevant.

My search for query X retrieves documents 7 and 8 but not 2; it also retrieves documents 4 and 5.

Precision for this search is 2 relevant documents out of 4 retrieved, or 50 percent.

Recall for this search is 2 relevant documents out of 3 possible relevant, or 67 percent.

Cranfield results

They do some math to normalize the results to a single number. And, the best performing index language is...

Single terms extracted from the document itself with some word forms (e.g., alloy + alloys) added.

OMG!

Cranfield legacy

The Cranfield methodology is used today in evaluating IR systems: the TREC (text retrieval conference) sponsored by NIST, in which researchers use a shared collection and set of queries to compare their systems’ effectiveness uses the same protocol...including human assessors.

The Cranfield measures and means of conceptualizing retrieval also persist.

2009 TReC Web track query

Query: appraisals

Description: How are home values appraised? I want to know how home appraisals are done.

Cranfield legacy

TREC assessors

Photo from NIST

Wait, what were those people assessing?

Relevance. Er, what is that, exactly?

Exactly.

Saracevic points out that IR—and, to some extent, information science as a whole—rests upon an uncertain concept. Is this problematic?

Relevance in information science

Saracevic noted a pattern in operational definitions of relevance as used by information scientists testing retrieval systems:

Relevance is the A of a B existing between a C and a D as determined by an E.

Relevance

Saracevic describes seven different views of relevance, which emphasize different elements of the concept:

1. Subject knowledge view. What is the relationship between a question and a subject?

2. Subject literature view. What is the relationship between a question and the academic literature on a subject?

Relevance

Saracevic describes seven different views of relevance, which emphasize different elements of the concept:

3. Logical view. What is the nature of the inference between a question and conclusions from a subject literature?

4. System view. What is the relationship between a file’s contents and a question, a user, and a subject?

Relevance

Saracevic describes seven different views of relevance, which emphasize different elements of the concept:

5. Destination’s view. What is the human judgment of the relation between a document and a question?

6. Pertinence view. What is the relationship between a seeker’s knowledge and a subject?

Relevance

Saracevic describes seven different views of relevance, which emphasize different elements of the concept:

7. Pragmatic view. What is the relation between a user’s problem and the information provided?

Back to representation and comparision

Diagram from Jaime Arguello, UNC-Chapel Hill

Document representations

In the Cranfield tests, documents were represented with a small set of index terms.

If terms from the document itself are the best index terms anyway (maybe!), then if we can use ALL the text of the document, why not? This is full-text indexing.

To speed processing, the text of documents is extracted and rearranged (tokenized) to form an inverted file.

Inverted index file

All the terms from all the documents are put in a table (T= term, and D= document). For a Boolean search (in a moment!) these tables just indicate presence and absence (1 or 0). Other models weight terms differently to rank retrieved documents.

Table from Joseph Tennis, UW

Other text processing

Retrieval systems may remove common words (stopwords) from the inverted file, although this is not as common as it once was.

Retrieval systems may also stem words to store just their roots.

Retrieval models

Most retrieval systems are variations of the following models:• Boolean.• Vector space.• Probabilistic.• Latent semantic indexing.

Boolean model

This is the oldest and simplest model; it puts most of the work on the searcher. Instead of searching for mere presence of index terms in a document (like the Cranfield tests), use Boolean operators (AND, OR, NOT) to describe the query more precisely.

( (“traumatic brain injury” OR TBI OR tbi OR “traumatic brain injuries”) AND (headache OR headaches) ) NOT concussion

Ranked results

Other retrieval models focus on using various statistical properties of texts (primarily) to produce ranked lists of results.

A key element in calculating these rankings is the frequency of significant terms. This measure relies on a property of language known as Zipf’s law.

Zipf’s law

Zipf’s law describes a distribution in which the ith most frequent object appears 1/iθ times the frequency of the most frequent object.

Er, what?

Zipf’s law applies to language, for any corpus, including a single document. So the most frequent word (say) takes up N percent of the document; the next most frequent word takes up N/2 percent of the document; the next most frequent word takes up N/3 percent, etc.

Zipf distributions appear for other phenomena as well.

Zipf’s law

A Zipf distribution at 300 data points.

Graph from Jacob Nielsen, useit.com

Implications of Zipf’s law

Zipf’s law holds true across languages, across types of text (written and spoken, etc) across complexity of topics, document genres, etc.

This statistical property implies that the most important words (for retrieval purposes) in a document are those that are most frequent in the document and least frequent in the collection.

Tf/idf

This relation is exploited by many retrieval models:

term frequency/ inverse document frequency

You can refine your calculation of tf-idf (e.g., by taking account of document length), but this is the basic idea.

Vector space model

The vector space model (originated by Salton), compares a query and a document as the correlation between two n-dimensional vectors. The cosine of the angle between the vectors is used to quantify the correlation.

Index terms in queries and documents are weighted based on tf-idf (and other properties).

Probabilistic model

The probabilistic model (introduced by Robertson and Sparck Jones) recursively refines an answer set based on guessing at the “ideal” answer.

Initial probabilistic models did not use weights, but later versions do.

Latent semantic indexing

Latent semantic indexing is a retrieval model that attempts to align documents with concepts, on the observation that terms that co-occur probably indicate something about a shared conceptual space. Documents and queries are mapped to a “concept space” and then compared.

LSA aims to return documents based on a conceptual match to a query, as opposed to a term match.

Lots of other stuff...

Individual systems may incorporate many different sorts of information to adjust the rankings produced by one of these basic models: using text structure to refine weights (titles, etc), using your location or previous Web history to adjust rankings, promoting recent content, etc.

What’s Google?

The original Google innovation was a ranking enhancement outside of the primary retrieval model.

They looked at the links to a page as a measure of information quality. This “PageRank” was used to adjust initial results of a retrieval system.

Google is not magic.