2007.1.18 - SLIDE 1IS 240 – Spring 2007

Prof. Ray Larson

University of California, Berkeley

School of Information

Tuesday and Thursday 10:30 am - 12:00 pm

Spring 2007http://courses.ischool.berkeley.edu/i240/s07/

Principles of Information Retrieval

Lecture 2: Basic Concepts

2007.1.18 - SLIDE 2IS 240 – Spring 2007

Review – IR History

• Journal Indexes

• “Information Explosion” following WWII– Cranfield Studies of indexing languages and

information retrieval– Paper by Joyce and Needham on Thesauri for

IR.– Development of bibliographic databases

• Chemical Abstracts • Index Medicus -- production and Medlars

searching

2007.1.18 - SLIDE 3IS 240 – Spring 2007

Development of IR Theory and Practice

• Phase I: circa 1955-1975– Foundational research– Fundamental IR concepts advanced in

research environment

• Phase II: 1975 to present– Slow adoption of IR research into operational

systems– Accelerated in mid-1990’s due to WWW

search engines

2007.1.18 - SLIDE 4IS 240 – Spring 2007

Historical Milestones in IR Research

• 1958 Statistical Language Properties (Luhn)• 1960 Probabilistic Indexing (Maron & Kuhns)• 1961 Term association and clustering (Doyle)• 1965 Vector Space Model (Salton)• 1968 Query expansion (Roccio, Salton)• 1972 Statistical Weighting (Sparck-Jones)• 1975 2-Poisson Model (Harter, Bookstein,

Swanson)• 1976 Relevance Weighting (Robertson, Sparck-

Jones)• 1980 Fuzzy sets (Bookstein)• 1981 Probability without training (Croft)

2007.1.18 - SLIDE 5IS 240 – Spring 2007

Historical Milestones in IR Research (cont.)

• 1983 Linear Regression (Fox)• 1983 Probabilistic Dependence (Salton, Yu)• 1985 Generalized Vector Space Model (Wong,

Rhagavan)• 1987 Fuzzy logic and RUBRIC/TOPIC (Tong,

et al.)• 1990 Latent Semantic Indexing (Dumais,

Deerwester)• 1991 Polynomial & Logistic Regression

(Cooper, Gey, Fuhr)• 1992 TREC (Harman)• 1992 Inference networks (Turtle, Croft)• 1994 Neural networks (Kwok)• 1998 Language Models (Ponte, Croft)

2007.1.18 - SLIDE 6IS 240 – Spring 2007

Development of Bibliographic Databases

• Chemical Abstracts Service first produced “Chemical Titles” by computer in 1961.

• Index Medicus from the National Library of Medicine soon followed with the creation of the MEDLARS database in 1961.

• By 1970 Most secondary publications (indexes, abstract journals, etc) were produced by machine

2007.1.18 - SLIDE 7IS 240 – Spring 2007

Boolean IR Systems

• Synthex at SDC, 1960• Project MAC at MIT, 1963 (interactive)• BOLD at SDC, 1964 (Harold Borko)• 1964 New York World’s Fair – Becker and

Hayes produced system to answer questions (based on airline reservation equipment)

• SDC began production for a commercial service in 1967 – ORBIT

• NASA-RECON (1966) becomes DIALOG• 1972 Data Central/Mead introduced LEXIS –

Full text of legal information• Online catalogs – late 1970’s and 1980’s

2007.1.18 - SLIDE 8IS 240 – Spring 2007

Experimental IR systems

• Probabilistic indexing – Maron and Kuhns, 1960• SMART – Gerard Salton at Cornell – Vector

space model, 1970’s• SIRE at Syracuse• I3R – Croft• Cheshire I (1990)• TREC – 1992• Inquery• Cheshire II (1994)• MG (1995?)• Lemur (2000?)

2007.1.18 - SLIDE 9IS 240 – Spring 2007

The Internet and the WWW

• Gopher, Archie, Veronica, WAIS• Tim Berners-Lee, 1991 creates WWW at

CERN – originally hypertext only• Web-crawler• Lycos• Alta Vista• Inktomi• Google• (and many others)

2007.1.18 - SLIDE 10IS 240 – Spring 2007

Information Retrieval – Historical View

• Boolean model, statistics of language (1950’s)

• Vector space model, probablistic indexing, relevance feedback (1960’s)

• Probabilistic querying (1970’s)

• Fuzzy set/logic, evidential reasoning (1980’s)

• Regression, neural nets, inference networks, latent semantic indexing, TREC (1990’s)

• DIALOG, Lexus-Nexus, • STAIRS (Boolean based) • Information industry

(O($B))• Verity TOPIC (fuzzy logic)• Internet search engines

(O($100B?)) (vector space, probabilistic)

Research Industry

2007.1.18 - SLIDE 11IS 240 – Spring 2007

Research Sources in Information Retrieval

• ACM Transactions on Information Systems• Am. Society for Information Science Journal• Document Analysis and IR Proceedings (Las

Vegas)• Information Processing and Management

(Pergammon)• Journal of Documentation• SIGIR Conference Proceedings• TREC Conference Proceedings• Much of this literature is now available online

2007.1.18 - SLIDE 12IS 240 – Spring 2007

Research Systems Software

• INQUERY (Croft)• OKAPI (Robertson)• PRISE (Harman)

– http://potomac.ncsl.nist.gov/prise• SMART (Buckley)• MG (Witten, Moffat)• CHESHIRE (Larson)

– http://cheshire.berkeley.edu• LEMUR toolkit (Callan)• Lucene (Pederson)• Others

2007.1.18 - SLIDE 13IS 240 – Spring 2007

Background Concepts for IR

• User Information Needs• Controlled Vocabularies (Pre and Post-

coordination)• Indexing Languages• IR definitions and concepts

– Documents– Queries– Collections– Evaluation– Relevance (we’ll spend the most time on this today)

2007.1.18 - SLIDE 14IS 240 – Spring 2007

User Information Need

• Why build IR systems at all?

• People have different and highly varied needs for information

• People often do not know what they want, or may not be able to express it in a usable form– Filling the gaps in Boulding’s “Image”

• How to satisfy these user needs for information?

2007.1.18 - SLIDE 15IS 240 – Spring 2007

Controlled Vocabularies

• Vocabulary control is the attempt to provide a standardized and consistent set of terms (such as subject headings, names, classifications, or the thesauri discussed by Joyce and Needham) with the intent of aiding the searcher in finding information.

• Controlled vocabularies are a kind of metadata:– Data about data– Information about information

2007.1.18 - SLIDE 16IS 240 – Spring 2007

Pre- and Postcoordination

• Precoordination relies on the indexer (librarian, etc.) to construct some adequate representation of the meaning of a document.

• Postcoordination relies on the user or searcher to combine more atomic concepts in the attempt to describe the documents that would be considered relevant.

2007.1.18 - SLIDE 17IS 240 – Spring 2007

Structure of an IR System

SearchLine

Interest profiles& Queries

Documents & data

Rules of the game =Rules for subject indexing +

Thesaurus (which consists of

Lead-InVocabulary

andIndexing

Language

StorageLine

Potentially Relevant

Documents

Comparison/Matching

Store1: Profiles/Search requests

Store2: Documentrepresentations

Indexing (Descriptive and

Subject)

Formulating query in terms of

descriptors

Storage of profiles

Storage of Documents

Information Storage and Retrieval System

Adapted from Soergel, p. 19

2007.1.18 - SLIDE 18IS 240 – Spring 2007

Uses of Controlled Vocabularies

• Library Subject Headings, Classification and Authority Files.

• Commercial Journal Indexing Services and databases

• Yahoo, and other Web classification schemes

• Online and Manual Systems within organizations– SunSolve– MacArthur

2007.1.18 - SLIDE 19IS 240 – Spring 2007

Types of Indexing Languages

• Uncontrolled Keyword Indexing• Folksonomies

– Uncontrolled but somewhat structured)

• Indexing Languages– Controlled, but not structured

• Thesauri– Controlled and Structured

• Classification Systems– Controlled, Structured, and Coded

• Faceted Classification Systems and Thesauri

2007.1.18 - SLIDE 20IS 240 – Spring 2007

Thesauri

• A Thesaurus is a collection of selected vocabulary (preferred terms or descriptors) with links among Synonymous, Equivalent, Broader, Narrower and other Related Terms

2007.1.18 - SLIDE 21IS 240 – Spring 2007

Thesauri (cont.)

• National and International Standards for Thesauri– ANSI/NISO z39.19--1994 -- American National

Standard Guidelines for the Construction, Format and Management of Monolingual Thesauri

– ANSI/NISO Draft Standard Z39.4-199x -- American National Standard Guidelines for Indexes in Information Retrieval

– ISO 2788 -- Documentation -- Guidelines for the establishment and development of monolingual thesauri

– ISO 5964-- Documentation -- Guidelines for the establishment and development of multilingual thesauri

2007.1.18 - SLIDE 22IS 240 – Spring 2007

Development of a Thesaurus

• Term Selection.

• Merging and Development of Concept Classes.

• Definition of Broad Subject Fields and Subfields.

• Development of Classificatory structure

• Review, Testing, Application, Revision.

2007.1.18 - SLIDE 23IS 240 – Spring 2007

Categorization Summary

• Processes of categorization underlie many of the issues having to do with information organization

• Categorization is messier than our computer systems would like

• Human categories have graded membership, consisting of family resemblances.

• Family resemblance is expressed in part by which subset of features are shared

• It is also determined by underlying understandings of the world that do not get represented in most systems

2007.1.18 - SLIDE 24IS 240 – Spring 2007

Classification Systems

• A classification system is an indexing language often based on a broad ordering of topical areas. Thesauri and classification systems both use this broad ordering and maintain a structure of broader, narrower, and related topics. Classification schemes commonly use a coded notation for representing a topic and it’s place in relation to other terms.

2007.1.18 - SLIDE 25IS 240 – Spring 2007

Classification Systems (cont.)

• Examples:– The Library of Congress Classification System– The Dewey Decimal Classification System– The ACM Computing Reviews Categories– The American Mathematical Society

Classification System

2007.1.18 - SLIDE 26IS 240 – Spring 2007

Central Concepts in IR

• Documents

• Queries

• Collections

• Evaluation

• Relevance

2007.1.18 - SLIDE 27IS 240 – Spring 2007

Documents

• What do we mean by a document?– Full document?– Document surrogates?– Pages?

• Buckland (JASIS, Sept. 1997) “What is a Document”

• Bates (JASIST, June 2006) “Fundamental Forms of Information”

• Are IR systems better called Document Retrieval systems?

• A document is a representation of some aggregation of information, treated as a unit.

2007.1.18 - SLIDE 28IS 240 – Spring 2007

Collection

• A collection is some physical or logical aggregation of documents– A database– A Library– A index?– Others?

2007.1.18 - SLIDE 29IS 240 – Spring 2007

Queries

• A query is some expression of a user’s information needs

• Can take many forms– Natural language description of need– Formal query in a query language

• Queries may not be (and probably aren’t) accurate expressions of the information need– Differences between conversation with a

person and formal query expression

2007.1.18 - SLIDE 30IS 240 – Spring 2007

Evaluation

• Why Evaluate?

• What to Evaluate?

• How to Evaluate?

2007.1.18 - SLIDE 31IS 240 – Spring 2007

Why Evaluate?

• Determine if the system is desirable

• Make comparative assessments

• Others?

2007.1.18 - SLIDE 32IS 240 – Spring 2007

What to Evaluate?

• How much of the information need is satisfied.

• How much was learned about a topic.

• Incidental learning:– How much was learned about the collection.– How much was learned about other topics.

• How inviting the system is.

2007.1.18 - SLIDE 33IS 240 – Spring 2007

What to Evaluate?

What can be measured that reflects users’ ability to use system? (Cleverdon 66)

– Coverage of Information– Form of Presentation– Effort required/Ease of Use– Time and Space Efficiency– Recall

• proportion of relevant material actually retrieved

– Precision• proportion of retrieved material actually relevant

effe

ctiv

enes

s

2007.1.18 - SLIDE 34IS 240 – Spring 2007

Relevance

• In what ways can a document be relevant to a query?– Answer precise question precisely.– Partially answer question.– Suggest a source for more information.– Give background information.– Remind the user of other knowledge.– Others ...

2007.1.18 - SLIDE 35IS 240 – Spring 2007

Relevance

• “Intuitively, we understand quite well what relevance means. It is a primitive “y’ know” concept, as is information for which we hardly need a definition. … if and when any productive contact [in communication] is desired, consciously or not, we involve and use this intuitive notion or relevance.”

» Saracevic, 1975 p. 324

2007.1.18 - SLIDE 36IS 240 – Spring 2007

Relevance

• How relevant is the document– for this user, for this information need.

• Subjective, but• Measurable to some extent

– How often do people agree a document is relevant to a query?

• How well does it answer the question?– Complete answer? Partial? – Background Information?– Hints for further exploration?

2007.1.18 - SLIDE 37IS 240 – Spring 2007

Relevance Research and Thought

• Review to 1975 by Saracevic

• Reconsideration of user-centered relevance by Schamber, Eisenberg and Nilan, 1990

• Special Issue of JASIS on relevance (April 1994, 45(3))

2007.1.18 - SLIDE 38IS 240 – Spring 2007

Saracevic

• Relevance is considered as a measure of effectiveness of the contact between a source and a destination in a communications process– Systems view– Destinations view– Subject Literature view– Subject Knowledge view– Pertinence– Pragmatic view

2007.1.18 - SLIDE 39IS 240 – Spring 2007

Define your own relevance

• Relevance is the (A) gage of relevance of an (B) aspect of relevance existing between an (C) object judged and a (D) frame of reference as judged by an (E) assessor

• Where…

From Saracevic, 1975 and Schamber 1990

2007.1.18 - SLIDE 40IS 240 – Spring 2007

A. Gages

• Measure

• Degree

• Extent

• Judgement

• Estimate

• Appraisal

• Relation

2007.1.18 - SLIDE 41IS 240 – Spring 2007

B. Aspect

• Utility

• Matching

• Informativeness

• Satisfaction

• Appropriateness

• Usefulness

• Correspondence

2007.1.18 - SLIDE 42IS 240 – Spring 2007

C. Object judged

• Document

• Document representation

• Reference

• Textual form

• Information provided

• Fact

• Article

2007.1.18 - SLIDE 43IS 240 – Spring 2007

D. Frame of reference

• Question

• Question representation

• Research stage

• Information need

• Information used

• Point of view

• request

2007.1.18 - SLIDE 44IS 240 – Spring 2007

E. Assessor

• Requester

• Intermediary

• Expert

• User

• Person

• Judge

• Information specialist

2007.1.18 - SLIDE 45IS 240 – Spring 2007

Schamber, Eisenberg and Nilan

• “Relevance is the measure of retrieval performance in all information systems, including full-text, multimedia, question-answering, database management and knowledge-based systems.”

• Systems-oriented relevance: Topicality

• User-Oriented relevance

• Relevance as a multi-dimensional concept

2007.1.18 - SLIDE 46IS 240 – Spring 2007

Schamber, et al. Conclusions

• “Relevance is a multidimensional concept whose meaning is largely dependent on users’ perceptions of information and their own information need situations

• Relevance is a dynamic concept that depends on users’ judgements of the quality of the relationship between information and information need at a certain point in time.

• Relevance is a complex but systematic and measureable concept if approached conceptually and operationally from the user’s perspective.”

2007.1.18 - SLIDE 47IS 240 – Spring 2007

Froelich

• Centrality and inadequacy of Topicality as the basis for relevance

• Suggestions for a synthesis of views

2007.1.18 - SLIDE 48IS 240 – Spring 2007

Janes’ View

Topicality

Pertinence

Relevance

Utility

Satisfaction

2007.1.18 - SLIDE 49IS 240 – Spring 2007

Operational Definition of Relevance

• From the point of view of IR evaluation (as typified in TREC and other IR evaluation efforts)– Relevance is a term used for the relationship

between a users information need and the contents of a document where the user determines whether or not the contents are responsive to his or her information need

2007.1.18 - SLIDE 50IS 240 – Spring 2007

Readings and Discussion

• Joyce and Needham– Assigned index terms or Automatic?– Lattice theory (extension of Boolean algebra to

partially ordered sets)– Notice the Vector suggestion?

• Luhn– Document/Document similarity calculations based on

term frequency– KWIC indexes

• Doyle– Term associations