1

Search strategy& tacticsGoverned byeffectiveness

& feedback

Tefko Saracevic

tefkos@rutgers.edu; http://comminfo.rutgers.edu/~tefko/

Central ideas

• Searches are all about being effective – finding what is needed, desired

• To measure effectiveness common measures of precision and recall are used

• Various tactics are used to affect effectiveness• Application of tactics for a desired results is a second

nature of professional searchers

2Tefko Saracevic

Knowing searching = also knowing tactics to reach toward desired effectiveness

As a searcher you are striving toward doing effective searches

ToC

1. Major concepts2. Measuring effectiveness3. Search tactics4. Feedback types

3 Tefko Saracevic

Effectiveness, relevance1. Major concepts

Tefko Saracevic 4

can

it be

don

e?

5

Some definitions

• Search statement (query):– set of search terms with logical connectors and attributes -

file and system dependent

• Search strategy (big picture):– overall approach to searching of a question

• selection of systems, files, search statements & tactics, sequence, output formats; cost, time aspects

• Search tactics (action choices):– choices & variations in search statements

• terms, connectors, attributes

Tefko Saracevic

6

Some definitions (cont.)

• Move :– modifications of search strategies or tactics that are aimed

at improving the results• e.g. from searching for digital AND libraries to digital(w)libraries

(Dialog) or “digital libraries” (Scopus)

• Cycle (particularly applicable to systems such as Dialog):– set of commands from start (begin) to viewing (type)

results, or from a viewing to a viewing command

Tefko Saracevic

Search strategy – big pictureall kinds of actions from start to end

• The entire approach to a search – selection of– files and sources to use– approaches in proceeding to search & combining

• search terms• operators to use• fields to search

– formats for viewing results– alternative actions if search yields

• too much• too little

– problem-solving heuristics

Tefko Saracevic 7

Search tactics – specific actionsconnected with a given search as it progresses

• A query - command line entered into a system in order to retrieve relevant information– terms, operators & attributes as allowed by a given system– vocabulary & syntax used in conjunction with connectors

&/or limiters to search a system• Again: depends on a system how it is done

– for example, a search statement might be: • in Dialog: b 47; ss (garbanzo? or chickpeas) and (hum?us or humus)• in Scopus you can enter: (garbanzo? or chickpeas) and (hum?us or

humus)• how would you do that in ?

Tefko Saracevic 8

9

Search performance is expressed in terms of:

• Effectiveness :– performance as to objectives

• to what degree did a search accomplish what desired?• how well done in terms of relevance?

• Efficiency :– performance as to costs

• at what cost and/or effort, time?

• But here we concentrate on effectiveness

Tefko Saracevic

Both are KEY concepts & criteria for selection of strategy, tactics & evaluation

10

Effectiveness criteria• Search tactics chosen & changed following some criteria

of accomplishment, such as:– none - no thought given – hard to imagine but happens sometimes

– relevance (most often) – is it relevant to a need, task, problem?

– magnitude (also very often)- is it to much retrieved to start with?

– output attributes- is it trustworthy? Authorities? Understandable? …

– topic – is it on the topic of the question?

• Tactics are chosen or altered interactively to match given criteria - feedback plays an important role

Tefko Saracevic

Knowing what choice of tactics may produce what results is key to professional searcher

11

Relevance:key concept in IR & key criterion for assessing effectiveness

“Relevant: having significant and demonstrable bearing on the matter at hand.”

“Relevance: the ability (as of an information retrieval system) to retrieve material that satisfies the needs of the user.” Merriam Webster (2005)

• Attribute/criterion reflecting effectiveness of exchange of inf. between people (users) & IR systems in communication contacts, based on valuation by people

Tefko Saracevic

Variation in names

• Relevance or relevant can be expressed in different terms– utility, pertinence, appropriate, significant, useful, germane,

applicable, valuable …

• But the concept still remains relevance That which we call relevance by any other word would

still be relevanceRelevance is relevance is relevance is relevance

thank you Shakespeare and Gertrude Stein

Tefko Saracevic 12

Some attributes of relevance

– in IR - user dependent – users are the final judges at the end

– multidimensional or faceted – users asses relevance on a number of factors, not only topic, but also authority, novelty, source …

– dynamic – users may change relevance assessments & criteria as search progresses or they learn or the problem is modified

– not only binary – users assess object not only as relevant/not relevant, but also more on a continuum (scale) - partially relevant included

– intuitively well understood – nobody has to explain to users what it is

Tefko Saracevic 13

ConsistencyIndividual differences

– consistency of assessment varies over time & between people to a (sometimes significant) degree – but (in)consistency is similar as in other processes e.g. indexing, classifying

• Subject expertise affects consistency of relevance judgments. Higher expertise results in higher consistency and stringency. Lower expertise results in lower consistency and more inclusion.

– relevance is not fixed; individual differences may be large

• It is most helpful to discuss with a user what kind of documents they may asses as relevant, what are their criteria and then try to incorporate that in searching & own assessments

Tefko Saracevic 14

15

Two major types of relevance – leading to a dichotomy of systems- & users-view of relevance:– Systems or algorithmic relevance

• the way a system or algorithm assesses relevance• relation between a query & objects in the file of a system as retrieved

or failed to be retrieved by a given algorithm

– User relevance• the way a user or user surrogate (searcher, specialist …) assesses

relevance• relation between information need or problem at hand of a user and

objects retrieved or out there in general

– But Topical relevance can be considered by both system & user • the degree to which topics or subjects of a query & topics or subjects

of objects (documents) in a file or retrieved match• relation between topic in the query & topic covered by the retrieved

objects, or objects in the file(s) of the system, or even in existence

Tefko Saracevic

16

User relevance can be further distinguished as to:

– Cognitive relevance or pertinence:• possible changes in user’s cognitive state due to objects retrieved• relation between state of knowledge & cognitive information need of a user

and the objects provided or in the file(s)

– Motivational or affective relevance• matching & satisfying user’s intentions, purposes, rationales, emotions• relation between intents, goals & motivations of a user & objects retrieved by a

system or in the file, or even in existence. Satisfaction

– Situational relevance or utility: • value of given objects or information for user’s situation or changes in situation• relation between the task or problem-at-hand & the objects retrieved (or in the

files). Relates to usefulness in decision-making, reduction of uncertainty ...

Tefko Saracevic

Precision and recall2. Measuring effectiveness

Tefko Saracevic 17

How?

The basic way by which effectiveness is established in IR searching is to compare

Tefko Saracevic 18

19

Effectiveness measuresTwo measures are used universally:Precision:

– of the stuff retrieved & given to user how much (what %) was relevant? – more formally: probability that given that an object is retrieved it is

relevant, or the ratio of relevant items retrieved to all items retrieved

Recall:– of the stuff that is relevant in the file how much (what %) was actually

retrieved?– more formally: probability that given that an object is relevant it is

retrieved, or the ratio of relevant items retrieved to all relevant items in a file

Tefko Saracevic

Tefko Saracevic 20

Items judgedRELEVANT

Items judgedNOT RELEVANT

ItemsRETRIEVED

aNo. of items retrieved & judged relevant

bNo. of items retrieved & judged not relevant (junk)

ItemsNOT RETRIEVED

cNo. of items not retrieved & relevant (missed relevant)

dNo. of items not retrieved & not relevant (missed junk)

Precision =a

a + b

Recall =a

a + c

High precision = maximize a, minimize b

High recall = maximize a, minimize c

Calculation

Examples of calculation

• If a system retrieved 16 documents and only 4 were assessed as relevant by a user then precision is 25%

• If a system had 40 documents in the file that were relevant but managed to retrieve only 12 of them then recall is 30%

• Precision is easy to establish, recall is not• union of retrievals is used as a “trick” to establish relative recall• you do a number of searches or use a number of tactics or

algorithms then you take together all that was retrieved (union) and have that assessed; then you can calculate a relative recall of each search, tactic or algorithm in respect to the union & see which one provides better or worse relative recall

Tefko Saracevic 21

22

Interpretation: PRECISION

• Precision= percent of relevant stuff you have in your set of answers retrieved– or conversely percent of junk (false drops) in the answer set– high precision = most stuff relevant– low precision = a lot of junk

• Some users demand high precision– do not want to wade through much stuff– but it comes at a price: relevant stuff may be missed

• Tradeoff almost always: high precision = low recall• we will get to tradeoff a bit later, but it is VERY important to consider

Tefko Saracevic

23

• A file may have a lot of relevant stuff• Recall = percent of that relevant stuff in the file that

you retrieved in your answer set– conversely percent of stuff you missed– high recall = you missed little– low recall = you missed a lot

• Some users demand high recall (e.g. PhD students doing a dissertation; patent lawyers; researchers writing a proposal or article)

– want to make sure that important stuff is not missed– but will have to pay a price of wading through a lot of junk

• Tradeoff almost always: high recall = low precision

Interpretation: RECALL

Tefko Saracevic

Tradeoff between precision and recallUsing it for different tactics

3. Search tactics

Tefko Saracevic 24

Aim of search tactics• Since there is no such thing as a perfect search:

– the aim is to search in a way that will insure a given, chosen or desired, level of effectiveness

• That means that we have to agree on– what do we mean by effectiveness in searching?

• general agreement is that retrieval of relevant answers is the major criterion for effectiveness of searching

– what measures do we use to express achievement of effectiveness in terms of relevance

• general agreement is that we use measures of precision and recall (or derivatives)

Tefko Saracevic 25

Thus many search tactics are geared toward achieving certain level of precision or recall

26

Precision-recall trade-off• USUALLY: precision & recall are inversely related

– higher recall usually means lower precision & vice versa

100 %

100 %0

Ideal

Usual

Impr

ovem

ents

Precision

Recall

Tefko Saracevic

27

Precision-recall trade-off

• It is like in life, usually:• you get some, lose some; you can't have your cake and eat it too

• Usually, but not always (keep in mind these are probabilities)

– when you have high precision most stuff you got is relevant or on the target but you also missed other stuff (could be a lot) that is relevant – it was left behind

– when you have high recall you did not miss much but you also got junk (could be a lot) - you have to wade through it

• There is price to pay either way– but then a lot of users are perfectly satisfied with and are aiming at high

precision • give me a few good things, that is all what I need

Tefko Saracevic

Effectiveness (Cleverdon’s) “laws”

High precision in retrieval is usually associated with low recall. High precision = low recall

High recall in retrieval is usually associated with low precision. High recall = low precision

You – with your user – have to decide if you are aiming toward high recall or high precision & you have to be aware of & explain tradeoffs

Tefko Saracevic 28

Originally formulated by Cyril Cleverdon,a UK librarian, in 1960s, in first IR tests done in Cranfield, UK, thus “Cranfield tests”You use different tactics for high

recall from those for high precision

Search tactics

Tefko Saracevic 29

Variable What variation possible?

1. LOGIC choice of connectors among terms AND, OR, NOT, W …

2. SCOPE no. of terms linked - ANDsA AND B vs A AND B AND C

3.EXHAUSTIVITY for each concept no. of related terms - OR connectionsA OR B vs. A OR B OR C

4. TERM SPECIFICITY for each concept level in hierarchybroader vs narrower terms

5. SEARCHABLE FIELDS choice for text terms & non-text attributese.g. titles only, limit as to years, various sources

6. FILE OR SYSTEM SPECIFIC CAPABILITIES

using capabilities, options availablee.g. given fields, ranking, sorting, linking

several ‘things’ or variables in a query can be selected & changed to affect effectiveness

Actions and consequences

Action ConsequenceSCOPE- adding more ANDs

Output size: downRecall: downPrecision: up

EXHAUSTIVITY- adding more ORs

Output size: upRecall: upPrecision: down

USE OF NOTs- adding more NOTs

Output size: downRecall: downPrecision: up

USING BROAD TERMS- low specificity

Output size: upRecall: upPrecision: down

USING PHRASES- high specificity

Output size: downRecall: downPrecision: up

Tefko Saracevic 30

BUT: each variation has consequence in outputif you do X then Y will happen

Tactics: what to do?

• These “laws” lead to precision & recall devices– tactics to increase/decrease precision or recall

Tefko Saracevic 31

To increase precision To increase recallSCOPE-add more ANDs-- you restrict

EXHAUSTIVITY-add more ORs- you enlarge

NOTs-add more NOTs-you eliminate

BROAD TERMS- you broaden terms

PHRASES- you become more specific

USE MORE TRUNCATION-you use grammatical variants of terms

Precision, recall devices

Tefko Saracevic 32

NARROWINGHigher precision

BOADENNINGHigher recall

More ANDsFewer ORsMore NOTsLess free textMore controlledLess synonymsNarrower termsMore specificLess truncationMore qualifiersMore limitsBuilding blocks

Fewer ANDsMore ORsFewer NOTsMore free textFewer controlledMore synonymsBroader termsLess specificMore truncationFewer qualifiers Fewer limitsCitation growing

With experience use of these devices will become second nature

33

Other tactics (but that gets us in the next unit on advanced searching)

• Citation growing:– find a relevant document– look for documents cited in– look for documents citing it– repeat on newly found relevant documents

• Building blocks– find documents with term A– review – add term B & so on

• Using different feedbacks– a most important tool

Tefko Saracevic

Types. Berry picking 4. Feedback

Tefko Saracevic 34

Feedback in searching

• Extremely important!• And constantly used, consciously or unconsciously• Formally:

– the process in which part of the output of a system is returned to its input in order to regulate its further output

• Simply put:– you search, find something that may be relevant, then look

at it, then on the basis of AHA! change your next query or tactic to get better or more stuff

Tefko Saracevic 35

36

Feedback in searching …• Any feedback implies loops

– a completion of a process provides information for modification, if any, for the next process

– information from output is used to change previous or create new input

• In searching:– some information taken from output of a search is used to

do something with next query (search statement)• examine what you got to decide what to do next in searching

– a basic tactic in searching

• Several feedback types used in searching– each used for different decisions

Tefko Saracevic

37

Feedback types• Content relevance feedback

– judge relevance of items retrieved– make decision what to do next

• switch files, change exhaustivity …

• Term relevance feedback– find relevant documents– examine what other terms used in those documents – search using additional terms

• also called query modification & in some systems done automatically

• Magnitude feedback– on the basis of size of output make tactical decisions

• often the size so big that documents are not examined but next search done to limit size

Tefko Saracevic

38

Feedback types (cont.)• Tactical review feedback

– after a number of queries (search statements) in the same search review tactics as to getting desired outputs

• review terms, logic, limits …– change tactics accordingly

• Strategic review feedback– after a while (or after consultation with user) review the

“big” picture on what searched and how• sources, terms, relevant documents, need satisfaction, changes in

question, query …– do next searches accordingly– used in reiterative searching

Tefko Saracevic

But then

• There are other ways to look at searching where many of these things are combined

• Here is one way that looks at the whole search as a process of complex wandering

• Shifting exploration & feedback are implied

Tefko Saracevic 39

40

Bates’ Berry-picking model of searching

“…moving through many actions towards a general goal of satisfactory completion of research related to information need.”– query is shifting (continually)

• as search progresses queries are changing• different tactics are used

– searcher (user) may move through a variety of sources

• new files, resources may be used

– think of your last serious search • isn’t this what you were doing?

Tefko Saracevic

Elaborated byMarcia BatesUCLA

41

Berry-picking …

– closer to the real behavior of information searchers– new information may provide new ideas, new directions

• feedback is used in various ways

– question is not satisfied by a single set of answers, but by a series of selections & bits of information found along the way

• results may vary & may have to be provided in appropriate ways & means

– you go as if berry-picking in a field

Tefko Saracevic

A berry-picking evolving search(from the article)

Tefko Saracevic 42

“1) The nature of the query   is an evolving one, rather than single and unchanging, and 2) the nature of the search process  is such that it follows a berrypicking pattern, instead of leading to a single best retrieved set.”

Conclusionssearch tactics have to be mastered

• Users are not concerned about searching but about finding

• Effective searching is a prerequisite for finding the right stuff

• Search tactics are critical for effective searching• They need to be understood and followed

if you do X you can expect Yin order to get to Y you have to know what X to do

Tefko Saracevic 43

Tefko Saracevic 44

Thank youM. C. Escher