Evaluating Novelty and Diversity Charles Clarke School of Computer Science University of Waterloo...
-
Upload
rhiannon-derbyshire -
Category
Documents
-
view
214 -
download
0
Transcript of Evaluating Novelty and Diversity Charles Clarke School of Computer Science University of Waterloo...
Evaluating Novelty and Diversity
Charles Clarke
School of Computer Science
University of Waterloo
two talks in one!
Goals for Evaluation Measures
• meaningful• tractable• reusable
Evaluation Framework
We examine a framework for evaluation.
Specific measures covered by the framework include:
Clarke et al. (SIGIR ’08)Agrawal et al. (WSDM ’09)Clarke et al. (ICTIR ‘09)
Talk #1: Evaluating Diversity
Charles Clarke
School of Computer Science
University of Waterloo
Query: “windows”
1. Microsoft Windowsa) When will Windows 7 be released?
b) What’s the Windows update URL?
c) I want to download Windows Live Essentials
2. House windowsa) Where can I buy replacement windows?
b) What brands are available?
c) Aluminum or vinyl?
3. Windows Restaurant, Las Vegas
Nuggets
Nugget = any binary property of a document
Provides address of a Pella dealer. Discusses history of the Windows OS. Is the Windows update page.
(factual, topical and navigational)
Problem: potentially thousands per query.
Evaluation
• Model user information needs using nuggets. Different users will be interested in different combinations of nuggets.
• Express judgments in terms of nuggets. Judgments may be automatic or manual. Judgments are binary: Does this document contain this nugget?
• Nuggets link users and documents
Interdependencies
Problem: Complex interdependencies between nuggets.
Three possible simplifying assumptions:
1. User interested in nugget A will always be interested in nugget B.
2. User interested in nugget A will never be interested in nugget B.
3. Nuggets A and B are independent.
Possible Assumption #1
If a user interested in nugget A will always be interested in nugget B, then A and B can be treated as the same nugget.
Possible Assumption #2
A user interested in nugget A will never be interested in nugget B (and vice versa). A user’s interest in nugget A depends on their interest in nugget B.
Nugget A and nugget be may be viewed as representing different interpretations of the query.
Query: “windows”
1. Microsoft Windowsa) When will Windows 7 be released?
b) What’s the Windows update URL?
c) I want to download Windows Live Essentials
2. House windowsa) Where can I buy replacement windows?
b) What brands are available?
c) Aluminum or vinyl?
3. Windows Restaurant, Las Vegas
Query Interpretations
• Assume M interpretations• Compute any effectiveness measure with
respect to each interpretation (Sj)
• Compute weighted average (where pj is probability of interpretation j)
• Agrawal et al, 2009
Possible Assumption #3
A user’s interest in nugget A is independent of their interest in nugget B.
The probability that the user is interested in nugget A is a constant (pA).
The probability that the user is interested in nugget B is a constant (pB).
Query: “windows”
1. Microsoft Windowsa) When will Windows 7 be released?
b) What’s the Windows update URL?
c) I want to download Windows Live Essentials
2. House windowsa) Where can I buy replacement windows?
b) What brands are available?
c) Aluminum or vinyl?
3. Windows Restaurant, Las Vegas
Relevance framework
A document is relevant if it contains any relevant information (with N nuggets).
Relevance
• Assume constant user probabilities• Assume constant document probabilities• J(d, i) = 1 iff document d is judged to
contain nugget i
count the nuggets
Probability of Relevance
Estimated probability of relevance replaces relevance in standard evaluation measures, including nDCG, MAP, and Rank-biased precision.
Assumptions #2 and #3 can then be combined.
Other estimation methods possible.
Research Issues (talk #1)
• Identifying nuggets automatically– Clustering– Co-clicks– Query refinement
• Automatic judging– Patterns– Classification
• How many nuggets are enough?• Estimating probability of relevance
Conclusions (talk #1)
• Evaluating diversity requires us to model and represent the diversity.
• Nuggets represent one possible solution.• Simple user model; simple assumptions;
simple judging.
Questions?
Talk #1: Evaluating Diversity
Charles Clarke
School of Computer Science
University of Waterloo
Intermission
The TREC 2009 Web Track• traditional adhoc task• novelty and diversity task• ClueWeb09 dataset (one billion pages)• explore effectiveness measures• http://plg.uwaterloo.ca/~trecweb
Intermission: Free sample topic
<topic number=0> <query> physical therapist </query> <description> The user requires information regarding the profession and the services it provides. </description> <subtopic number=1> What does a physical therapist do? </subtopic> <subtopic number=2> Where can I find a physical therapist? </subtopic> <subtopic number=3> How much does physical therapy cost per hour? </subtopic> …
Talk #2: Evaluating Novelty
Charles Clarke
School of Computer Science
University of Waterloo
Novelty
• Novelty depends on diversity.• Previous talk considered probability of
relevance in isolation (e.g., for the top-ranked document).
• In this talk we will examine how user context impacts the probability of relevance.
User context
Simplest context model
• Ranked list• User scans result 1, 2, 3, 4, 5, … in order.• Novelty of result k considered in light of
the first k-1 results.
Relevance framework
Relevance
Assuming constant probabilities.
Beyond the ranked list
Research issues (talk #2)
• Better user models• Prior browsing context, local context, etc.• Evaluating impact of result presentation
methods– Better captions– Query suggestions– Instant answers (stock quotes, weather,
product prices, definitions)
Conclusions (talk #2)
• Modeling and representing diversity allows us to consider novelty.
• User models should be simple enough to be tractable.
• User models should be complex enough to be meaningful.
Questions?
Talk #2: Evaluating Novelty
Charles Clarke
School of Computer Science
University of Waterloo