Evaluation of Image Retrieval Results
-
Upload
nash-mcfarland -
Category
Documents
-
view
26 -
download
8
description
Transcript of Evaluation of Image Retrieval Results
Evaluation of Image Retrieval Results
Relevant: images which meet user’s information need
Irrelevant: images which don’t meet user’s information need
Query: cat
Relevant Irrelevant
1
Accuracy
• Given a query, an engine classifies each image as “Relevant” or “Nonrelevant”
• The accuracy of an engine: the fraction of these classifications that are correct– (tp + tn) / ( tp + fp + fn + tn)
• Accuracy is a commonly used evaluation measure in machine learning classification work
• Why is this not a very useful evaluation measure in IR? 2
Relevant Nonrelevant
Retrieved tp fpNot Retrieved fn tn
3
Why not just use accuracy?
• How to build a 99.9999% accurate search engine on a low budget….
• People doing information retrieval want to find something and have a certain tolerance for junk.
Search for:
0 matching results found.
Unranked retrieval evaluation:Precision and Recall
Precision: fraction of retrieved images that are relevant
Recall: fraction of relevant image that are retrieved
Precision: P = tp/(tp + fp)Recall: R = tp/(tp + fn)
4
Relevant Nonrelevant
Retrieved tp fpNot Retrieved fn tn
5
Precision/Recall
You can get high recall (but low precision) by retrieving all images for all queries!
Recall is a non-decreasing function of the number of images retrieved
In a good system, precision decreases as either the number of images retrieved or recall increases This is not a theorem, but a result with strong empirical
confirmation
6
A combined measure: F
• Combined measure that assesses precision/recall tradeoff is F measure (weighted harmonic mean):
• People usually use balanced F1 measure– i.e., with = 1 or = ½
• Harmonic mean is a conservative average– See CJ van Rijsbergen, Information Retrieval
RP
PR
RP
F
2
2 )1(1)1(
11
7
Evaluating ranked results
• Evaluation of ranked results:– The system can return any number of results– By taking various numbers of the top returned
images (levels of recall), the evaluator can produce a precision-recall curve
8
A precision-recall curve
0.0
0.2
0.4
0.6
0.8
1.0
0.0 0.2 0.4 0.6 0.8 1.0
Pre
cisi
on
Recall
10
Interpolated precision
• Idea: If locally precision increases with increasing recall, then you should get to count that…
• So you take the max of precisions to right of value
11
A precision-recall curve
0.0
0.2
0.4
0.6
0.8
1.0
0.0 0.2 0.4 0.6 0.8 1.0
Pre
cisi
on
Recall
12
Summarizing a Ranking
• Graphs are good, but people want summary measures!
1. Precision and recall at fixed retrieval level• Precision-at-k: Precision of top k results• Recall-at-k: Recall of top k results• Perhaps appropriate for most of web search: all
people want are good matches on the first one or two results pages
• But: averages badly and has an arbitrary parameter of k
Summarizing a Ranking
14
2. Calculating precision at standard recall levels, from 0.0 to 1.0– 11-point interpolated average precision
• The standard measure in the early TREC competitions: you take the precision at 11 levels of recall varying from 0 to 1 by tenths of the images, using interpolation (the value for 0 is always interpolated!), and average them
• Evaluates performance at all recall levels
Summarizing a Ranking
15
A precision-recall curve
0.0
0.2
0.4
0.6
0.8
1.0
0.0 0.2 0.4 0.6 0.8 1.0
Pre
cisi
on
Recall
16
Typical 11 point precisions
• SabIR/Cornell 8A1 11pt precision from TREC 8 (1999)
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Recall
Precision
17
3. Average precision (AP)– Averaging the precision values from the rank positi
ons where a relevant image was retrieved– Avoids interpolation, use of fixed recall levels– MAP for query collection is arithmetic average
• Macro-averaging: each query counts equally
Summarizing a Ranking
18
Average Precision
19
3. Mean average precision (MAP)– summarize rankings from multiple queries by aver
aging average precision– most commonly used measure in research papers– assumes user is interested in finding many relevan
t images for each query
Summarizing a Ranking
21
4. R-precision– If we have a known (though perhaps incomplete)
set of relevant images of size Rel, then calculate precision of the top Rel images returned
– Perfect system could score 1.0.
Summarizing a Ranking
22
Summarizing a Ranking for Multiple Relevance levels
Query Excellent Relevant Irrelevant Key ideas
Van gogh paintings
- painting & good quality
Bridge - Can see full bridge, visually pleasing
<random person name>
- picture of one person verse group or no person
Hugh Grant
- picture of head, clear image, good quality
23
5. NDCG: Normalized Discounted Cumulative Gain– Popular measure for evaluating web search and rel
ated tasks– Two assumptions:
• Highly relevant images are more useful than marginally relevant image
• the lower the ranked position of a relevant image, the less useful it is for the user, since it is less likely to be examined
Summarizing a Ranking for Multiple Relevance levels
24
5. DCG: Discounted Cumulative Gain– the total gain accumulated at a particular rank p:
– Alternative formulation• emphasis on retrieving highly relevant images
Summarizing a Ranking for Multiple Relevance levels
25
5. DCG: Discounted Cumulative Gain– 10 ranked images judged on 0 3 relevance scale:‐
3, 2, 3, 0, 0, 1, 2, 2, 3, 0– discounted gain:
3, 2/1, 3/1.59, 0, 0, 1/2.59, 2/2.81, 2/3, 3/3.17, 0 = 3, 2, 1.89, 0, 0, 0.39, 0.71, 0.67, 0.95, 0
– DCG: 3, 5, 6.89, 6.89, 6.89, 7.28, 7.99, 8.66, 9.61, 9.61
Summarizing a Ranking for Multiple Relevance levels
26
5. NDCG– DCG values are often normalized by comparing the DCG at
each rank with the DCG value for the perfect ranking• makes averaging easier for queries with different numbers of rel
evant images
– Perfect ranking:3, 3, 3, 2, 2, 2, 1, 0, 0, 0
– ideal DCG values:3, 6, 7.89, 8.89, 9.75, 10.52, 10.88, 10.88, 10.88, 10
– NDCG values (divide actual by ideal):1, 0.83, 0.87, 0.76, 0.71, 0.69, 0.73, 0.8, 0.88, 0.88NDCG ≤1 at any rank position
Summarizing a Ranking for Multiple Relevance levels
27
Variance
• For a test collection, it is usual that a system does crummily on some information needs (e.g., MAP = 0.1) and excellently on others (e.g., MAP = 0.7)
• Indeed, it is usually the case that the variance in performance of the same system across queries is much greater than the variance of different systems on the same query.
• That is, there are easy information needs and hard ones!
28
Significance Tests
• Given the results from a number of queries, how can we conclude that ranking algorithm A is better than algorithm B?
• A significance test enables us to reject the null hypothesis (no difference) in favor of the alternative hypothesis (B is better than A)– the power of a test is the probability that the test will rejec
t the null hypothesis correctly
– increasing the number of queries in the experiment also increases power of test