Rank Aggregation Methods for the Webklarson/teaching/F08-886/talks/WLuo.pdf · Rank Aggregation...
Transcript of Rank Aggregation Methods for the Webklarson/teaching/F08-886/talks/WLuo.pdf · Rank Aggregation...
![Page 1: Rank Aggregation Methods for the Webklarson/teaching/F08-886/talks/WLuo.pdf · Rank Aggregation Methods for the Web Cynthia Dwork Ravi Kumar Moni Naor D. Sivakumar Presented by: Wanying](https://reader035.fdocuments.us/reader035/viewer/2022062606/5fe1994a35ee9307b14f0a76/html5/thumbnails/1.jpg)
1
CS 886 Advanced Topics in Artificial Intelligence: Multiagent Systems
Rank Aggregation Methods for the WebCynthia Dwork Ravi Kumar Moni Naor D. Sivakumar
Presented by: Wanying Luo
![Page 2: Rank Aggregation Methods for the Webklarson/teaching/F08-886/talks/WLuo.pdf · Rank Aggregation Methods for the Web Cynthia Dwork Ravi Kumar Moni Naor D. Sivakumar Presented by: Wanying](https://reader035.fdocuments.us/reader035/viewer/2022062606/5fe1994a35ee9307b14f0a76/html5/thumbnails/2.jpg)
2
Outline
What is rank aggregation problemMotivationChallengesPreliminariesFirst result: spam resistance in meta-searchSecond result: Markov chain methodsApplicationsExperimentsConclusion
![Page 3: Rank Aggregation Methods for the Webklarson/teaching/F08-886/talks/WLuo.pdf · Rank Aggregation Methods for the Web Cynthia Dwork Ravi Kumar Moni Naor D. Sivakumar Presented by: Wanying](https://reader035.fdocuments.us/reader035/viewer/2022062606/5fe1994a35ee9307b14f0a76/html5/thumbnails/3.jpg)
3
What is rank aggregation problem
A
B
D
C
B
D
A
C
B
A
C
D
Based on different ranking techniques and criteria, we may get different results
![Page 4: Rank Aggregation Methods for the Webklarson/teaching/F08-886/talks/WLuo.pdf · Rank Aggregation Methods for the Web Cynthia Dwork Ravi Kumar Moni Naor D. Sivakumar Presented by: Wanying](https://reader035.fdocuments.us/reader035/viewer/2022062606/5fe1994a35ee9307b14f0a76/html5/thumbnails/4.jpg)
4
What is rank aggregation problem
A
B
D
C
B
D
A
C
B
A
C
D
Need to obtain a ”consensus” ranking of all the individual rankings
B
A
D
C
![Page 5: Rank Aggregation Methods for the Webklarson/teaching/F08-886/talks/WLuo.pdf · Rank Aggregation Methods for the Web Cynthia Dwork Ravi Kumar Moni Naor D. Sivakumar Presented by: Wanying](https://reader035.fdocuments.us/reader035/viewer/2022062606/5fe1994a35ee9307b14f0a76/html5/thumbnails/5.jpg)
5
Outline
What is rank aggregation problemMotivationChallengesPreliminariesFirst result: spam resistance in meta-searchSecond result: Markov chain methodsApplicationsExperimentsConclusion
![Page 6: Rank Aggregation Methods for the Webklarson/teaching/F08-886/talks/WLuo.pdf · Rank Aggregation Methods for the Web Cynthia Dwork Ravi Kumar Moni Naor D. Sivakumar Presented by: Wanying](https://reader035.fdocuments.us/reader035/viewer/2022062606/5fe1994a35ee9307b14f0a76/html5/thumbnails/6.jpg)
6
Motivation
Provide robust meta searchExamples of meta search engines
ClustyDogpileMetacrawler
Spamhttp://searchenginewatch.com/showPage.html?page=3483601
Commercial interests, e.g., sponsored links
![Page 7: Rank Aggregation Methods for the Webklarson/teaching/F08-886/talks/WLuo.pdf · Rank Aggregation Methods for the Web Cynthia Dwork Ravi Kumar Moni Naor D. Sivakumar Presented by: Wanying](https://reader035.fdocuments.us/reader035/viewer/2022062606/5fe1994a35ee9307b14f0a76/html5/thumbnails/7.jpg)
7
![Page 8: Rank Aggregation Methods for the Webklarson/teaching/F08-886/talks/WLuo.pdf · Rank Aggregation Methods for the Web Cynthia Dwork Ravi Kumar Moni Naor D. Sivakumar Presented by: Wanying](https://reader035.fdocuments.us/reader035/viewer/2022062606/5fe1994a35ee9307b14f0a76/html5/thumbnails/8.jpg)
8
![Page 9: Rank Aggregation Methods for the Webklarson/teaching/F08-886/talks/WLuo.pdf · Rank Aggregation Methods for the Web Cynthia Dwork Ravi Kumar Moni Naor D. Sivakumar Presented by: Wanying](https://reader035.fdocuments.us/reader035/viewer/2022062606/5fe1994a35ee9307b14f0a76/html5/thumbnails/9.jpg)
9
![Page 10: Rank Aggregation Methods for the Webklarson/teaching/F08-886/talks/WLuo.pdf · Rank Aggregation Methods for the Web Cynthia Dwork Ravi Kumar Moni Naor D. Sivakumar Presented by: Wanying](https://reader035.fdocuments.us/reader035/viewer/2022062606/5fe1994a35ee9307b14f0a76/html5/thumbnails/10.jpg)
10
Motivation
Provide robust meta searchExamples of meta search engines
ClustyDogpileMetacrawler
Spamhttp://searchenginewatch.com/showPage.html?page=3483601
Commercial interests, e.g., sponsored links
![Page 11: Rank Aggregation Methods for the Webklarson/teaching/F08-886/talks/WLuo.pdf · Rank Aggregation Methods for the Web Cynthia Dwork Ravi Kumar Moni Naor D. Sivakumar Presented by: Wanying](https://reader035.fdocuments.us/reader035/viewer/2022062606/5fe1994a35ee9307b14f0a76/html5/thumbnails/11.jpg)
11
Motivation
Provide robust meta searchExamples of meta search engines
ClustyDogpileMetacrawler
Spamhttp://searchenginewatch.com/showPage.html?page=3483601
Commercial interests, e.g., sponsored links
![Page 12: Rank Aggregation Methods for the Webklarson/teaching/F08-886/talks/WLuo.pdf · Rank Aggregation Methods for the Web Cynthia Dwork Ravi Kumar Moni Naor D. Sivakumar Presented by: Wanying](https://reader035.fdocuments.us/reader035/viewer/2022062606/5fe1994a35ee9307b14f0a76/html5/thumbnails/12.jpg)
12
Motivation
User may provide a variety of searching criteria
![Page 13: Rank Aggregation Methods for the Webklarson/teaching/F08-886/talks/WLuo.pdf · Rank Aggregation Methods for the Web Cynthia Dwork Ravi Kumar Moni Naor D. Sivakumar Presented by: Wanying](https://reader035.fdocuments.us/reader035/viewer/2022062606/5fe1994a35ee9307b14f0a76/html5/thumbnails/13.jpg)
13
Outline
What is rank aggregation problemMotivationChallengesPreliminariesFirst result: spam resistance in meta-searchSecond result: Markov chain methodsApplicationsExperimentsConclusion
![Page 14: Rank Aggregation Methods for the Webklarson/teaching/F08-886/talks/WLuo.pdf · Rank Aggregation Methods for the Web Cynthia Dwork Ravi Kumar Moni Naor D. Sivakumar Presented by: Wanying](https://reader035.fdocuments.us/reader035/viewer/2022062606/5fe1994a35ee9307b14f0a76/html5/thumbnails/14.jpg)
14
Challenges
Unrealistic to rank the entire collection of pages on the web
29.7 billion pages on the World Wide Web as of February 2007 (http://www.boutell.com/)
Most search engines rank only the top few hundred entries
![Page 15: Rank Aggregation Methods for the Webklarson/teaching/F08-886/talks/WLuo.pdf · Rank Aggregation Methods for the Web Cynthia Dwork Ravi Kumar Moni Naor D. Sivakumar Presented by: Wanying](https://reader035.fdocuments.us/reader035/viewer/2022062606/5fe1994a35ee9307b14f0a76/html5/thumbnails/15.jpg)
15
Challenges
Unrealistic to rank the entire collection of pages on the web
29.7 billion pages on the World Wide Web as of February 2007 (http://www.boutell.com/)
Most search engines rank only the top few hundred entries
![Page 16: Rank Aggregation Methods for the Webklarson/teaching/F08-886/talks/WLuo.pdf · Rank Aggregation Methods for the Web Cynthia Dwork Ravi Kumar Moni Naor D. Sivakumar Presented by: Wanying](https://reader035.fdocuments.us/reader035/viewer/2022062606/5fe1994a35ee9307b14f0a76/html5/thumbnails/16.jpg)
16
Outline
What is rank aggregation problemMotivationChallengesPreliminariesFirst result: spam resistance in meta-searchSecond result: Markov chain methodsApplicationsExperimentsConclusion
![Page 17: Rank Aggregation Methods for the Webklarson/teaching/F08-886/talks/WLuo.pdf · Rank Aggregation Methods for the Web Cynthia Dwork Ravi Kumar Moni Naor D. Sivakumar Presented by: Wanying](https://reader035.fdocuments.us/reader035/viewer/2022062606/5fe1994a35ee9307b14f0a76/html5/thumbnails/17.jpg)
17
Preliminaries
Ordered listGiven a universe U, an ordered list τ with respect to U is anordering(aka ranking) of a subset S ⊆ U, i.e.
τ=[χ1 ≥ χ2 ≥ …≥ χd ]
Full listτ contains all the elements in U
Partial list|τ| < |U|
![Page 18: Rank Aggregation Methods for the Webklarson/teaching/F08-886/talks/WLuo.pdf · Rank Aggregation Methods for the Web Cynthia Dwork Ravi Kumar Moni Naor D. Sivakumar Presented by: Wanying](https://reader035.fdocuments.us/reader035/viewer/2022062606/5fe1994a35ee9307b14f0a76/html5/thumbnails/18.jpg)
18
Preliminaries
Ordered listGiven a universe U, an ordered list τ with respect to U is anordering(aka ranking) of a subset S ⊆ U, i.e.
τ=[χ1 ≥ χ2≥ …≥ χd ]
Full listτ contains all the elements in U
Partial list|τ| < |U|
![Page 19: Rank Aggregation Methods for the Webklarson/teaching/F08-886/talks/WLuo.pdf · Rank Aggregation Methods for the Web Cynthia Dwork Ravi Kumar Moni Naor D. Sivakumar Presented by: Wanying](https://reader035.fdocuments.us/reader035/viewer/2022062606/5fe1994a35ee9307b14f0a76/html5/thumbnails/19.jpg)
19
Preliminaries
Ordered listGiven a universe U, an ordered list τ with respect to U is anordering(aka ranking) of a subset S ⊆ U, i.e.
τ=[χ1 ≥ χ2≥ …≥ χd ]
Full listτ contains all the elements in U
Partial list|τ| < |U|
![Page 20: Rank Aggregation Methods for the Webklarson/teaching/F08-886/talks/WLuo.pdf · Rank Aggregation Methods for the Web Cynthia Dwork Ravi Kumar Moni Naor D. Sivakumar Presented by: Wanying](https://reader035.fdocuments.us/reader035/viewer/2022062606/5fe1994a35ee9307b14f0a76/html5/thumbnails/20.jpg)
20
Preliminaries
Rank aggregation approachGoal: minimize the total disagreement between several rankings Spearman footrule distance
Given two full lists σ and τ, F(σ,τ)=Σi=1|σ(i)-τ(i)|Kendall tau distance
Given two full lists σ and τ, K(σ,τ)=|{(i,j) | i<j, σ(i)<σ(j) but τ(i)>τ(j) }|These two measurements can be generalized to several listsCan also be generalized to partial lists
![Page 21: Rank Aggregation Methods for the Webklarson/teaching/F08-886/talks/WLuo.pdf · Rank Aggregation Methods for the Web Cynthia Dwork Ravi Kumar Moni Naor D. Sivakumar Presented by: Wanying](https://reader035.fdocuments.us/reader035/viewer/2022062606/5fe1994a35ee9307b14f0a76/html5/thumbnails/21.jpg)
21
Preliminaries
Rank aggregation approachGoal: minimize the total disagreement between several rankings Spearman footrule distance
Given two full lists σ and τ, F(σ,τ)=Σi=1|σ(i)-τ(i)|Kendall tau distance
Given two full lists σ and τ, K(σ,τ)=|{(i,j) | i<j, σ(i)<σ(j) but τ(i)>τ(j) }|These two measurements can be generalized to several listsCan also be generalized to partial lists
![Page 22: Rank Aggregation Methods for the Webklarson/teaching/F08-886/talks/WLuo.pdf · Rank Aggregation Methods for the Web Cynthia Dwork Ravi Kumar Moni Naor D. Sivakumar Presented by: Wanying](https://reader035.fdocuments.us/reader035/viewer/2022062606/5fe1994a35ee9307b14f0a76/html5/thumbnails/22.jpg)
22
Preliminaries
Rank aggregation approachGoal: minimize the total disagreement between several rankings Spearman footrule distance
Given two full lists σ and τ, F(σ,τ)=Σi=1|σ(i)-τ(i)|Kendall tau distance
Given two full lists σ and τ, K(σ,τ)=|{(i,j) | i<j, σ(i)<σ(j) but τ(i)>τ(j) }|These two measurements can be generalized to several listsCan also be generalized to partial lists
![Page 23: Rank Aggregation Methods for the Webklarson/teaching/F08-886/talks/WLuo.pdf · Rank Aggregation Methods for the Web Cynthia Dwork Ravi Kumar Moni Naor D. Sivakumar Presented by: Wanying](https://reader035.fdocuments.us/reader035/viewer/2022062606/5fe1994a35ee9307b14f0a76/html5/thumbnails/23.jpg)
23
Preliminaries
Rank aggregation approachGoal: minimize the total disagreement between several rankings Spearman footrule distance
Given two full lists σ and τ, F(σ,τ)=Σi=1|σ(i)-τ(i)|Kendall tau distance
Given two full lists σ and τ, K(σ,τ)=|{(i,j) | i<j, σ(i)<σ(j) but τ(i)>τ(j) }|These two measurements can be generalized to several listsCan also be generalized to partial lists
![Page 24: Rank Aggregation Methods for the Webklarson/teaching/F08-886/talks/WLuo.pdf · Rank Aggregation Methods for the Web Cynthia Dwork Ravi Kumar Moni Naor D. Sivakumar Presented by: Wanying](https://reader035.fdocuments.us/reader035/viewer/2022062606/5fe1994a35ee9307b14f0a76/html5/thumbnails/24.jpg)
24
Preliminaries
Rank aggregation approachGoal: minimize the total disagreement between several rankings Spearman footrule distance
Given two full lists σ and τ, F(σ,τ)=Σi=1|σ(i)-τ(i)|Kendall tau distance
Given two full lists σ and τ, K(σ,τ)=|{(i,j) | i<j, σ(i)<σ(j) but τ(i)>τ(j) }|These two measurements can be generalized to several listsCan also be generalized to partial lists
![Page 25: Rank Aggregation Methods for the Webklarson/teaching/F08-886/talks/WLuo.pdf · Rank Aggregation Methods for the Web Cynthia Dwork Ravi Kumar Moni Naor D. Sivakumar Presented by: Wanying](https://reader035.fdocuments.us/reader035/viewer/2022062606/5fe1994a35ee9307b14f0a76/html5/thumbnails/25.jpg)
25
Outline
What is rank aggregation problemMotivationChallengesPreliminariesFirst result: spam resistance in meta-searchSecond result: Markov chain methodsApplicationsExperimentsConclusion
![Page 26: Rank Aggregation Methods for the Webklarson/teaching/F08-886/talks/WLuo.pdf · Rank Aggregation Methods for the Web Cynthia Dwork Ravi Kumar Moni Naor D. Sivakumar Presented by: Wanying](https://reader035.fdocuments.us/reader035/viewer/2022062606/5fe1994a35ee9307b14f0a76/html5/thumbnails/26.jpg)
26
First result: spam resistance in meta-search
Extended Condorcet Criterion (ECC) If there is a partition (C, C’) of S such that for any x∈Cand y∈C’ the majority prefers x to y, then x must be ranked above y
ECC can be used to fight spam in meta-searchHow to achieve ECC efficiently
Local Kemenization method
![Page 27: Rank Aggregation Methods for the Webklarson/teaching/F08-886/talks/WLuo.pdf · Rank Aggregation Methods for the Web Cynthia Dwork Ravi Kumar Moni Naor D. Sivakumar Presented by: Wanying](https://reader035.fdocuments.us/reader035/viewer/2022062606/5fe1994a35ee9307b14f0a76/html5/thumbnails/27.jpg)
27
First result: spam resistance in meta-search
Extended Condorcet Criterion (ECC) If there is a partition (C, C’) of S such that for any x∈Cand y∈C’ the majority prefers x to y, then x must be ranked above y
ECC can be used to fight spam in meta-searchHow to achieve ECC efficiently
Local Kemenization method
![Page 28: Rank Aggregation Methods for the Webklarson/teaching/F08-886/talks/WLuo.pdf · Rank Aggregation Methods for the Web Cynthia Dwork Ravi Kumar Moni Naor D. Sivakumar Presented by: Wanying](https://reader035.fdocuments.us/reader035/viewer/2022062606/5fe1994a35ee9307b14f0a76/html5/thumbnails/28.jpg)
28
First result: spam resistance in meta-search
Extended Condorcet Criterion (ECC) If there is a partition (C, C’) of S such that for any x∈Cand y∈C’ the majority prefers x to y, then x must be ranked above y
ECC can be used to fight spam in meta-searchHow to achieve ECC efficiently
Local Kemenization method
![Page 29: Rank Aggregation Methods for the Webklarson/teaching/F08-886/talks/WLuo.pdf · Rank Aggregation Methods for the Web Cynthia Dwork Ravi Kumar Moni Naor D. Sivakumar Presented by: Wanying](https://reader035.fdocuments.us/reader035/viewer/2022062606/5fe1994a35ee9307b14f0a76/html5/thumbnails/29.jpg)
29
First result: spam resistance in meta-search
An example to illustrate Local Kemenization …
![Page 30: Rank Aggregation Methods for the Webklarson/teaching/F08-886/talks/WLuo.pdf · Rank Aggregation Methods for the Web Cynthia Dwork Ravi Kumar Moni Naor D. Sivakumar Presented by: Wanying](https://reader035.fdocuments.us/reader035/viewer/2022062606/5fe1994a35ee9307b14f0a76/html5/thumbnails/30.jpg)
30
First result: spam resistance in meta-search
B
A
C
E
D
A
B
D
E
C
B
C
A
D
E
A
B
D
C
E
![Page 31: Rank Aggregation Methods for the Webklarson/teaching/F08-886/talks/WLuo.pdf · Rank Aggregation Methods for the Web Cynthia Dwork Ravi Kumar Moni Naor D. Sivakumar Presented by: Wanying](https://reader035.fdocuments.us/reader035/viewer/2022062606/5fe1994a35ee9307b14f0a76/html5/thumbnails/31.jpg)
31
First result: spam resistance in meta-search
B
A
C
E
D
A
B
D
E
C
B
C
A
D
E
A
B
D
C
E
A
B
![Page 32: Rank Aggregation Methods for the Webklarson/teaching/F08-886/talks/WLuo.pdf · Rank Aggregation Methods for the Web Cynthia Dwork Ravi Kumar Moni Naor D. Sivakumar Presented by: Wanying](https://reader035.fdocuments.us/reader035/viewer/2022062606/5fe1994a35ee9307b14f0a76/html5/thumbnails/32.jpg)
32
First result: spam resistance in meta-search
B
A
C
E
D
A
B
D
E
C
B
C
A
D
E
A
B
D
C
E
A
B
B>A A>B B>A
![Page 33: Rank Aggregation Methods for the Webklarson/teaching/F08-886/talks/WLuo.pdf · Rank Aggregation Methods for the Web Cynthia Dwork Ravi Kumar Moni Naor D. Sivakumar Presented by: Wanying](https://reader035.fdocuments.us/reader035/viewer/2022062606/5fe1994a35ee9307b14f0a76/html5/thumbnails/33.jpg)
33
First result: spam resistance in meta-search
B
A
C
E
D
A
B
D
E
C
B
C
A
D
E
A
B
D
C
E
B
A
![Page 34: Rank Aggregation Methods for the Webklarson/teaching/F08-886/talks/WLuo.pdf · Rank Aggregation Methods for the Web Cynthia Dwork Ravi Kumar Moni Naor D. Sivakumar Presented by: Wanying](https://reader035.fdocuments.us/reader035/viewer/2022062606/5fe1994a35ee9307b14f0a76/html5/thumbnails/34.jpg)
34
First result: spam resistance in meta-search
B
A
C
E
D
A
B
D
E
C
B
C
A
D
E
A
B
D
C
E
B
A
D
![Page 35: Rank Aggregation Methods for the Webklarson/teaching/F08-886/talks/WLuo.pdf · Rank Aggregation Methods for the Web Cynthia Dwork Ravi Kumar Moni Naor D. Sivakumar Presented by: Wanying](https://reader035.fdocuments.us/reader035/viewer/2022062606/5fe1994a35ee9307b14f0a76/html5/thumbnails/35.jpg)
35
First result: spam resistance in meta-search
B
A
C
E
D
A
B
D
E
C
B
C
A
D
E
A
B
D
C
E
B
A
D
A>D A>D A>D
![Page 36: Rank Aggregation Methods for the Webklarson/teaching/F08-886/talks/WLuo.pdf · Rank Aggregation Methods for the Web Cynthia Dwork Ravi Kumar Moni Naor D. Sivakumar Presented by: Wanying](https://reader035.fdocuments.us/reader035/viewer/2022062606/5fe1994a35ee9307b14f0a76/html5/thumbnails/36.jpg)
36
First result: spam resistance in meta-search
B
A
C
E
D
A
B
D
E
C
B
C
A
D
E
A
B
D
C
E
B
A
D
![Page 37: Rank Aggregation Methods for the Webklarson/teaching/F08-886/talks/WLuo.pdf · Rank Aggregation Methods for the Web Cynthia Dwork Ravi Kumar Moni Naor D. Sivakumar Presented by: Wanying](https://reader035.fdocuments.us/reader035/viewer/2022062606/5fe1994a35ee9307b14f0a76/html5/thumbnails/37.jpg)
37
First result: spam resistance in meta-search
B
A
C
E
D
A
B
D
E
C
B
C
A
D
E
A
B
D
C
E
B
A
D
C
![Page 38: Rank Aggregation Methods for the Webklarson/teaching/F08-886/talks/WLuo.pdf · Rank Aggregation Methods for the Web Cynthia Dwork Ravi Kumar Moni Naor D. Sivakumar Presented by: Wanying](https://reader035.fdocuments.us/reader035/viewer/2022062606/5fe1994a35ee9307b14f0a76/html5/thumbnails/38.jpg)
38
First result: spam resistance in meta-search
B
A
C
E
D
A
B
D
E
C
B
C
A
D
E
A
B
D
C
E
B
A
D
C is preferred to D in two lists
D is preferred to C in one list
C
C>D D>C C>D
![Page 39: Rank Aggregation Methods for the Webklarson/teaching/F08-886/talks/WLuo.pdf · Rank Aggregation Methods for the Web Cynthia Dwork Ravi Kumar Moni Naor D. Sivakumar Presented by: Wanying](https://reader035.fdocuments.us/reader035/viewer/2022062606/5fe1994a35ee9307b14f0a76/html5/thumbnails/39.jpg)
39
First result: spam resistance in meta-search
B
A
C
E
D
A
B
D
E
C
B
C
A
D
E
A
B
D
C
E
B
A
C
D
![Page 40: Rank Aggregation Methods for the Webklarson/teaching/F08-886/talks/WLuo.pdf · Rank Aggregation Methods for the Web Cynthia Dwork Ravi Kumar Moni Naor D. Sivakumar Presented by: Wanying](https://reader035.fdocuments.us/reader035/viewer/2022062606/5fe1994a35ee9307b14f0a76/html5/thumbnails/40.jpg)
40
First result: spam resistance in meta-search
B
A
C
E
D
A
B
D
E
C
B
C
A
D
E
A
B
D
C
E
B
A
C
D
A>C A>C C>A
![Page 41: Rank Aggregation Methods for the Webklarson/teaching/F08-886/talks/WLuo.pdf · Rank Aggregation Methods for the Web Cynthia Dwork Ravi Kumar Moni Naor D. Sivakumar Presented by: Wanying](https://reader035.fdocuments.us/reader035/viewer/2022062606/5fe1994a35ee9307b14f0a76/html5/thumbnails/41.jpg)
41
First result: spam resistance in meta-search
B
A
C
E
D
A
B
D
E
C
B
C
A
D
E
A
B
D
C
E
B
A
C
D
![Page 42: Rank Aggregation Methods for the Webklarson/teaching/F08-886/talks/WLuo.pdf · Rank Aggregation Methods for the Web Cynthia Dwork Ravi Kumar Moni Naor D. Sivakumar Presented by: Wanying](https://reader035.fdocuments.us/reader035/viewer/2022062606/5fe1994a35ee9307b14f0a76/html5/thumbnails/42.jpg)
42
Outline
What is rank aggregation problemMotivationChallengesPreliminariesFirst result: spam resistance in meta-searchSecond result: Markov chain methodsApplicationsExperimentsConclusion
![Page 43: Rank Aggregation Methods for the Webklarson/teaching/F08-886/talks/WLuo.pdf · Rank Aggregation Methods for the Web Cynthia Dwork Ravi Kumar Moni Naor D. Sivakumar Presented by: Wanying](https://reader035.fdocuments.us/reader035/viewer/2022062606/5fe1994a35ee9307b14f0a76/html5/thumbnails/43.jpg)
43
Second result: Markov chain methods
Markov chain A set of states S={1,2,...,n}An n x n matrix MBegins with an initial state xAt each step the system moves from state i to state j with probability Mij
![Page 44: Rank Aggregation Methods for the Webklarson/teaching/F08-886/talks/WLuo.pdf · Rank Aggregation Methods for the Web Cynthia Dwork Ravi Kumar Moni Naor D. Sivakumar Presented by: Wanying](https://reader035.fdocuments.us/reader035/viewer/2022062606/5fe1994a35ee9307b14f0a76/html5/thumbnails/44.jpg)
44
Second result: Markov chain methods
Under some nice condition, system eventually reaches a fixed point irrespective of the initial state x
![Page 45: Rank Aggregation Methods for the Webklarson/teaching/F08-886/talks/WLuo.pdf · Rank Aggregation Methods for the Web Cynthia Dwork Ravi Kumar Moni Naor D. Sivakumar Presented by: Wanying](https://reader035.fdocuments.us/reader035/viewer/2022062606/5fe1994a35ee9307b14f0a76/html5/thumbnails/45.jpg)
45
Second result: Markov chain methods
B
A
C
A
B
C
A
C
B
Original rankings
![Page 46: Rank Aggregation Methods for the Webklarson/teaching/F08-886/talks/WLuo.pdf · Rank Aggregation Methods for the Web Cynthia Dwork Ravi Kumar Moni Naor D. Sivakumar Presented by: Wanying](https://reader035.fdocuments.us/reader035/viewer/2022062606/5fe1994a35ee9307b14f0a76/html5/thumbnails/46.jpg)
46
Second result: Markov chain methods
B
A
C
A
B
C
A
C
B
0.1 0.1 0.3 0.3
0.2
0.20.50.7
0.6
A
BC
Original rankings
![Page 47: Rank Aggregation Methods for the Webklarson/teaching/F08-886/talks/WLuo.pdf · Rank Aggregation Methods for the Web Cynthia Dwork Ravi Kumar Moni Naor D. Sivakumar Presented by: Wanying](https://reader035.fdocuments.us/reader035/viewer/2022062606/5fe1994a35ee9307b14f0a76/html5/thumbnails/47.jpg)
47
Second result: Markov chain methods
B
A
C
A
B
C
A
C
B
0.1 0.1 0.3 0.3
0.2
0.20.50.7
0.6
A
BC
A
B
C
Original rankings Aggregated ranking
![Page 48: Rank Aggregation Methods for the Webklarson/teaching/F08-886/talks/WLuo.pdf · Rank Aggregation Methods for the Web Cynthia Dwork Ravi Kumar Moni Naor D. Sivakumar Presented by: Wanying](https://reader035.fdocuments.us/reader035/viewer/2022062606/5fe1994a35ee9307b14f0a76/html5/thumbnails/48.jpg)
48
Second result: Markov chain methods
Assume the current state is page PMC1: The next state is chosen uniformly from the multiset of all pages that were ranked higher than or equal to P by some search engine that ranked PPlease refer to the paper for the rest …
MC2MC3
MC4
![Page 49: Rank Aggregation Methods for the Webklarson/teaching/F08-886/talks/WLuo.pdf · Rank Aggregation Methods for the Web Cynthia Dwork Ravi Kumar Moni Naor D. Sivakumar Presented by: Wanying](https://reader035.fdocuments.us/reader035/viewer/2022062606/5fe1994a35ee9307b14f0a76/html5/thumbnails/49.jpg)
49
Second result: Markov chain methods
Assume the current state is page PMC1: The next state is chosen uniformly from the multiset of all pages that were ranked higher than or equal to P by some search engine that ranked PPlease refer to the paper for the rest …
MC2MC3MC4
![Page 50: Rank Aggregation Methods for the Webklarson/teaching/F08-886/talks/WLuo.pdf · Rank Aggregation Methods for the Web Cynthia Dwork Ravi Kumar Moni Naor D. Sivakumar Presented by: Wanying](https://reader035.fdocuments.us/reader035/viewer/2022062606/5fe1994a35ee9307b14f0a76/html5/thumbnails/50.jpg)
50
Outline
What is rank aggregation problemMotivationChallengesPreliminariesFirst result: spam resistance in meta-searchSecond result: Markov chain methodsApplicationsExperimentsConclusion
![Page 51: Rank Aggregation Methods for the Webklarson/teaching/F08-886/talks/WLuo.pdf · Rank Aggregation Methods for the Web Cynthia Dwork Ravi Kumar Moni Naor D. Sivakumar Presented by: Wanying](https://reader035.fdocuments.us/reader035/viewer/2022062606/5fe1994a35ee9307b14f0a76/html5/thumbnails/51.jpg)
51
Applications
Meta-searchSpam reductionMulti-criteria searchSearch engine comparison
![Page 52: Rank Aggregation Methods for the Webklarson/teaching/F08-886/talks/WLuo.pdf · Rank Aggregation Methods for the Web Cynthia Dwork Ravi Kumar Moni Naor D. Sivakumar Presented by: Wanying](https://reader035.fdocuments.us/reader035/viewer/2022062606/5fe1994a35ee9307b14f0a76/html5/thumbnails/52.jpg)
52
Applications
Meta-searchSpam reductionMulti-criteria searchSearch engine comparison
![Page 53: Rank Aggregation Methods for the Webklarson/teaching/F08-886/talks/WLuo.pdf · Rank Aggregation Methods for the Web Cynthia Dwork Ravi Kumar Moni Naor D. Sivakumar Presented by: Wanying](https://reader035.fdocuments.us/reader035/viewer/2022062606/5fe1994a35ee9307b14f0a76/html5/thumbnails/53.jpg)
53
Applications
Meta-searchSpam reductionMulti-criteria searchSearch engine comparison
![Page 54: Rank Aggregation Methods for the Webklarson/teaching/F08-886/talks/WLuo.pdf · Rank Aggregation Methods for the Web Cynthia Dwork Ravi Kumar Moni Naor D. Sivakumar Presented by: Wanying](https://reader035.fdocuments.us/reader035/viewer/2022062606/5fe1994a35ee9307b14f0a76/html5/thumbnails/54.jpg)
54
Applications
Meta-searchSpam reductionMulti-criteria searchSearch engine comparison
![Page 55: Rank Aggregation Methods for the Webklarson/teaching/F08-886/talks/WLuo.pdf · Rank Aggregation Methods for the Web Cynthia Dwork Ravi Kumar Moni Naor D. Sivakumar Presented by: Wanying](https://reader035.fdocuments.us/reader035/viewer/2022062606/5fe1994a35ee9307b14f0a76/html5/thumbnails/55.jpg)
55
Outline
What is rank aggregation problemMotivationChallengesPreliminariesFirst result: spam resistance in meta-searchSecond result: Markov chain methodsApplicationsExperimentsConclusion
![Page 56: Rank Aggregation Methods for the Webklarson/teaching/F08-886/talks/WLuo.pdf · Rank Aggregation Methods for the Web Cynthia Dwork Ravi Kumar Moni Naor D. Sivakumar Presented by: Wanying](https://reader035.fdocuments.us/reader035/viewer/2022062606/5fe1994a35ee9307b14f0a76/html5/thumbnails/56.jpg)
56
Experiments
Experiments were conducted by using the following search engines: Altavista(AV), Alltheweb(AW), Excite(EX), Google(GG), Hotbot(HB), Lycos(LY) and Northernlight(NL)Experiment on meta-search using several keywords: “affirmative action”, alcoholism, sushi, ...
![Page 57: Rank Aggregation Methods for the Webklarson/teaching/F08-886/talks/WLuo.pdf · Rank Aggregation Methods for the Web Cynthia Dwork Ravi Kumar Moni Naor D. Sivakumar Presented by: Wanying](https://reader035.fdocuments.us/reader035/viewer/2022062606/5fe1994a35ee9307b14f0a76/html5/thumbnails/57.jpg)
57
Experiments
Experiments were conducted by using the following search engines: Altavista(AV), Alltheweb(AW), Excite(EX), Google(GG), Hotbot(HB), Lycos(LY) and Northernlight(NL)Experiment on meta-search using several keywords: “affirmative action”, alcoholism, sushi, ...
![Page 58: Rank Aggregation Methods for the Webklarson/teaching/F08-886/talks/WLuo.pdf · Rank Aggregation Methods for the Web Cynthia Dwork Ravi Kumar Moni Naor D. Sivakumar Presented by: Wanying](https://reader035.fdocuments.us/reader035/viewer/2022062606/5fe1994a35ee9307b14f0a76/html5/thumbnails/58.jpg)
58
Experiments
![Page 59: Rank Aggregation Methods for the Webklarson/teaching/F08-886/talks/WLuo.pdf · Rank Aggregation Methods for the Web Cynthia Dwork Ravi Kumar Moni Naor D. Sivakumar Presented by: Wanying](https://reader035.fdocuments.us/reader035/viewer/2022062606/5fe1994a35ee9307b14f0a76/html5/thumbnails/59.jpg)
59
Experiments
SFO and MC4 outperform the other 4 algorithms
MC4 performs better than SFO most of the time
![Page 60: Rank Aggregation Methods for the Webklarson/teaching/F08-886/talks/WLuo.pdf · Rank Aggregation Methods for the Web Cynthia Dwork Ravi Kumar Moni Naor D. Sivakumar Presented by: Wanying](https://reader035.fdocuments.us/reader035/viewer/2022062606/5fe1994a35ee9307b14f0a76/html5/thumbnails/60.jpg)
60
Experiments
Experiment on spam reduction using queries: Feng shui, organic vegetable, gardening
![Page 61: Rank Aggregation Methods for the Webklarson/teaching/F08-886/talks/WLuo.pdf · Rank Aggregation Methods for the Web Cynthia Dwork Ravi Kumar Moni Naor D. Sivakumar Presented by: Wanying](https://reader035.fdocuments.us/reader035/viewer/2022062606/5fe1994a35ee9307b14f0a76/html5/thumbnails/61.jpg)
61
Experiments
![Page 62: Rank Aggregation Methods for the Webklarson/teaching/F08-886/talks/WLuo.pdf · Rank Aggregation Methods for the Web Cynthia Dwork Ravi Kumar Moni Naor D. Sivakumar Presented by: Wanying](https://reader035.fdocuments.us/reader035/viewer/2022062606/5fe1994a35ee9307b14f0a76/html5/thumbnails/62.jpg)
62
Experiments
![Page 63: Rank Aggregation Methods for the Webklarson/teaching/F08-886/talks/WLuo.pdf · Rank Aggregation Methods for the Web Cynthia Dwork Ravi Kumar Moni Naor D. Sivakumar Presented by: Wanying](https://reader035.fdocuments.us/reader035/viewer/2022062606/5fe1994a35ee9307b14f0a76/html5/thumbnails/63.jpg)
63
Experiments
![Page 64: Rank Aggregation Methods for the Webklarson/teaching/F08-886/talks/WLuo.pdf · Rank Aggregation Methods for the Web Cynthia Dwork Ravi Kumar Moni Naor D. Sivakumar Presented by: Wanying](https://reader035.fdocuments.us/reader035/viewer/2022062606/5fe1994a35ee9307b14f0a76/html5/thumbnails/64.jpg)
64
Experiments
![Page 65: Rank Aggregation Methods for the Webklarson/teaching/F08-886/talks/WLuo.pdf · Rank Aggregation Methods for the Web Cynthia Dwork Ravi Kumar Moni Naor D. Sivakumar Presented by: Wanying](https://reader035.fdocuments.us/reader035/viewer/2022062606/5fe1994a35ee9307b14f0a76/html5/thumbnails/65.jpg)
65
Experiments
Local Kemenization works!
![Page 66: Rank Aggregation Methods for the Webklarson/teaching/F08-886/talks/WLuo.pdf · Rank Aggregation Methods for the Web Cynthia Dwork Ravi Kumar Moni Naor D. Sivakumar Presented by: Wanying](https://reader035.fdocuments.us/reader035/viewer/2022062606/5fe1994a35ee9307b14f0a76/html5/thumbnails/66.jpg)
66
Outline
What is rank aggregation problemMotivationChallengesPreliminariesFirst result: spam resistance in meta-searchSecond result: Markov chain methodsApplicationsExperimentsConclusion
![Page 67: Rank Aggregation Methods for the Webklarson/teaching/F08-886/talks/WLuo.pdf · Rank Aggregation Methods for the Web Cynthia Dwork Ravi Kumar Moni Naor D. Sivakumar Presented by: Wanying](https://reader035.fdocuments.us/reader035/viewer/2022062606/5fe1994a35ee9307b14f0a76/html5/thumbnails/67.jpg)
67
Conclusion
Proposed several rank aggregation techniques using Markov chainEstablished the value of Extended CondorcetCriterion (ECC)
Spam resistanceFuture work
Obtain a qualitative understanding of why Markov chain methods perform well
![Page 68: Rank Aggregation Methods for the Webklarson/teaching/F08-886/talks/WLuo.pdf · Rank Aggregation Methods for the Web Cynthia Dwork Ravi Kumar Moni Naor D. Sivakumar Presented by: Wanying](https://reader035.fdocuments.us/reader035/viewer/2022062606/5fe1994a35ee9307b14f0a76/html5/thumbnails/68.jpg)
68
Conclusion
Proposed several rank aggregation techniques using Markov chainEstablished the value of Extended CondorcetCriterion (ECC)
Spam resistanceFuture work
Obtain a qualitative understanding of why Markov chain methods perform well
![Page 69: Rank Aggregation Methods for the Webklarson/teaching/F08-886/talks/WLuo.pdf · Rank Aggregation Methods for the Web Cynthia Dwork Ravi Kumar Moni Naor D. Sivakumar Presented by: Wanying](https://reader035.fdocuments.us/reader035/viewer/2022062606/5fe1994a35ee9307b14f0a76/html5/thumbnails/69.jpg)
69
Conclusion
Proposed several rank aggregation techniques using Markov chainEstablished the value of Extended CondorcetCriterion (ECC)
Spam resistanceFuture work
Obtain a qualitative understanding of why Markov chain methods perform well
![Page 70: Rank Aggregation Methods for the Webklarson/teaching/F08-886/talks/WLuo.pdf · Rank Aggregation Methods for the Web Cynthia Dwork Ravi Kumar Moni Naor D. Sivakumar Presented by: Wanying](https://reader035.fdocuments.us/reader035/viewer/2022062606/5fe1994a35ee9307b14f0a76/html5/thumbnails/70.jpg)
70
Questions???