Experimental Projects on Web Algorithms - Yury Lifshits
Transcript of Experimental Projects on Web Algorithms - Yury Lifshits
![Page 1: Experimental Projects on Web Algorithms - Yury Lifshits](https://reader031.fdocuments.us/reader031/viewer/2022020705/61fb89dd2e268c58cd5f5cbd/html5/thumbnails/1.jpg)
Experimental Projectson Web Algorithms
Yury Lifshitshttp://yury.name
CalTech, Fall’07Invited lecture at CS141a
1 / 19
![Page 2: Experimental Projects on Web Algorithms - Yury Lifshits](https://reader031.fdocuments.us/reader031/viewer/2022020705/61fb89dd2e268c58cd5f5cbd/html5/thumbnails/2.jpg)
Invitation to CS101.2
New Caltech courseAlgorithmic Problems Around the Web:
http://yury.name/algoweb.html
MW 11:00-11:55, Jorgensen 287
Lectures: algorithms for nearest neighbor search
Projects: adjusting above algorithms to webtechnologies
Datasets: friendship graph, users-ads graph
2 / 19
![Page 3: Experimental Projects on Web Algorithms - Yury Lifshits](https://reader031.fdocuments.us/reader031/viewer/2022020705/61fb89dd2e268c58cd5f5cbd/html5/thumbnails/3.jpg)
Course Philosophy
Challenges inWeb Technologies
Recs, Ads,Social Networks
Existing Theory:
Algorithms for NearestNeighbor Search
New Math Problems
New Algorithms
New Experiments
3 / 19
![Page 4: Experimental Projects on Web Algorithms - Yury Lifshits](https://reader031.fdocuments.us/reader031/viewer/2022020705/61fb89dd2e268c58cd5f5cbd/html5/thumbnails/4.jpg)
Course Philosophy
Challenges inWeb Technologies
Recs, Ads,Social Networks
Existing Theory:
Algorithms for NearestNeighbor Search
New Math Problems
New Algorithms
New Experiments
3 / 19
![Page 5: Experimental Projects on Web Algorithms - Yury Lifshits](https://reader031.fdocuments.us/reader031/viewer/2022020705/61fb89dd2e268c58cd5f5cbd/html5/thumbnails/5.jpg)
Outline
1 Challenges in Web Technologies
2 Existing Theory: Nearest Neighbors
3 Topics for Experimental Projects
4 / 19
![Page 6: Experimental Projects on Web Algorithms - Yury Lifshits](https://reader031.fdocuments.us/reader031/viewer/2022020705/61fb89dd2e268c58cd5f5cbd/html5/thumbnails/6.jpg)
Outline
1 Challenges in Web Technologies
2 Existing Theory: Nearest Neighbors
3 Topics for Experimental Projects
4 / 19
![Page 7: Experimental Projects on Web Algorithms - Yury Lifshits](https://reader031.fdocuments.us/reader031/viewer/2022020705/61fb89dd2e268c58cd5f5cbd/html5/thumbnails/7.jpg)
Outline
1 Challenges in Web Technologies
2 Existing Theory: Nearest Neighbors
3 Topics for Experimental Projects
4 / 19
![Page 8: Experimental Projects on Web Algorithms - Yury Lifshits](https://reader031.fdocuments.us/reader031/viewer/2022020705/61fb89dd2e268c58cd5f5cbd/html5/thumbnails/8.jpg)
Part IChallenges in Web Technologies
5 / 19
![Page 9: Experimental Projects on Web Algorithms - Yury Lifshits](https://reader031.fdocuments.us/reader031/viewer/2022020705/61fb89dd2e268c58cd5f5cbd/html5/thumbnails/9.jpg)
Recommendation Systems
Approaches:Content-basedCollaborative filtering
6 / 19
![Page 10: Experimental Projects on Web Algorithms - Yury Lifshits](https://reader031.fdocuments.us/reader031/viewer/2022020705/61fb89dd2e268c58cd5f5cbd/html5/thumbnails/10.jpg)
Behavioral Targeting
FOR SALEwww.home.org
FOR SALEwww.home.org
FOR SALEwww.home.org
Ad targeting:Ancient: broadcastingCurrent: contextualFuture: behavioral
7 / 19
![Page 11: Experimental Projects on Web Algorithms - Yury Lifshits](https://reader031.fdocuments.us/reader031/viewer/2022020705/61fb89dd2e268c58cd5f5cbd/html5/thumbnails/11.jpg)
Personalized News Aggregation
Factors to take into account:FriendshipContentFeedback (previous ratings)Popularity (votes, comments, hyperlinks)
8 / 19
![Page 12: Experimental Projects on Web Algorithms - Yury Lifshits](https://reader031.fdocuments.us/reader031/viewer/2022020705/61fb89dd2e268c58cd5f5cbd/html5/thumbnails/12.jpg)
Social Networks Analysis
Social network:NodesEdges
Examples of relations: financial exchange, friends,dislike, conflict, trade, web links, sexual relations,disease transmission, airline routes, etc.
Our focusCommunity discoveryBurst detection
9 / 19
![Page 13: Experimental Projects on Web Algorithms - Yury Lifshits](https://reader031.fdocuments.us/reader031/viewer/2022020705/61fb89dd2e268c58cd5f5cbd/html5/thumbnails/13.jpg)
Social Networks Analysis
Social network:NodesEdges
Examples of relations: financial exchange, friends,dislike, conflict, trade, web links, sexual relations,disease transmission, airline routes, etc.
Our focusCommunity discoveryBurst detection
9 / 19
![Page 14: Experimental Projects on Web Algorithms - Yury Lifshits](https://reader031.fdocuments.us/reader031/viewer/2022020705/61fb89dd2e268c58cd5f5cbd/html5/thumbnails/14.jpg)
Part II Theory of
Nearest Neighbors
10 / 19
![Page 15: Experimental Projects on Web Algorithms - Yury Lifshits](https://reader031.fdocuments.us/reader031/viewer/2022020705/61fb89dd2e268c58cd5f5cbd/html5/thumbnails/15.jpg)
Nearest Neighbors Informally
To preprocess a database of n objectsso that given a query object,one can effectively determine
its nearest neighbors in database
11 / 19
![Page 16: Experimental Projects on Web Algorithms - Yury Lifshits](https://reader031.fdocuments.us/reader031/viewer/2022020705/61fb89dd2e268c58cd5f5cbd/html5/thumbnails/16.jpg)
More Formally
Search space: object domain U, similarity function σ
Input: database S = {p1, . . . , pn} ⊆ UQuery: q ∈ UTask: find argmaxpi
σ(pi , q)
p1
p2
p3
p4p5
p6
q
12 / 19
![Page 17: Experimental Projects on Web Algorithms - Yury Lifshits](https://reader031.fdocuments.us/reader031/viewer/2022020705/61fb89dd2e268c58cd5f5cbd/html5/thumbnails/17.jpg)
More Formally
Search space: object domain U, similarity function σ
Input: database S = {p1, . . . , pn} ⊆ UQuery: q ∈ UTask: find argmaxpi
σ(pi , q)
p1
p2
p3
p4p5
p6
q
12 / 19
![Page 18: Experimental Projects on Web Algorithms - Yury Lifshits](https://reader031.fdocuments.us/reader031/viewer/2022020705/61fb89dd2e268c58cd5f5cbd/html5/thumbnails/18.jpg)
More Formally
Search space: object domain U, similarity function σ
Input: database S = {p1, . . . , pn} ⊆ UQuery: q ∈ UTask: find argmaxpi
σ(pi , q)
p1
p2
p3
p4p5
p6
q
12 / 19
![Page 19: Experimental Projects on Web Algorithms - Yury Lifshits](https://reader031.fdocuments.us/reader031/viewer/2022020705/61fb89dd2e268c58cd5f5cbd/html5/thumbnails/19.jpg)
Some Solutions for NN ProblemSphere Rectangle Tree Orchard’s Algorithm LAESA
k-d-B tree Geometric near-neighbor access treeExcluded middle vantage point forest mvp-tree Fixed-height
fixed-queries tree AESA Vantage-pointtree R∗-tree Burkhard-Keller tree BBD tree
Navigating Nets Voronoi tree Balanced aspect ratio tree Metric tree
vps -tree M-tree Locality-Sensitive HashingSS-tree R-tree Spatial approximation tree Multi-vantage
point tree Bisector tree mb-tree
Generalized hyperplane treeHybrid tree Slim tree Spill Tree Fixed queries tree X-tree k-dtree Balltree Quadtree Octree Post-office tree
13 / 19
![Page 20: Experimental Projects on Web Algorithms - Yury Lifshits](https://reader031.fdocuments.us/reader031/viewer/2022020705/61fb89dd2e268c58cd5f5cbd/html5/thumbnails/20.jpg)
Part III
Topics for Experimental Projects
14 / 19
![Page 21: Experimental Projects on Web Algorithms - Yury Lifshits](https://reader031.fdocuments.us/reader031/viewer/2022020705/61fb89dd2e268c58cd5f5cbd/html5/thumbnails/21.jpg)
E1 Recommendations for Blog Posts
Available information:Friendship graphComments, hyperlinksKeywords of interests, post content
Task: For every user recommend 10 posts from last daythat seems to be the most interesting for him/her
15 / 19
![Page 22: Experimental Projects on Web Algorithms - Yury Lifshits](https://reader031.fdocuments.us/reader031/viewer/2022020705/61fb89dd2e268c58cd5f5cbd/html5/thumbnails/22.jpg)
E2 CTR Prediction
Available information:Click-or-not bipartite graph
Task: Predict click-through rate for given pair “user-ad”
16 / 19
![Page 23: Experimental Projects on Web Algorithms - Yury Lifshits](https://reader031.fdocuments.us/reader031/viewer/2022020705/61fb89dd2e268c58cd5f5cbd/html5/thumbnails/23.jpg)
E3 Social Networks Visualization
Input:Friendship graph
Similarity:Number of joint friendsLength of shortest path
Task:Construct embedding into 2Dthat put similar people close to each other
17 / 19
![Page 24: Experimental Projects on Web Algorithms - Yury Lifshits](https://reader031.fdocuments.us/reader031/viewer/2022020705/61fb89dd2e268c58cd5f5cbd/html5/thumbnails/24.jpg)
E3 Social Networks Visualization
Input:Friendship graph
Similarity:Number of joint friendsLength of shortest path
Task:Construct embedding into 2Dthat put similar people close to each other
17 / 19
![Page 25: Experimental Projects on Web Algorithms - Yury Lifshits](https://reader031.fdocuments.us/reader031/viewer/2022020705/61fb89dd2e268c58cd5f5cbd/html5/thumbnails/25.jpg)
E4 Disorder Analysis
Disorder inequality for some constant D:
∀p, r , s ∈ {q}∪S : rankr(s) ≤ D ·(rankp(r)+rankp(s))
Tasks:
Compute disorder values for various datasets
Implement disorder-based algorithms for NNS
Study their performance
18 / 19
![Page 26: Experimental Projects on Web Algorithms - Yury Lifshits](https://reader031.fdocuments.us/reader031/viewer/2022020705/61fb89dd2e268c58cd5f5cbd/html5/thumbnails/26.jpg)
E4 Disorder Analysis
Disorder inequality for some constant D:
∀p, r , s ∈ {q}∪S : rankr(s) ≤ D ·(rankp(r)+rankp(s))
Tasks:
Compute disorder values for various datasets
Implement disorder-based algorithms for NNS
Study their performance
18 / 19
![Page 27: Experimental Projects on Web Algorithms - Yury Lifshits](https://reader031.fdocuments.us/reader031/viewer/2022020705/61fb89dd2e268c58cd5f5cbd/html5/thumbnails/27.jpg)
Last Slide
Challenges inWeb Technologies
Recs, Ads,Social Networks
Existing Theory:
Algorithms for NearestNeighbor Search
New Math Problems
New Algorithms
New Experiments p1
p2
p3
p4
p5
p6
q
6
Thanks for your attention! Questions?
19 / 19
![Page 28: Experimental Projects on Web Algorithms - Yury Lifshits](https://reader031.fdocuments.us/reader031/viewer/2022020705/61fb89dd2e268c58cd5f5cbd/html5/thumbnails/28.jpg)
Last Slide
Challenges inWeb Technologies
Recs, Ads,Social Networks
Existing Theory:
Algorithms for NearestNeighbor Search
New Math Problems
New Algorithms
New Experiments p1
p2
p3
p4
p5
p6
q
6
Thanks for your attention! Questions?19 / 19