Presentations: times (22 Oct (week 6) and 19 November-week 10), groups, format
Discovering groups {week 11}
description
Transcript of Discovering groups {week 11}
![Page 1: Discovering groups {week 11}](https://reader035.fdocuments.us/reader035/viewer/2022062814/5681677d550346895ddc8341/html5/thumbnails/1.jpg)
Discovering groups{week 11}
Rensselaer Polytechnic InstituteCSCI-4220 – Network ProgrammingDavid Goldschmidt, Ph.D.
from Programming Collective Intelligence by Toby Segaran, O’Reilly Media, 2007, ISBN 978-0-596-52932-1
![Page 2: Discovering groups {week 11}](https://reader035.fdocuments.us/reader035/viewer/2022062814/5681677d550346895ddc8341/html5/thumbnails/2.jpg)
Text transformation
![Page 3: Discovering groups {week 11}](https://reader035.fdocuments.us/reader035/viewer/2022062814/5681677d550346895ddc8341/html5/thumbnails/3.jpg)
User interaction and querying
![Page 4: Discovering groups {week 11}](https://reader035.fdocuments.us/reader035/viewer/2022062814/5681677d550346895ddc8341/html5/thumbnails/4.jpg)
Data clustering (i)
A cluster is a group of related things Automatic detection of clusters
is a powerful data discovery tool Detect similar user interests,
buying patterns, clickthroughpatterns, etc.
Also applicable to the sciences▪ In computational biology, find groups
(or clusters) of genes that exhibit similar behavior
![Page 5: Discovering groups {week 11}](https://reader035.fdocuments.us/reader035/viewer/2022062814/5681677d550346895ddc8341/html5/thumbnails/5.jpg)
Data clustering (ii)
Data clustering is an example ofan unsupervised learning algorithm... ...which is an AI technique for
discovering structure within one or more datasets
The key goal is to find the distinct group(s) that exist within a given dataset We don’t know what we’ll find
![Page 6: Discovering groups {week 11}](https://reader035.fdocuments.us/reader035/viewer/2022062814/5681677d550346895ddc8341/html5/thumbnails/6.jpg)
Data clustering (iii)We need to first identify a common setof numerical attributes that we can compare to see how similar they are.Can we do anything with word
frequencies?
![Page 7: Discovering groups {week 11}](https://reader035.fdocuments.us/reader035/viewer/2022062814/5681677d550346895ddc8341/html5/thumbnails/7.jpg)
Clustering blogs via feeds (i) If we cluster blogs based on their
word frequencies, maybe we canidentify groups of blogs that are... ...similar in terms of blog content ...similar in terms of writing style ...of interest for searching, cataloging,
etc.
![Page 8: Discovering groups {week 11}](https://reader035.fdocuments.us/reader035/viewer/2022062814/5681677d550346895ddc8341/html5/thumbnails/8.jpg)
Clustering blogs via feeds (ii) A feed is a simple XML document
containing information about a blog and its entries Reader apps enable users
to read multiple blogs ina single window
Being structured data,feeds are generally moresearch-friendly
![Page 9: Discovering groups {week 11}](https://reader035.fdocuments.us/reader035/viewer/2022062814/5681677d550346895ddc8341/html5/thumbnails/9.jpg)
Clustering blogs via feeds (iii) Check out these feeds:
http://blogs.abcnews.com/theblotter/index.rdf
http://www.wired.com/rss/index.xml http://www.tmz.com/rss.xml http://scienceblogs.com/sample/combine
d.xml http://www.neilgaiman.com/journal/feed/
rss.xml
![Page 10: Discovering groups {week 11}](https://reader035.fdocuments.us/reader035/viewer/2022062814/5681677d550346895ddc8341/html5/thumbnails/10.jpg)
Clustering blogs via feeds (iv) Techniques for avoiding stop words:
Ignore words on a predefined stop list Select words from within a predefined
rangeof occurrence percentages ▪ Lower bound of 10%▪ Upper bound of 50%▪ Tune as necessary
![Page 11: Discovering groups {week 11}](https://reader035.fdocuments.us/reader035/viewer/2022062814/5681677d550346895ddc8341/html5/thumbnails/11.jpg)
What next?
Study the resulting blog data Identify any patterns in the data Which blogs are very similar? Which blogs are very different?
How can these techniques beapplied to other types of search? Web search? Enterprise search?
![Page 12: Discovering groups {week 11}](https://reader035.fdocuments.us/reader035/viewer/2022062814/5681677d550346895ddc8341/html5/thumbnails/12.jpg)
Hierarchical clustering (i) Hierarchical clustering is an
algorithmthat groups similar items together At each iteration, the two most similar
items (or groups) are merged For example, given five items A-E:
AB
C
D
E
![Page 13: Discovering groups {week 11}](https://reader035.fdocuments.us/reader035/viewer/2022062814/5681677d550346895ddc8341/html5/thumbnails/13.jpg)
Hierarchical clustering (ii)
Calculate the distances between all items
Group the two items that are closest:
Repeat!
ABAB
C
D
E
![Page 14: Discovering groups {week 11}](https://reader035.fdocuments.us/reader035/viewer/2022062814/5681677d550346895ddc8341/html5/thumbnails/14.jpg)
Hierarchical clustering (iii)
How do we compare group AB to other items? ▪ Use the midpoint of items A and B
ABAB
C
D
EABC
DEx
![Page 15: Discovering groups {week 11}](https://reader035.fdocuments.us/reader035/viewer/2022062814/5681677d550346895ddc8341/html5/thumbnails/15.jpg)
Hierarchical clustering (iv)
When do we stop? ▪ When we have a top-level group that includes
all itemsABAB
C
D
EABC
DE
ABCDE
x
![Page 16: Discovering groups {week 11}](https://reader035.fdocuments.us/reader035/viewer/2022062814/5681677d550346895ddc8341/html5/thumbnails/16.jpg)
Hierarchical clustering (v) The hierarchical part is based on the
discovery order of clusters
This diagram is called a dendrogram...
A
B
C
D
E
AB
DE
ABC
ABCDE
![Page 17: Discovering groups {week 11}](https://reader035.fdocuments.us/reader035/viewer/2022062814/5681677d550346895ddc8341/html5/thumbnails/17.jpg)
Hierarchical clustering (vi) A dendrogram is a graph (or tree)
Distances between nodes of the dendrogram show how similar items (or groups) are
AB is closer (to A and B) than DEis (to D and E), so A and B aremore similarthan D and E
How can wedefine closeness?
A
B
C
D
E
AB
DE
ABC
ABCDE
![Page 18: Discovering groups {week 11}](https://reader035.fdocuments.us/reader035/viewer/2022062814/5681677d550346895ddc8341/html5/thumbnails/18.jpg)
Similarity scores
A similarity score compares two distinct elements from a given set To measure closeness, we need to
calculate a similarity score for each pair of items in the set
Options include:▪ The Euclidean distance score, which is based
onthe distance formula in two-dimensional geometry▪ The Pearson correlation score, which is based
on fitting data points to a line
![Page 19: Discovering groups {week 11}](https://reader035.fdocuments.us/reader035/viewer/2022062814/5681677d550346895ddc8341/html5/thumbnails/19.jpg)
Euclidean distance score
To find the Euclidean distance betweentwo data points, use the distance formula:
distance = √ (y2 – y1)2 + (x2 – x1)2
The larger the distance between two items,the less similar they are
So use the reciprocal of distance as a measure of similarity (but be careful of division by zero)
![Page 20: Discovering groups {week 11}](https://reader035.fdocuments.us/reader035/viewer/2022062814/5681677d550346895ddc8341/html5/thumbnails/20.jpg)
Pearson correlation score (i) The Pearson correlation score is
derived by determining the best-fit line for a given set
x
x
xx
xxx
The best-fit line, on average, comes as close as possible to each item
The Pearson correlation score is a coefficient measuring the degree to which items are on the best-fit line
v1
v2
x
![Page 21: Discovering groups {week 11}](https://reader035.fdocuments.us/reader035/viewer/2022062814/5681677d550346895ddc8341/html5/thumbnails/21.jpg)
Pearson correlation score (ii) The Pearson correlation score tells
us how closely items are correlated to one another 1.0 is a perfect match; ~0.0 is no
relationship correlation score: 0.4 correlation score: 0.8
x
x
xx
xxx xx xxx
xx
xx
v2 v2
v1 v1
![Page 22: Discovering groups {week 11}](https://reader035.fdocuments.us/reader035/viewer/2022062814/5681677d550346895ddc8341/html5/thumbnails/22.jpg)
Pearson correlation score (iii) The algorithm is:
Calculate sum(v1) and sum(v2) Calculate the sum of the
squares of v1 and v2▪ Call them sum1Sq and sum2Sq
Calculate the sum of the products of v1 and v2▪ (v1[0] * v2[0]) + (v1[1] * v2[1]) + ... + (v1[n-
1] * v2[n-1])▪ Call this pSum
x
x
xx
xxxx
v2
v1
![Page 23: Discovering groups {week 11}](https://reader035.fdocuments.us/reader035/viewer/2022062814/5681677d550346895ddc8341/html5/thumbnails/23.jpg)
Pearson correlation score (iv)
Calculate the Pearson score:
Much more complex, but often better thanthe Euclidean distance score
sum(v1) * sum(v2) pSum – ( ) nr = sum1Sq – sum(v1)2 sum2Sq – sum(v2)2
* n n√
![Page 24: Discovering groups {week 11}](https://reader035.fdocuments.us/reader035/viewer/2022062814/5681677d550346895ddc8341/html5/thumbnails/24.jpg)
What next?
Review the blog-data dendrograms Identify any patterns in the data Which blogs are very similar? Which blogs are very different?
How can these techniques beapplied to other types of search? Web search? Enterprise search?