Photo Clustering of Social Events by Extending PhotoTOC to a Rich Context
-
Upload
xavier-giro -
Category
Technology
-
view
235 -
download
0
description
Transcript of Photo Clustering of Social Events by Extending PhotoTOC to a Rich Context
![Page 1: Photo Clustering of Social Events by Extending PhotoTOC to a Rich Context](https://reader033.fdocuments.us/reader033/viewer/2022060110/5561e470d8b42aa5068b4d2b/html5/thumbnails/1.jpg)
Photo Clustering of Social Events by Extending PhotoTOC to a Rich Context
Daniel Manchón-Vizuete Irene Gris-Sarabia Xavier Giró-i-Nieto
ICMR 2014 Workshop on Social Events in Web Multimedia, Glasgow (Scotland/UK), 01/04/2014
![Page 2: Photo Clustering of Social Events by Extending PhotoTOC to a Rich Context](https://reader033.fdocuments.us/reader033/viewer/2022060110/5561e470d8b42aa5068b4d2b/html5/thumbnails/2.jpg)
OutlineMotivation Related workApproachResultsFuture workConclusions
Photo Clustering of Social
Events by Extending
PhotoTOC to a Rich Context
D. Manchón-Vizuete I. Gris-Sarabia X. Giró-i-Nieto
![Page 3: Photo Clustering of Social Events by Extending PhotoTOC to a Rich Context](https://reader033.fdocuments.us/reader033/viewer/2022060110/5561e470d8b42aa5068b4d2b/html5/thumbnails/3.jpg)
Motivation
![Page 4: Photo Clustering of Social Events by Extending PhotoTOC to a Rich Context](https://reader033.fdocuments.us/reader033/viewer/2022060110/5561e470d8b42aa5068b4d2b/html5/thumbnails/4.jpg)
Motivation
![Page 5: Photo Clustering of Social Events by Extending PhotoTOC to a Rich Context](https://reader033.fdocuments.us/reader033/viewer/2022060110/5561e470d8b42aa5068b4d2b/html5/thumbnails/5.jpg)
Motivation
Credit: National Institute on Aging and FDA.
![Page 6: Photo Clustering of Social Events by Extending PhotoTOC to a Rich Context](https://reader033.fdocuments.us/reader033/viewer/2022060110/5561e470d8b42aa5068b4d2b/html5/thumbnails/6.jpg)
Motivation
![Page 7: Photo Clustering of Social Events by Extending PhotoTOC to a Rich Context](https://reader033.fdocuments.us/reader033/viewer/2022060110/5561e470d8b42aa5068b4d2b/html5/thumbnails/7.jpg)
Motivation
Credit: Alan Smeaton and Cathal Gurrin (Dublin City University, 2013)
![Page 8: Photo Clustering of Social Events by Extending PhotoTOC to a Rich Context](https://reader033.fdocuments.us/reader033/viewer/2022060110/5561e470d8b42aa5068b4d2b/html5/thumbnails/8.jpg)
OutlineMotivation Related workApproachResultsFuture workConclusions
Photo Clustering of Social
Events by Extending
PhotoTOC to a Rich Context
D. Manchón-Vizuete I. Gris-Sarabia X. Giró-i-Nieto
![Page 9: Photo Clustering of Social Events by Extending PhotoTOC to a Rich Context](https://reader033.fdocuments.us/reader033/viewer/2022060110/5561e470d8b42aa5068b4d2b/html5/thumbnails/9.jpg)
Related work
PhotoTOC[Platt et al, PACRIM 2003]
![Page 10: Photo Clustering of Social Events by Extending PhotoTOC to a Rich Context](https://reader033.fdocuments.us/reader033/viewer/2022060110/5561e470d8b42aa5068b4d2b/html5/thumbnails/10.jpg)
Related work
[Becker, Naaman & Gravano, ACM WSDM 2010]
Rich context
User-provided annotations
Automatically generated information
TitleDescription
Creation timeGeo-location
![Page 11: Photo Clustering of Social Events by Extending PhotoTOC to a Rich Context](https://reader033.fdocuments.us/reader033/viewer/2022060110/5561e470d8b42aa5068b4d2b/html5/thumbnails/11.jpg)
OutlineMotivation Related workApproachResultsFuture workConclusions
Photo Clustering of Social
Events by Extending
PhotoTOC to a Rich Context
D. Manchón-Vizuete I. Gris-Sarabia X. Giró-i-Nieto
![Page 12: Photo Clustering of Social Events by Extending PhotoTOC to a Rich Context](https://reader033.fdocuments.us/reader033/viewer/2022060110/5561e470d8b42aa5068b4d2b/html5/thumbnails/12.jpg)
Approach(a) Temporal sorting by each user independently
Hi, I’m John. Hi, I’m Emily.
![Page 13: Photo Clustering of Social Events by Extending PhotoTOC to a Rich Context](https://reader033.fdocuments.us/reader033/viewer/2022060110/5561e470d8b42aa5068b4d2b/html5/thumbnails/13.jpg)
Approach(b) Temporal-based oversegmentation in mini-clusters
PhotoTOC[Platt et al, PacRim 2003]
![Page 14: Photo Clustering of Social Events by Extending PhotoTOC to a Rich Context](https://reader033.fdocuments.us/reader033/viewer/2022060110/5561e470d8b42aa5068b4d2b/html5/thumbnails/14.jpg)
Approach(b) Temporal-based oversegmentation in mini-clusters
![Page 15: Photo Clustering of Social Events by Extending PhotoTOC to a Rich Context](https://reader033.fdocuments.us/reader033/viewer/2022060110/5561e470d8b42aa5068b4d2b/html5/thumbnails/15.jpg)
Approach(c) Sequential merging of mini-clusters
? tavg(·) avg(·) avg(·)avg(·)
![Page 16: Photo Clustering of Social Events by Extending PhotoTOC to a Rich Context](https://reader033.fdocuments.us/reader033/viewer/2022060110/5561e470d8b42aa5068b4d2b/html5/thumbnails/16.jpg)
Approach(c) Sequential merging of mini-clusters
Weightedmodalities
● creation (or upload) time● geolocation● textual labels● same user
![Page 17: Photo Clustering of Social Events by Extending PhotoTOC to a Rich Context](https://reader033.fdocuments.us/reader033/viewer/2022060110/5561e470d8b42aa5068b4d2b/html5/thumbnails/17.jpg)
Approach(c) Sequential merging of mini-clusters
Geolocation (d=haversine)Time stamp (d=L1)
Text labels (d=Jaccard) Same user (d=boolean)
![Page 18: Photo Clustering of Social Events by Extending PhotoTOC to a Rich Context](https://reader033.fdocuments.us/reader033/viewer/2022060110/5561e470d8b42aa5068b4d2b/html5/thumbnails/18.jpg)
Approach(c) Sequential merging of mini-clusters
![Page 19: Photo Clustering of Social Events by Extending PhotoTOC to a Rich Context](https://reader033.fdocuments.us/reader033/viewer/2022060110/5561e470d8b42aa5068b4d2b/html5/thumbnails/19.jpg)
Approach(c) Sequential merging of mini-clusters
Mean and std. deviation learned on
pairs of photos within the same training event.
![Page 20: Photo Clustering of Social Events by Extending PhotoTOC to a Rich Context](https://reader033.fdocuments.us/reader033/viewer/2022060110/5561e470d8b42aa5068b4d2b/html5/thumbnails/20.jpg)
Approach(c) Sequential merging of mini-clusters
phi function
![Page 21: Photo Clustering of Social Events by Extending PhotoTOC to a Rich Context](https://reader033.fdocuments.us/reader033/viewer/2022060110/5561e470d8b42aa5068b4d2b/html5/thumbnails/21.jpg)
Approach(c) Sequential merging of mini-clusters
decision threhold
![Page 22: Photo Clustering of Social Events by Extending PhotoTOC to a Rich Context](https://reader033.fdocuments.us/reader033/viewer/2022060110/5561e470d8b42aa5068b4d2b/html5/thumbnails/22.jpg)
Approach(c) Sequential merging of mini-clusters
![Page 23: Photo Clustering of Social Events by Extending PhotoTOC to a Rich Context](https://reader033.fdocuments.us/reader033/viewer/2022060110/5561e470d8b42aa5068b4d2b/html5/thumbnails/23.jpg)
OutlineMotivation Related workApproachExperimentsFuture workConclusions
Photo Clustering of Social
Events by Extending
PhotoTOC to a Rich Context
D. Manchón-Vizuete I. Gris-Sarabia X. Giró-i-Nieto
![Page 24: Photo Clustering of Social Events by Extending PhotoTOC to a Rich Context](https://reader033.fdocuments.us/reader033/viewer/2022060110/5561e470d8b42aa5068b4d2b/html5/thumbnails/24.jpg)
DatasetReSEED - Social Event Detection Dataset
Timo Reuter, Symeon Papadopoulos, Vasilios Mezaris & Philipp Cimiano (ACM MM Sys 2014)[Paper: http://dx.doi.org/10.1145/2557642.2563674] [Dataset: http://greententacle.techfak.uni-bielefeld.de/reseed/]
![Page 25: Photo Clustering of Social Events by Extending PhotoTOC to a Rich Context](https://reader033.fdocuments.us/reader033/viewer/2022060110/5561e470d8b42aa5068b4d2b/html5/thumbnails/25.jpg)
Results
![Page 26: Photo Clustering of Social Events by Extending PhotoTOC to a Rich Context](https://reader033.fdocuments.us/reader033/viewer/2022060110/5561e470d8b42aa5068b4d2b/html5/thumbnails/26.jpg)
Results
![Page 27: Photo Clustering of Social Events by Extending PhotoTOC to a Rich Context](https://reader033.fdocuments.us/reader033/viewer/2022060110/5561e470d8b42aa5068b4d2b/html5/thumbnails/27.jpg)
Results
Diversity of cameras (qualities)
![Page 28: Photo Clustering of Social Events by Extending PhotoTOC to a Rich Context](https://reader033.fdocuments.us/reader033/viewer/2022060110/5561e470d8b42aa5068b4d2b/html5/thumbnails/28.jpg)
Diversity of locations
Results
![Page 29: Photo Clustering of Social Events by Extending PhotoTOC to a Rich Context](https://reader033.fdocuments.us/reader033/viewer/2022060110/5561e470d8b42aa5068b4d2b/html5/thumbnails/29.jpg)
Results
Visual redundancy (not exploited)
![Page 30: Photo Clustering of Social Events by Extending PhotoTOC to a Rich Context](https://reader033.fdocuments.us/reader033/viewer/2022060110/5561e470d8b42aa5068b4d2b/html5/thumbnails/30.jpg)
Results
False split
![Page 31: Photo Clustering of Social Events by Extending PhotoTOC to a Rich Context](https://reader033.fdocuments.us/reader033/viewer/2022060110/5561e470d8b42aa5068b4d2b/html5/thumbnails/31.jpg)
Results
False merge
![Page 32: Photo Clustering of Social Events by Extending PhotoTOC to a Rich Context](https://reader033.fdocuments.us/reader033/viewer/2022060110/5561e470d8b42aa5068b4d2b/html5/thumbnails/32.jpg)
OutlineMotivation Related workApproachExperimentsFuture workConclusions
Photo Clustering of Social
Events by Extending
PhotoTOC to a Rich Context
D. Manchón-Vizuete I. Gris-Sarabia X. Giró-i-Nieto
![Page 33: Photo Clustering of Social Events by Extending PhotoTOC to a Rich Context](https://reader033.fdocuments.us/reader033/viewer/2022060110/5561e470d8b42aa5068b4d2b/html5/thumbnails/33.jpg)
Future work
● Median-based normalisation for robustness against outliers.
● Better optimisation of parameters K and d.
● Improve fusion approach (SVM ?).
● Event-dependent merging criteria.
● Efficient use of visual features (higher computation).
![Page 34: Photo Clustering of Social Events by Extending PhotoTOC to a Rich Context](https://reader033.fdocuments.us/reader033/viewer/2022060110/5561e470d8b42aa5068b4d2b/html5/thumbnails/34.jpg)
OutlineMotivation Related workApproachExperimentsFuture workConclusions
Photo Clustering of Social
Events by Extending
PhotoTOC to a Rich Context
D. Manchón-Vizuete I. Gris-Sarabia X. Giró-i-Nieto
![Page 35: Photo Clustering of Social Events by Extending PhotoTOC to a Rich Context](https://reader033.fdocuments.us/reader033/viewer/2022060110/5561e470d8b42aa5068b4d2b/html5/thumbnails/35.jpg)
Conclusions● All considered context (geo, user, tags) can help.
● Watch out for outliers.
● Divide and conquer… sequentially.
Thank you
SEWM !
![Page 36: Photo Clustering of Social Events by Extending PhotoTOC to a Rich Context](https://reader033.fdocuments.us/reader033/viewer/2022060110/5561e470d8b42aa5068b4d2b/html5/thumbnails/36.jpg)
Thank you !
http://bitsearch.blogspot.com
https://imatge.upc.edu/web/people/xavier-giro
@DocXavi
/ProfessorXavi