MediaEval 2015 - JRS at Synchronization of Multi-user Event Media Task
-
Upload
multimediaeval -
Category
Education
-
view
98 -
download
0
Transcript of MediaEval 2015 - JRS at Synchronization of Multi-user Event Media Task
![Page 1: MediaEval 2015 - JRS at Synchronization of Multi-user Event Media Task](https://reader030.fdocuments.us/reader030/viewer/2022021506/587abf9e1a28abc0478b692f/html5/thumbnails/1.jpg)
JRS at Synchronization of
Multi-user Event Media Task Hannes Fassold, Harald Stiegler, Felix Lee, Werner Bailer
MediaEval Workshop, September 14-15, 2015, Wurzen
![Page 2: MediaEval 2015 - JRS at Synchronization of Multi-user Event Media Task](https://reader030.fdocuments.us/reader030/viewer/2022021506/587abf9e1a28abc0478b692f/html5/thumbnails/2.jpg)
Gallery synchronization – Approach Overview
Based on visual information (images, key frames
of videos) and given time stamps
Probabilistic approach
Uses visual similarity of image pairs
(SIFT descriptors)
Many potential solutions (‚hypotheses‘) are calculated
in a ‚probabilistic way‘ (so with a inherent random
component)
„Best“ hypothesis is calculated from these hypotheses
2
![Page 3: MediaEval 2015 - JRS at Synchronization of Multi-user Event Media Task](https://reader030.fdocuments.us/reader030/viewer/2022021506/587abf9e1a28abc0478b692f/html5/thumbnails/3.jpg)
Gallery synchronization – Approach Determine image matches
Determine good image matches (I, J)
Images with a high degree of visual similarity
Determine best match J for image I via exhaustive
matching of their set of SIFT descriptors
Descriptor calculation and matching done one GPU
SIFT very robust to orientation, scale change
Apply geometric verification step on match (I, J)
Gives a variable number of homographies and the
number of points ht supporting a homography
3
![Page 4: MediaEval 2015 - JRS at Synchronization of Multi-user Event Media Task](https://reader030.fdocuments.us/reader030/viewer/2022021506/587abf9e1a28abc0478b692f/html5/thumbnails/4.jpg)
Gallery synchronization – Approach Determine visual similarity score s(i,j)
Calculation of similarity score s(i,j) for a match (I,J)
Discard all homographies with too less supporting
points (with ht < threshold)
Pick the k homographies with the highest number of
supporting points ht, and clip the values ht to range
Average and sum of the k remaining (and clipped)
values ht,clip is calculated
s(i,j) is geometric average of ‘average’ and ‘sum’
4
![Page 5: MediaEval 2015 - JRS at Synchronization of Multi-user Event Media Task](https://reader030.fdocuments.us/reader030/viewer/2022021506/587abf9e1a28abc0478b692f/html5/thumbnails/5.jpg)
Gallery synchronization – Approach Calculate ‚connection‘ magnitude c(k,l)
c(k,l) calculated for each gallery pair (k, l)
c(k,l) gives some information about the ‘stability’ of the gallery pair (k, l)
More ‘stable’ gallery pairs have a higher number of matches and a low deviation of the time difference
values of the matches
c(k,l) calculated from: number of matches, average
visual similarity score of the matches, average
deviation of the time differences of the matches
5
![Page 6: MediaEval 2015 - JRS at Synchronization of Multi-user Event Media Task](https://reader030.fdocuments.us/reader030/viewer/2022021506/587abf9e1a28abc0478b692f/html5/thumbnails/6.jpg)
Gallery synchronization – Approach Calculate one hypothesis
Pick a gallery pair (k,l) randomly
Probability of picking the pair (k,l) proportional to
connection magnitude c(k,l)
Calculate time difference (k,l) and propagate it
Apply k-means clustering (k=3-5) on the time
difference values of all matches, and pick one cluster
center randomly as (k,l)
Propagate it to calculate other time differences (k’,l)
Iterate this process until we have all time
differences (for all galleries)
6
![Page 7: MediaEval 2015 - JRS at Synchronization of Multi-user Event Media Task](https://reader030.fdocuments.us/reader030/viewer/2022021506/587abf9e1a28abc0478b692f/html5/thumbnails/7.jpg)
Gallery synchronization – Approach Calculate ‚best‘ hypothesis
Calculate many hypotheses
Several hundreds or thousands
‘Best’ hypothesis is calculated as the medoid of
all hypotheses
Can be seen as the ‘most-inner’ point, when interpreting the hypotheses as n-dimensional points
7
![Page 8: MediaEval 2015 - JRS at Synchronization of Multi-user Event Media Task](https://reader030.fdocuments.us/reader030/viewer/2022021506/587abf9e1a28abc0478b692f/html5/thumbnails/8.jpg)
Sub-event clustering – Approach
Relies solely on time information
Calculate ‘corrected’ time stamps
Using the calculated gallery time offsets
Apply 1-dimensional k-means algorithm
Value ‘k’ is determined on the data set size and on a user-defined ‘granularity’ parameter
8
![Page 9: MediaEval 2015 - JRS at Synchronization of Multi-user Event Media Task](https://reader030.fdocuments.us/reader030/viewer/2022021506/587abf9e1a28abc0478b692f/html5/thumbnails/9.jpg)
Results Gallery sychronization
9
Results for Tour de France 2014 (TDF14)
and NAMM 2015 (NAMM15) dataset
![Page 10: MediaEval 2015 - JRS at Synchronization of Multi-user Event Media Task](https://reader030.fdocuments.us/reader030/viewer/2022021506/587abf9e1a28abc0478b692f/html5/thumbnails/10.jpg)
Results Subevent clustering
10
Subevent clustering results for Tour de France 2014 (left) and
NAMM 2015 (right).
Run 2 has a finer ‚granularity‘ (higher k) than run 1.
![Page 11: MediaEval 2015 - JRS at Synchronization of Multi-user Event Media Task](https://reader030.fdocuments.us/reader030/viewer/2022021506/587abf9e1a28abc0478b692f/html5/thumbnails/11.jpg)
Conclusion
Gallery synchronization
High accuracy (~ 80 – 90 %) for both TDF and NAMM
But precision very low on NAMM (so many galleries
not correctly synchronized)
NAMM 2015 content visually more ‚challenging‘ (much more wrong image matches)
Subevent clustering
Worse results (precision, recall) for NAMM 2015
might be due to less successfull gallery
synchronization for this dataset
11
![Page 12: MediaEval 2015 - JRS at Synchronization of Multi-user Event Media Task](https://reader030.fdocuments.us/reader030/viewer/2022021506/587abf9e1a28abc0478b692f/html5/thumbnails/12.jpg)
12 Acknowledgments
This work was supported by the European Commission under the grant agreement no. FP7-610370, „ICoSOLE“ http://www.icosole.eu
![Page 13: MediaEval 2015 - JRS at Synchronization of Multi-user Event Media Task](https://reader030.fdocuments.us/reader030/viewer/2022021506/587abf9e1a28abc0478b692f/html5/thumbnails/13.jpg)
The research leading to these results has received funding from the European Union's
Seventh Framework Programme (FP7/2007-2013) under grant agreement n° 610370,
ICoSOLE (Immersive Coverage of Spatially Outspread Live Events“ (http://www.icosole.eu/).
Hannes Fassold
JOANNEUM RESEARCH – DIGITAL
http://www.joanneum.at/digital