Post on 21-Dec-2015
Using Relevance Feedback in Multimedia Databases
Chotirat “Ann” Ratanamahatana
Eamonn Keogh
7th International Conference on VISual Information Systemsat 10th International Conference on Distributed Multimedia Systems
September 9, 2004
Roadmap
• Time series in multimedia databases and their similarity
measures
• Euclidean distance and its limitation
• Dynamic time warping (DTW)
• Global constraints and R-K Band
• Relevance Feedback and Query Refinement
• Experimental Evaluation
• Conclusions and future work
What are Time Series• A collection of observations made sequentially
in time.• People measure things…
and things…change over time…
• Their blood pressure• George Bush's popularity rating• The annual rainfall in San Francisco• The value of their Google stock
• Their blood pressure• George Bush's popularity rating• The annual rainfall in San Francisco• The value of their Google stock
Time Series in Multimedia Databases
Image data may best be thought of as time series…
Image to Time Series
Video to Time Series
0 50 100 150-1.5
-1
-0.5
0
0.5
1
1.5
Hand moving above holster
Hand moving downto grasp gun
Steady pointing
Hand moving toshoulder level
Hand at rest
Time Series in Multimedia Databases
0 10 20 30 40 50 60 70 80 90
Hand at rest
Aiming gun
Hand moving toshoulder level
Hand moving downto grasp gun
Hand movingabove holster
0 10 20 30 40 50 60 70 80 90
Hand at rest
Aiming gun
Hand moving toshoulder level
Hand moving downto grasp gun
Hand movingabove holster
0 10 20 30 40 50 60 70 80 90
Hand at rest
Aiming gun
Hand moving toshoulder level
Hand moving downto grasp gun
Hand movingabove holster
0 10 20 30 40 50 60 70 80 90
Hand at rest
Aiming gun
Hand moving toshoulder level
Hand moving downto grasp gun
Hand movingabove holster
0 10 20 30 40 50 60 70 80 90
Hand at rest
Aiming gun
Hand moving toshoulder level
Hand moving downto grasp gun
Hand movingabove holster
0 10 20 30 40 50 60 70 80 90
Hand at rest
Aiming gun
Hand moving toshoulder level
Hand moving downto grasp gun
Hand movingabove holster
0 10 20 30 40 50 60 70 80 90
Hand at rest
Aiming gun
Hand moving toshoulder level
Hand moving downto grasp gun
Hand movingabove holster
Video
0 50 100 150 200 250 300 350 400 4500
0.5
1
0 50 100 150 200 250 300 350 400 4500
0.5
1
George Washington’sManuscript
Classification in Time Series
Pattern Recognition is a type of supervised classification where an input pattern is classified into one of the classes based on its similarity to these predefined classes.
Class BClass BClass AClass A
Which class does
belong to?
Euclidean Distance MetricGiven 2 time series
Q = q1, …, qn and
C = c1, …, cn
their Euclidean distance is
defined as
n
iii cqCQD
1
2)(),(
0 50 100 150-1.5
-1
-0.5
0
0.5
1
1.5
0 50 100 150-1.5
-1
-0.5
0
0.5
1
1.5
Q
C
Limitations of Euclidean MetricVery sensitive to some distortion in the data
Training data consistsof 10 instances fromeach of the 3 classes
Training data consistsof 10 instances fromeach of the 3 classes
Perform a 1-nearest neighbor algorithm, with “leaving-one-out”
evaluation, averaged over 100 runs.
Euclidean distance Error rate:29.77%
DTW Error rate:3.33 %
Dynamic Time Warping (DTW)
Euclidean DistanceOne-to-one alignments
Time Warping DistanceNon-linear alignments are allowed
How Is DTW Calculated? (I)
QC
K
k kwCQDTW1
min),(
Warping path w
Q
C
Q
C
How Is DTW Calculated? (II)Each warping path w can be found using dynamic programming to evaluatethe following recurrence:
)}1,(),,1(),1,1(min{),(),( jijijicqdji ji
where γ(i, j) is the cumulative distance of the distance d(i, j) and its minimumcumulative distance among the adjacent cells.
(i-1, j)
(i, j-1)
(i, j)
(i-1, j-1)
Global Constraints (I)
C
Q
C
Q
C
Q
C
Q
Sakoe-Chiba Band Itakura Parallelogram
Prevent any unreasonable
warping
Prevent any unreasonable
warping
Global Constraints (II)
Ri
Sakoe-Chiba Band Itakura Parallelogram
A Global Constraint for a sequence of size m is defined by R, whereRi = d 0 d m, 1 i m.
Ri defines a freedom of warping above and to the right of the diagonal at any given point i in the sequence.
Ratanamahatana-Keogh Band (R-K Band)
Solution: we create an arbitrary shape and size of the band that is appropriate for the data we want to classify.
How Do We Create an R-K Band?First Attempt: We could look at the data and manually create the shape of the bands.
(then we need to adjust the width of each band as well until we get a good result)
50 100 150 200 250
50
100
150
200
250
50 100 150 200 250
50
100
150
200
250
0 50 100 150 200 250-2.5
-2
-1.5
-1
-0.5
0
0.5
1
1.5
0 50 100 150 200 250-2.5
-2
-1.5
-1
-0.5
0
0.5
1
100 % Accuracy!
Learning an R-K Band Automatically
50 100 150 200 250
50
100
150
200
250
0 50 100 150 200 250-2.5
-2
-1.5
-1
-0.5
0
0.5
1
0 50 100 150 200 250-2.5
-2
-1.5
-1
-0.5
0
0.5
1
1.5
50 100 150 200 250
50
100
150
200
250
Our heuristic search algorithm automatically learns the bands from the data.(sometimes, we can even get an unintuitive shape that give a good result.)
100 % Accuracy as well!
Calculate h(1)
Calculate h(2)
h(2) > h(1) ? Yes No
Calculate h(1)
Calculate h(2)
h(2) > h(1) ? Yes No
R-K Band Learning With Heuristic Search
R-K Band Learning in Action!
Classification Examples with R-K Bands
Error rate
Euclidean 32.13%
DTW 10% 4.52%
R-K Bands 0.9%
Face Classification
Relevance Feedback
• A well-known and effective method in improving the query performance, especially in text-mining domains.– Refining the query based on user’s reaction
• Only relatively little research has been done on relevance feedback in images or multimedia data.
Query Refinement
Averaging a collection of time series using DTW, according to their weights and warping (DTW) alignments.
Averaged SequenceAveraged Sequence
1. Gun Problem
2. Leaf Dataset
3. Handwritten Word Spotting data
Experiment: Datasets
0 50 100 150-1
-0.5
0
0.5
1
1.5
2
2.5
0 50 100 150-1
-0.5
0
0.5
1
1.5
2
2.5
Experimental Design
Given an initial query, we measure the precision and recall for each round of the relevance feedback retrieval.• Show the 10 best matches (k-nearest neighbors).• User ranks each result.• Accumulatively build the training set.• Learn an R-K band according to the current training data.• Generate a new query (query refinement), and repeat.
Results: Gun
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Recall
Pre
cisi
on
Iteration 1Iteration 2Iteration 3Iteration 4Iteration 5
Gun
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Recall
Pre
cisi
on
Iteration 1Iteration 2Iteration 3Iteration 4Iteration 5
Gun
Results: Leaf
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Recall
Pre
cis
ion
Iteration 1Iteration 2Iteration 3Iteration 4Iteration 5
Leaf
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Recall
Pre
cis
ion
Iteration 1Iteration 2Iteration 3Iteration 4Iteration 5
Leaf
Results: Wordspotting
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Recall
Pre
cisi
on
Iteration 1Iteration 2Iteration 3Iteration 4Iteration 5
WordSpotting 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Recall
Pre
cisi
on
Iteration 1Iteration 2Iteration 3Iteration 4Iteration 5
WordSpotting
Conclusions
• Different shapes and widths of the band contributes to the classification accuracy / precision.
• We have shown that incorporating R-K Band into relevance feedback can reduce the error rate in classification, and improve the precision at all recall levels in video and image retrieval.
Future Work
• Investigate other choices that may make envelope learning more accurate.– Heuristic functions– Search algorithm (refining the search)
• Is there a way to always guarantee an optimal solution?• Examine the best way to deal with multi-variate time
series for more complex data.• Explore other utilities of R-K Band and relevance
feedback, specifically on real-world problems: music, bioinformatics, biomedical data, etc.
UCR Time Series Data Mining Archive: http://www.cs.ucr.edu/~eamonn/TSDMA
Contact: ratana@cs.ucr.edu eamonn@cs.ucr.edu
Homepage: http://www.cs.ucr.edu/~ratana
All datasets are publicly available at: