Ter Haar Romeny, ICPR 2010 Introduction to Scale-Space and Deep Structure.
-
Upload
houston-leppert -
Category
Documents
-
view
219 -
download
1
Transcript of Ter Haar Romeny, ICPR 2010 Introduction to Scale-Space and Deep Structure.
![Page 1: Ter Haar Romeny, ICPR 2010 Introduction to Scale-Space and Deep Structure.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551b1e6e550346f70d8b65d0/html5/thumbnails/1.jpg)
ter Haar Romeny, ICPR 2010
Introduction toScale-Space
and Deep Structure
![Page 2: Ter Haar Romeny, ICPR 2010 Introduction to Scale-Space and Deep Structure.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551b1e6e550346f70d8b65d0/html5/thumbnails/2.jpg)
ter Haar Romeny, ICPR 2010
Importance of Scale
Painting by Dali
• Objects exist at certain ranges of scale.
• It is not known a priory at what scale to look.
![Page 3: Ter Haar Romeny, ICPR 2010 Introduction to Scale-Space and Deep Structure.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551b1e6e550346f70d8b65d0/html5/thumbnails/3.jpg)
ter Haar Romeny, ICPR 2010
At the original scale of a dithered image we cannot calculate a derivative.We need to observe the image at a certain scale.
BLUR
![Page 4: Ter Haar Romeny, ICPR 2010 Introduction to Scale-Space and Deep Structure.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551b1e6e550346f70d8b65d0/html5/thumbnails/4.jpg)
ter Haar Romeny, ICPR 2010
Solution?
Look at all scales simultaneously
Scale
x
y
Scale Space
![Page 5: Ter Haar Romeny, ICPR 2010 Introduction to Scale-Space and Deep Structure.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551b1e6e550346f70d8b65d0/html5/thumbnails/5.jpg)
ter Haar Romeny, ICPR 2010
Scale Space in Human Vision
The human visual system is a
multi-scale sampling device
The retina contains receptive
fields; groups of receptors
assembled in such a way that they
form a set of apertures of widely
varying size.
![Page 6: Ter Haar Romeny, ICPR 2010 Introduction to Scale-Space and Deep Structure.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551b1e6e550346f70d8b65d0/html5/thumbnails/6.jpg)
ter Haar Romeny, ICPR 2010
Practical Implementation
Convolve the image with a Gaussian Kernel
2
221
2/22
)2(1),(
D
D
xx
exG
![Page 7: Ter Haar Romeny, ICPR 2010 Introduction to Scale-Space and Deep Structure.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551b1e6e550346f70d8b65d0/html5/thumbnails/7.jpg)
ter Haar Romeny, ICPR 2010
We can calculate derivatives and combinations of them at all scales
22yx LL
yyxx LL
Gradient Magnitude
Laplacian
Original Image
![Page 8: Ter Haar Romeny, ICPR 2010 Introduction to Scale-Space and Deep Structure.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551b1e6e550346f70d8b65d0/html5/thumbnails/8.jpg)
ter Haar Romeny, ICPR 2010
Main Topic
In this presentation we will show how
we can exploit the deep structure of
images to define invariant interest
points and features which can be used
for matching problems in computer
vision.
We consider only grey-value images.
![Page 9: Ter Haar Romeny, ICPR 2010 Introduction to Scale-Space and Deep Structure.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551b1e6e550346f70d8b65d0/html5/thumbnails/9.jpg)
ter Haar Romeny, ICPR 2010
Interest Points
The locations of particularly characteristic points
are called the interest points or key points.
These interest points have to be as invariant as
possible, but at the same time they have to carry
a lot of distinctive information.
![Page 10: Ter Haar Romeny, ICPR 2010 Introduction to Scale-Space and Deep Structure.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551b1e6e550346f70d8b65d0/html5/thumbnails/10.jpg)
ter Haar Romeny, ICPR 2010
Why interest points in scale-space?
Information in interest points is defined
by their neighborhood. But how big
should we choose this neighborhood?
• Let’s take the corners of the mouth as interest points.
• The red circles are the areas in which the information is gathered.
• If we make the picture bigger, the size of the neighborhood is too small.
• The neighborhood should scale with the image
![Page 11: Ter Haar Romeny, ICPR 2010 Introduction to Scale-Space and Deep Structure.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551b1e6e550346f70d8b65d0/html5/thumbnails/11.jpg)
ter Haar Romeny, ICPR 2010
When the interest points are detected in
scale space they do not only have spatial
coordinates x and y, but also a scale .
This scale tells us how big the
neighborhood should be.
Why interest points in scale-space?
![Page 12: Ter Haar Romeny, ICPR 2010 Introduction to Scale-Space and Deep Structure.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551b1e6e550346f70d8b65d0/html5/thumbnails/12.jpg)
ter Haar Romeny, ICPR 2010
Which interest points to use?
Our interest points have to be detected in scale space.They also have to…
…contain a lot of information…be reproducible…be stable…be well understood
![Page 13: Ter Haar Romeny, ICPR 2010 Introduction to Scale-Space and Deep Structure.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551b1e6e550346f70d8b65d0/html5/thumbnails/13.jpg)
ter Haar Romeny, ICPR 2010
We suggest Top-Points
The points we introduce have these
desired properties.
![Page 14: Ter Haar Romeny, ICPR 2010 Introduction to Scale-Space and Deep Structure.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551b1e6e550346f70d8b65d0/html5/thumbnails/14.jpg)
ter Haar Romeny, ICPR 2010
Critical Points, Paths and Top-Points
Maxima
Minimum
SaddlesL=0Critical Points
![Page 15: Ter Haar Romeny, ICPR 2010 Introduction to Scale-Space and Deep Structure.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551b1e6e550346f70d8b65d0/html5/thumbnails/15.jpg)
ter Haar Romeny, ICPR 2010
Critical Points, Paths and Top-Points
Maxima
Minimum
SaddlesL=0Critical Points
Det(H)=0Top-Points
![Page 16: Ter Haar Romeny, ICPR 2010 Introduction to Scale-Space and Deep Structure.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551b1e6e550346f70d8b65d0/html5/thumbnails/16.jpg)
ter Haar Romeny, ICPR 2010
Possible to calculate them for every function of the image L(x,y,)
Original Gradient Magnitude
Laplacian Det(H)
![Page 17: Ter Haar Romeny, ICPR 2010 Introduction to Scale-Space and Deep Structure.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551b1e6e550346f70d8b65d0/html5/thumbnails/17.jpg)
ter Haar Romeny, ICPR 2010
Detecting critical paths
Since for a critical path
L=0, intersection of
level surfaces Lx=0 with
Ly=0
will give the critical
paths.
![Page 18: Ter Haar Romeny, ICPR 2010 Introduction to Scale-Space and Deep Structure.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551b1e6e550346f70d8b65d0/html5/thumbnails/18.jpg)
ter Haar Romeny, ICPR 2010
Detecting Top-Points
Since for a top-point both L=0 and det[H]=Lxx Lyy-Lxy
2=0, we can find them by intersecting the paths with the level surface Det[H]=0
![Page 19: Ter Haar Romeny, ICPR 2010 Introduction to Scale-Space and Deep Structure.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551b1e6e550346f70d8b65d0/html5/thumbnails/19.jpg)
ter Haar Romeny, ICPR 2010
![Page 20: Ter Haar Romeny, ICPR 2010 Introduction to Scale-Space and Deep Structure.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551b1e6e550346f70d8b65d0/html5/thumbnails/20.jpg)
ter Haar Romeny, ICPR 2010
Original image
Top-points and features
Reconstruction
![Page 21: Ter Haar Romeny, ICPR 2010 Introduction to Scale-Space and Deep Structure.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551b1e6e550346f70d8b65d0/html5/thumbnails/21.jpg)
ter Haar Romeny, ICPR 2010
Metameric class
OriginalBy adjusting boundary and smoothness constraints we can improve the visual performance.
For this 300x300 picture 1000 top-points with 6 features were used.
![Page 22: Ter Haar Romeny, ICPR 2010 Introduction to Scale-Space and Deep Structure.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551b1e6e550346f70d8b65d0/html5/thumbnails/22.jpg)
ter Haar Romeny, ICPR 2010
Localization of Top-Points
For points close to top-points it is
possible to calculate a vector pointing
towards the position of the top-point.
x
y
Approximated Top-PointsDisplacement VectorsReal Locations
![Page 23: Ter Haar Romeny, ICPR 2010 Introduction to Scale-Space and Deep Structure.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551b1e6e550346f70d8b65d0/html5/thumbnails/23.jpg)
ter Haar Romeny, ICPR 2010
Stability of Top-Points
We can calculate the variance
of the displacement of top-
points under noise.
We need 4th order derivatives
in the top-points for that.
![Page 24: Ter Haar Romeny, ICPR 2010 Introduction to Scale-Space and Deep Structure.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551b1e6e550346f70d8b65d0/html5/thumbnails/24.jpg)
ter Haar Romeny, ICPR 2010
![Page 25: Ter Haar Romeny, ICPR 2010 Introduction to Scale-Space and Deep Structure.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551b1e6e550346f70d8b65d0/html5/thumbnails/25.jpg)
ter Haar Romeny, ICPR 2010
Thresholding on stability
Stable Paths Unstable Paths
![Page 26: Ter Haar Romeny, ICPR 2010 Introduction to Scale-Space and Deep Structure.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551b1e6e550346f70d8b65d0/html5/thumbnails/26.jpg)
ter Haar Romeny, ICPR 2010
Invariance of top-points
Top-points are invariant to certain transformations.
By invariant we mean that they move according to the
transformation.
Allo
wed
Tra
ns.
![Page 27: Ter Haar Romeny, ICPR 2010 Introduction to Scale-Space and Deep Structure.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551b1e6e550346f70d8b65d0/html5/thumbnails/27.jpg)
ter Haar Romeny, ICPR 2010
Differential invariants
We use the complete set
of irreducible 3rd order
differential invariants.
These features are
rotation and scaling
invariant.
![Page 28: Ter Haar Romeny, ICPR 2010 Introduction to Scale-Space and Deep Structure.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551b1e6e550346f70d8b65d0/html5/thumbnails/28.jpg)
ter Haar Romeny, ICPR 2010
The task
We have a scene and from
that scene we want to
retrieve the location of the
query object.
![Page 29: Ter Haar Romeny, ICPR 2010 Introduction to Scale-Space and Deep Structure.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551b1e6e550346f70d8b65d0/html5/thumbnails/29.jpg)
ter Haar Romeny, ICPR 2010
The top-points and differential invariants are
calculated for the query object and the scene.
![Page 30: Ter Haar Romeny, ICPR 2010 Introduction to Scale-Space and Deep Structure.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551b1e6e550346f70d8b65d0/html5/thumbnails/30.jpg)
ter Haar Romeny, ICPR 2010
Distance between feature vectors
A sensible distance between feature vectors is essential.
We have used Euclidean distance on ‘normalized’
differential invariants.
We tried Mahalanobis distance obtained from a training
set.
![Page 31: Ter Haar Romeny, ICPR 2010 Introduction to Scale-Space and Deep Structure.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551b1e6e550346f70d8b65d0/html5/thumbnails/31.jpg)
ter Haar Romeny, ICPR 2010
Similarity measure
We can calculate the propagation of noise in scale
space*.
This enables us to calculate a covariance matrix for
each feature vector.
The dissimilarity (“distance”) measure is expressed
as:
*Topological and Geometrical Aspects of Image Structure, Johan Blom
![Page 32: Ter Haar Romeny, ICPR 2010 Introduction to Scale-Space and Deep Structure.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551b1e6e550346f70d8b65d0/html5/thumbnails/32.jpg)
ter Haar Romeny, ICPR 2010
We now compare the
differential invariant features.
compare
distance = 0.5distance = 0.2distance = 0.3
![Page 33: Ter Haar Romeny, ICPR 2010 Introduction to Scale-Space and Deep Structure.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551b1e6e550346f70d8b65d0/html5/thumbnails/33.jpg)
ter Haar Romeny, ICPR 2010
The vectors with
the smallest
distance are paired.
smallest distance
distance = 0.2
![Page 34: Ter Haar Romeny, ICPR 2010 Introduction to Scale-Space and Deep Structure.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551b1e6e550346f70d8b65d0/html5/thumbnails/34.jpg)
ter Haar Romeny, ICPR 2010
A set of coordinates is
formed from the
differences in scale
(Log(o1)- Log(s2)) and in
angles (o1- s2).
(1, 1)
![Page 35: Ter Haar Romeny, ICPR 2010 Introduction to Scale-Space and Deep Structure.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551b1e6e550346f70d8b65d0/html5/thumbnails/35.jpg)
ter Haar Romeny, ICPR 2010
Dq
Important Clusters
For these clusters we calculate the mean and
Clustering (,)
If these coordinates are plotted
in a scatter plot clusters can be
identified.
• In this scatter plot we find two dense clusters
![Page 36: Ter Haar Romeny, ICPR 2010 Introduction to Scale-Space and Deep Structure.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551b1e6e550346f70d8b65d0/html5/thumbnails/36.jpg)
ter Haar Romeny, ICPR 2010
The stability criterion removes much of the scatter:
![Page 37: Ter Haar Romeny, ICPR 2010 Introduction to Scale-Space and Deep Structure.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551b1e6e550346f70d8b65d0/html5/thumbnails/37.jpg)
ter Haar Romeny, ICPR 2010
Rotate and scale according to the cluster means.
![Page 38: Ter Haar Romeny, ICPR 2010 Introduction to Scale-Space and Deep Structure.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551b1e6e550346f70d8b65d0/html5/thumbnails/38.jpg)
ter Haar Romeny, ICPR 2010
The translations we find correspond to the location of the objects in the scene.
![Page 39: Ter Haar Romeny, ICPR 2010 Introduction to Scale-Space and Deep Structure.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551b1e6e550346f70d8b65d0/html5/thumbnails/39.jpg)
ter Haar Romeny, ICPR 2010
In this example we have two clusters of correctly matched points.
C1
C2
![Page 40: Ter Haar Romeny, ICPR 2010 Introduction to Scale-Space and Deep Structure.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551b1e6e550346f70d8b65d0/html5/thumbnails/40.jpg)
ter Haar Romeny, ICPR 2010
The transformation of each object in the scene
matching to the query object is known from the
clustering.
![Page 41: Ter Haar Romeny, ICPR 2010 Introduction to Scale-Space and Deep Structure.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551b1e6e550346f70d8b65d0/html5/thumbnails/41.jpg)
ter Haar Romeny, ICPR 2010
We can transform the outline of the query object and project it on the scene image.
![Page 42: Ter Haar Romeny, ICPR 2010 Introduction to Scale-Space and Deep Structure.](https://reader036.fdocuments.us/reader036/viewer/2022062404/551b1e6e550346f70d8b65d0/html5/thumbnails/42.jpg)
ter Haar Romeny, ICPR 2010
VideoGoogle?