ERLKG: Entity Representation Learning and Knowledge Graph ...
Shape Representation Shape Representation - what for? 4 4 To reason about an entity we must first...
-
date post
21-Dec-2015 -
Category
Documents
-
view
219 -
download
3
Transcript of Shape Representation Shape Representation - what for? 4 4 To reason about an entity we must first...
Shape Shape RepresentationRepresentation
Shape Representation - what for?Shape Representation - what for?
To reason about an entity we must first represent the entity.
• Real world
Image processing
• Shape representation
•Image understanding \ shape recognition
011001011001000100010001000000100111
00101001100111101
Shape Representation - what for?Shape Representation - what for?
TopicsTopics
DefinitionsAttributesPopular Strategies
TopicsTopics
DefinitionsDefinitionsObject Abstraction Representation
ObjectObject
An object is something that
can be seen or touched,
material thing
(Oxford dictionary)
AbstractionAbstraction
Idea of quality separate from
actual examples
(Oxford dictionary)
• ball
• sphere
RepresentationRepresentation
A way of symbolizing an object
2222 zyxr
Attributes Of A Good Attributes Of A Good RepresentationRepresentation
sufficient wide domain unique unambiguous generative stable convenient All will be explained in next All will be explained in next
slidesslides
SufficientSufficient
...Is this representation sufficient enough? Depends on the application
Wide DomainWide Domain
Able to represent many different classes of entities E.g. numbers for elements in a queue
123
UniqueUnique every distinct member of its domain has a
single distinct representation.
Not unique : dog dog dog
unique : pitbull collie cocker-spaniel
UnambiguousUnambiguous
An entity may have different representations but no two distinct entities may have a common representation.
3 three III 2 two II
GenerativeGenerative Capable of directly generating
(recovering) the represented entity
Chain-code :436476872832
12
3
45
6
7
8
StableStable small perturbations do not induce large changes
in the representation of the entity.
ConvenientConvenient A representation may exhibit all the characteristics
that we have discussed so far and yet not be convenient for a task.– e.g. an assembly line robotassembly line robot who’s task is to filter out
rectangels from other shapes might use chaine code representation.
Yet for a police computerized camera that is supposed to compare faces in the crowd to a data base of suspects this is not enough.
Popular Strategies for shape Popular Strategies for shape encodingencoding
Volume based :– Describe the object volumetrically
– Combination of primitive volumes commonly used, eg. cubes, tetrahedra, and discs
– Provide more access to global relationships
– Do not provide direct surface information about the object
– Can represent only closed (boundaryless) surfaces
e.g.
Symmetric Axis Transform
Popular Strategies for shape Popular Strategies for shape encodingencoding
Surface based :– Surface of object represented by a single closed
parametric grid.
– Better suited for partial surfaces.
E.g.
Parametric Bicubic Patches
Gaussian- Image Representations
Popular StrategiesPopular Strategies Others:
– Chain code.– Distance vs. Angle.– Fourier Transform Moment– Generalized Cylinders – Visual Potential
We will discuss these We will discuss these strategiesstrategies
Chain CodeChain Code
Chain CodeChain Code
This example shows that the Chain Code is This example shows that the Chain Code is independent of Location, Starting Point and independent of Location, Starting Point and
orientationorientation
AttributesAttributes
1. wide domain
2. unique
3. unambiguous
4. generative - 2D only
5. stable - depends on tolerance
Distance vs. AngleDistance vs. Angle
Distance vs. AngleDistance vs. Angle
1. Find point of balance.
2. Measure distance to edges.
3. Plot the graph of distance vs. angle.
Taking the dist. vs. angle one step further we get ...
FourierFourier DescriptorDescriptor
AttributesAttributes
1. wide domain
2. unique
3. unambiguous
4. generative
5. stable - depends upon tolerance
MomentsMoments 1. find center of mass
2. find the moment:
Feature extraction takes segmented image data and outputs a list of features which are combined to give the feature vector v.– Area of a binary object can be calculated by simply adding up the
pixel values in the neighbourhood.– Perimeter obtained by subtracting eroded object from original and
adding up the resulting pixel values.– Compactness is a basic roundness measurement. Circle has low
compactness.– Correlation is the convolution of an image with a reference shape.
Zero-and first-order moments give area and centroid.
Conclusion: Conclusion: How to calculate area How to calculate area and centroid position using moments.and centroid position using moments.
What are Moments?What are Moments?1. Used to calculate:
1. area,
2. position,
3. other more complex shape features such as elongation.
2. Provides a general framework.
3. For an image A, where each pixel brightness is denoted by Ai,j, the moment of order k+l is given by:
i j
jilk
kl Ajim ,
Area = Area = mm0000
i jji
i jji
i jji
i jji
lkkl
Am
AAjim
ji
lk
Ajim
,00
,,00
00
00
,
11
1,1
0,0
i j
jiAArea ,
Object PositionObject PositionCalculate the object position
for a specific example.
Using the same process show how you
can generate a general expression
for object position.
Show that this general expression is equal
to a combination of zero- and first-order moments.
j 1 2 3 4 5 6 7j 1 2 3 4 5 6 7i i 11 22 33 44 55 66 77
Determine pixel at center of pattern. Its position gives the object position.
Called the centroid,– denoted by (ic,jc).
ic is the average i value of all white pixels.
jc is the average j value of all white pixels.
Object PositionObject Position
How do we calculate How do we calculate jjcc??
j 1 2 3 4 5 6 7j 1 2 3 4 5 6 7i i 11 22 33 44 55 66 77
j 1 2 3 4 5 6 7j 1 2 3 4 5 6 7
Generate 1-d plot of the number of Generate 1-d plot of the number of pixels with a particular value of j versus jpixels with a particular value of j versus j
PositionPosition
j 1 2 3 4 5 6 7j 1 2 3 4 5 6 7
49.3
17/66172524124
55644322
c
c
c
j
j
Areaj
jc = sum of j values
over all white pixels divided by
the total number of white pixels Total number of white pixels =Area
Position Position jjcc
i jji
i jji
i jji
j ijic
ii
ii
i iii
ii
ii
iic
AAjAAjj
AreaAAAAAAAj
,,,,
7,6,5,4,3,2,1, 7654321
j 1 2 3 4 5 6 7j 1 2 3 4 5 6 7
i
jiA ,
Centroid PositionCentroid Position Centroid position, (ic,jc) can
be calculated using zero-, m00,
and first-order, m10 and m01, moments.
jc = m01 / m00
i jji
i jji
c
i jji
i jji
i jji
i jji
lkkl
A
Aj
j
AjAjim
lk
Am
Ajim
,
,
,,10
01
,00
,
1,0
How do we calculate How do we calculate iicc??
j 1 2 3 4 5 6 7j 1 2 3 4 5 6 7 i i 11 22 33 44 55 66 77
i i 11 22 33 44 55 66 77
PositionPosition
43.4
17/731714121516124
272635444322
c
c
c
i
i
Areai
ic = sum of i values over all pixels divided by the
total number of pixels Total number of pixels
=Areai 1 2 3 4 5 6 7i 1 2 3 4 5 6 7
Position Position iicc
i jji
i jji
i jji
i jjic
j jj
jj
jjj
jj
jj
jjc
AAiAAii
AreaAAAAAAAi
,,,,
,7,6,5,4,3,2,1 7654321
i 1 2 3 4 5 6 7i 1 2 3 4 5 6 7
j
jiA ,
Centroid PositionCentroid Position
Centroid position, (ic,jc) can be calculated using zero-, m00,
and first-order, m10 and m01, moments.
ic = m10 / m00
i jji
i jji
c
i jji
i jji
i jji
i jji
lkkl
A
Ai
i
AiAjim
lk
Am
Ajim
,
,
,,01
10
,00
,
0,1
Centroid PositionCentroid Position (ic,jc)=(4,4)
j 1 2 3 4 5 6 7j 1 2 3 4 5 6 7i i 11 22 33 44 55 66 77
XX
2-2-D Invariant D Invariant DescriptorsDescriptors
Why Invariant Descriptors?Why Invariant Descriptors? Shape-Based Image Retrieval 2-D Invariant Descriptors
– Invariant to translation, rotation, scale change
Moment invariants (region-based measure)– Hu's moment invariants [Hu‘61]
• M.-K. Hu, Pattern recognition by moment invariants, Proc. IRE, vol. 49, p.1428, Sept. 1961
– Taubin's moment invariants [Taubin‘92]– Flusser's moment invariants [Flusser‘93]– Zernike moments [Taegu‘80]
• M. R. Taegue, Image analysis via the general theory of moments, J. Opt. Soc, Amer., vol. 70, pp. 920-930, Aug. 1980
Fourier descriptors [Zhan‘72] (boundary-based measure)
• C. T. Zhan, R. S. Roskies, Fourier descriptors for plane closed curves, IEEE Trans. Comput., vol. C-21, 1972, pp. 269-281
What Kind of?What Kind of?
1,...,0))},(),({( Niiyix
1,...,0),()()( Niijyixiz
1
0
)2
exp()()(N
i N
kijizkZ
12
,...,2,1,...,2
},|)1(|
|)(|{
mmk
Z
kZ
Boundary pointsBoundary points
Considering in complex planeConsidering in complex plane
Fourier descriptorsFourier descriptors
NormalizationNormalization
Fourier descriptorsFourier descriptors
)8(),...,( 81 mFFF
What Kind of?What Kind of?
Syx
qppq yxSm
),(
)(
)(
)(,
)(
)(,)()()(
00
01
00
10
),( Sm
Smy
Sm
SmxyyxxS
Syx
qppq
,..3,2,12
,)(00
qpqp
S pqpq
Moment invariantsMoment invariants
Region pointsRegion points
MomentMoment
Central momentCentral moment
Normalized central momentNormalized central moment
}1),(|),{( yxfyxS
Various waysVarious ways
Moment InvariantsMoment Invariants
])()(3)[)(3(
])(3))[()(3(
))((4
])())[((
])())[()(3(
])(3))[()(3(
)()(
)3()3(
4)(
20321
2123003211230
20321
21230123003217
0321123011
20321
2123002206
20321
2123003210321
20321
21230123012305
20321
212304
20321
212303
211
202202
02201
H
H
H
H
H
H
H
M
M
M
M
M
M
M),...,( 71 HHH MMM
• HuHu[‘61] : 2nd-3rd order NCM -> 7 invariant moments[‘61] : 2nd-3rd order NCM -> 7 invariant moments
• TaubinTaubin & Cooper[‘92] : the concept of covariant matrix & Cooper[‘92] : the concept of covariant matrix
-> affine invariant moments-> affine invariant moments
)(/, 00'
'02
'11
'11
'20
11 SmM pqpq
ILLMll
lL T
11
2221
11 ,0
qp
Syxpq yylxxlxxl
Sm)]()([)]([
)(
12221
),(11
00
'
2
1,
2
1,, 21
'042
'131
'222
'131
'22
'311
'222
'312
'402
'22'
031'12
'211
'121
'21
'301'
12
cc
ccc
cc
ccc
Mcc
ccM
),...,( 81 TTT MMM
)(41
),(),(),( '04
'40122212221212 TT MMMMMM
CM of order 2 -> 2x2 matrixCM of order 2 -> 2x2 matrix
Lower triangular matrixLower triangular matrix
New momentsNew moments
3rd-4th order moments -> two matrices3rd-4th order moments -> two matrices
Moment invariantsMoment invariants
Moment InvariantsMoment Invariants
MomentMoment
MomentsMoments
since we get invariant values the moments are not affected by transition or rotation.
By normalizing the values, we can make the
moment indifferent to scaling
Attributes of momentsAttributes of moments
1. wide domain
2. unique - no
3. unambiguous - no
4. generative - no
5. stable - no
)2(
)34(
)61296
8186
12966(
)]()()([
3446(
)(
322
23104
21240122231220440
9006
22212310440
6005
230
3022130
20211123002
211
221
202201230
20220
033031112210211200330021120
032121120
21202
220032102
220031211
220
303
320
11004
2211230021221033011
212032120
7003
212
221
32130
3123003122130
203
230
10002
2110220
4001
F
F
F
F
F
F
M
M
M
M
M
M
),...,( 61 FFF MMM
• FlusserFlusser & Suk[‘93]: affine moment invariants & Suk[‘93]: affine moment invariants
• Taegue[‘80]: Taegue[‘80]: ZernikeZernike moments -> moment invariants moments -> moment invariants• Invariant under rotationInvariant under rotation
• Invariant under general 2-D affine transformationInvariant under general 2-D affine transformation
• Normalization -> invariant under translation and scalingNormalization -> invariant under translation and scaling• translate origin to centroidtranslate origin to centroid• scale so that max distance from centroid == 1scale so that max distance from centroid == 1• ignore first two momentsignore first two moments
2/|)|(
0
2
1,),(
*
!2/|)|(!2/|)|(!
)!()1()(
),exp()(),(,),(1
22
mn
s
snsnm
nmnmyxSyxnmnm
smnsmns
snR
jmRVVn
A
evenis||and,,...,1,0 mnnmnn
|| nmA
),...,( 101 ZZZ MMM
|||,| 1100 AA
|||,||,||,||,||,||,||,||,||,| 55535144424033312220 AAAAAAAAAA
)2(
)34(
)61296
8186
12966(
)]()()([
3446(
)(
322
23104
21240122231220440
9006
22212310440
6005
230
3022130
20211123002
211
221
202201230
20220
033031112210211200330021120
032121120
21202
220032102
220031211
220
303
320
11004
2211230021221033011
212032120
7003
212
221
32130
3123003122130
203
230
10002
2110220
4001
F
F
F
F
F
F
M
M
M
M
M
M
),...,( 61 FFF MMM
• FlusserFlusser & Suk[‘93]: affine moment invariants & Suk[‘93]: affine moment invariants
• Taegue[‘80]: Taegue[‘80]: ZernikeZernike moments -> moment invariants moments -> moment invariants• Invariant under rotationInvariant under rotation
• Invariant under general 2-D affine transformationInvariant under general 2-D affine transformation
• Normalization -> invariant under translation and scalingNormalization -> invariant under translation and scaling• translate origin to centroidtranslate origin to centroid• scale so that max distance from centroid == 1scale so that max distance from centroid == 1• ignore first two momentsignore first two moments
2/|)|(
0
2
1,),(
*
!2/|)|(!2/|)|(!
)!()1()(
),exp()(),(,),(1
22
mn
s
snsnm
nmnmyxSyxnmnm
smnsmns
snR
jmRVVn
A
evenis||and,,...,1,0 mnnmnn
|| nmA
),...,( 101 ZZZ MMM
|||,| 1100 AA
|||,||,||,||,||,||,||,||,||,| 55535144424033312220 AAAAAAAAAA
What’s the Best?What’s the Best? 44Size : 337 x 145Size : 337 x 145Rotation : -400,400,8Rotation : -400,400,8Scaling : -2,2,0.04Scaling : -2,2,0.04
Recent Solution.Recent Solution. Difficulty (Current Problem in Digital Image
Analysis)– Moment invariants are not sufficient for distinguishing– Sensitive to noise
Region-based measure + boundary-based measure– Gives the better average retrieval efficiency
Two-stage similarity scheme – Moment invariants (1st retrieval stage) -> n images– Fourier descriptors (2nd verification stage) -> m < n
images
Comparison of computation time. (unit=sec)--> Indigo2 IMPACT (MIPS R1000), 337 x 145, 718 boundary points, 17761 region points
Method FD HM TM FM ZM FH FZ
Time 0.01 0.02 0.02 0.02 0.93 0.03 0.94
• Comparison to other similarity schemes (Top5 Comparison to other similarity schemes (Top5 / Top 10)/ Top 10)--> 10 rabbit images / 85 animal images--> 10 rabbit images / 85 animal images
Query\method FD HM TM FM ZM FH FZquery 1 5/8 4/7 4/7 2/3 5/9 5/8 5/9query 2 5/8 5/8 4/5 2/5 5/9 5/8 5/9query 3 5/8 2/6 1/1 1/3 4/8 5/8 5/8query 4 5/8 5/7 4/6 1/3 5/9 5/8 5/9
FD=Fourier descriptorsFD=Fourier descriptors
HM=Hu's moment invariantsHM=Hu's moment invariants
TM=Taubin's moment invariantsTM=Taubin's moment invariants
FM=Flusser's moment invariantsFM=Flusser's moment invariants
ZM=Zernike momentsZM=Zernike moments
FH=FD+HMFH=FD+HM
FZ=FD+ZMFZ=FD+ZM
Recent Solution.Recent Solution.
QueryQuery
FH/FZ Top 5FH/FZ Top 5
FH/FZ Top 10FH/FZ Top 10 FZ Top 10FZ Top 10 MissMiss
FalseFalse
Database: 1195 animal imagesDatabase: 1195 animal imagesSadegh Abbasi, Farzin Mokhtarian, Josef Kittler, S. SclaroffSadegh Abbasi, Farzin Mokhtarian, Josef Kittler, S. Sclaroffhttp://www.ee.surrey.ac.uk/Research/VSSP/imagedb/demo.htmhttp://www.ee.surrey.ac.uk/Research/VSSP/imagedb/demo.htmll
Parametric Parametric Bicubic PatchesBicubic Patches
Split the object to simpler bicubic patches that can be represented by relatively simple mathematical equations.
Piecewise Parametric SurfacesPiecewise Parametric Surfaces
Piecewise PatchesPiecewise Patches
Bezier PatchesBezier Patches
Bezier PatchesBezier Patches
AttributesAttributes
1. generative
2. stable
3. popular in computer graphics
4. Possibility of several equally acceptable bicubic approximations to any given surface makes it inappropriate for surface matching
Symmetric Axis Symmetric Axis TransformTransform
• A.k.a Blum Transform, Medial Axis Transform.• Formally : the object is the logical union of all its maximal discs.
May be thought of as grass fire spreading from the border inwards.
Description : – Locus of centers a.k.a symmetric axis.Locus of centers a.k.a symmetric axis.– Radius at each pointRadius at each point
Symmetric Axis TransformSymmetric Axis Transform
AttributesAttributes
1. wide domain
2. unique
3. unambiguous
4. generative
5. not stable - small changes affect dramatically
6. Primarily used in biological applications
Generalized Generalized CylindersCylinders
Binford 1971 an extension of SAT. Shape represented by an ordinary cylinder sweeping the cross-section along
an arbitrary space curve (axis/spine)
Generalized CylindersGeneralized Cylinders
Decomposition of 3D shape-description problems into lower-order problems
Generalized CylindersGeneralized CylindersA generalized cylinder is thus defined by
a cross section, a cross section, an axisan axis a sweeping rule. a sweeping rule.
Generalized CylindersGeneralized Cylinders
Generalized CylindersGeneralized Cylinders
AttributesAttributes
1. wide domain - depends on implementation (redundancy vs. wide domain )
2. unique
3. generative
4. stability is doubtful
5. Decomposition of complex structures is difficult.
6. Used mainly for object recognition
7. been used in working systems.
Gaussian Gaussian ImageImage
Surface normal vector information for any object can be mapped onto a unit sphere, called the Gaussian sphere.
Mapping is called the Gaussian image of the object.
Extended Gaussian ImageExtended Gaussian Image
The mapping is: Surface normals for each point of the object are placed so that their
tails lie at the center of the Gaussian spheretails lie at the center of the Gaussian sphere
heads lie on a point on the sphere appropriate to heads lie on a point on the sphere appropriate to the particular surface orientationthe particular surface orientation
Extended Gaussian Image (EGI)Extended Gaussian Image (EGI)
We can extend this process so that a weight is assigned to each point on the Gaussian sphere
equal to the area of the surface having the given normal This mapping is called the extended Gaussian image (EGI). Weights are represented by vectors parallel to the surface
normals, with length equal to the weight
EGIEGIExtended Gaussian Image (EGI)Extended Gaussian Image (EGI)
AttributesAttributes
1. wide domain
2. not unique
3. not unambiguous
4. not generative
5. stable - with limitations
6. invariant to translation and scaling
7. position independence
Visual Visual PotentialPotential
Aspect - topological structure of the singularities in a single view.
A graph in which each node represents an aspect of the object, and each edge the possibility of transiting from one aspect to another under motion of the observer [Turner74].
Represents in a concise way any visual experience an observer can obtain by looking at the object when traversing any orbit through space.
Visual PotentialVisual Potential
AttributesAttributes
1. wide domain
2. unique
3. unambiguous
4. generative
5. not stable
SummarySummary
The above methods are deterministic; in practice, uncertainties in shape that result from the noise in measurement have to be considered [Ayache88, Ikeuchi88]
The above methods are based on geometric models, they are more appropriate for describing specific objects, particularly artificial ones with regular structures
ConclusionConclusion
The Proposed Hybrid Method– FD + Hu's/Zernike MI– FD + Hu's MI
Future Works– Invariant to 3-D perspective
transform– Recognition of partially
recovered/occluded objects
SummarySummary
Symbolic models are more appropriate for natural objects that are better defined in terms of generic characteristics (e.g.. small, red, rough) than precise shape; useful for matching
ReferencesReferences
Books :– Nalwa “a guided tour of computer vision”
– Gonzales, “Image analysis and computer vision”
– Jain
Web:– http://web.mit.edu/manoli/ecimorph/www/code/MMorph.html
– http://www.cs.ubc.ca/~clee/cg/proj/index.html
– http://www.dai.ed.ac.uk/CVonline/LOCAL_COPIES/MARSHALL/node53.html#figuresame_EGI
– http://www.cs.uwa.edu.au/~cheng/index.html
[email protected]@[email protected]@etri.re.krHoward Schultz, UMass Howard Schultz, UMass Yaron BerlinskyYaron Berlinsky
SourcesSources