Post on 23-Jan-2017
IntroductionRelated WorkMethodology
Results and Discussion
Global Bilateral Symmetry Detection UsingMultiscale Mirror Histograms
M. ELAWADY1, C. BARAT1, C. DUCOTTET1 and P. COLANTONI2
1Universite de Lyon, CNRS, UMR 5516, Laboratoire Hubert Curien,Universite de Saint-Etienne, Jean-Monnet, F-42000 Saint-Etienne, France
2Universite Jean Monnet, CIEREC EA n0 3068, Saint-Etienne, France
ACIVS Conference, October 2016
UMR • CNRS • 5516 • SAINT-ETIENNE
M. ELAWADY, C. BARAT, C. DUCOTTET and P. COLANTONI Image Analysis and Understanding (Hubert Curien Lab, FR) 1 / 31
IntroductionRelated WorkMethodology
Results and Discussion
Table of Contents
1 IntroductionBackgroundApplicationsProblem Definition
2 Related WorkIntensity-based MethodsEdge-based Methods
3 MethodologyMotivationAlgorithm Details
4 Results and Discussion
M. ELAWADY, C. BARAT, C. DUCOTTET and P. COLANTONI Image Analysis and Understanding (Hubert Curien Lab, FR) 2 / 31
IntroductionRelated WorkMethodology
Results and Discussion
BackgroundApplicationsProblem Definition
Table of Contents
1 IntroductionBackgroundApplicationsProblem Definition
2 Related WorkIntensity-based MethodsEdge-based Methods
3 MethodologyMotivationAlgorithm Details
4 Results and Discussion
M. ELAWADY, C. BARAT, C. DUCOTTET and P. COLANTONI Image Analysis and Understanding (Hubert Curien Lab, FR) 3 / 31
IntroductionRelated WorkMethodology
Results and Discussion
BackgroundApplicationsProblem Definition
Bilateral Symmetry
1Image from book: The Photographer’s Eye by Michael Freeman
M. ELAWADY, C. BARAT, C. DUCOTTET and P. COLANTONI Image Analysis and Understanding (Hubert Curien Lab, FR) 4 / 31
IntroductionRelated WorkMethodology
Results and Discussion
BackgroundApplicationsProblem Definition
Bilateral Symmetry in Computer Vision I
Medial Image Compression [1]
Depth Estimation [2]
M. ELAWADY, C. BARAT, C. DUCOTTET and P. COLANTONI Image Analysis and Understanding (Hubert Curien Lab, FR) 5 / 31
IntroductionRelated WorkMethodology
Results and Discussion
BackgroundApplicationsProblem Definition
Bilateral Symmetry in Computer Vision II
Object Segmentation [3]
Aesthetic Analysis [4]
M. ELAWADY, C. BARAT, C. DUCOTTET and P. COLANTONI Image Analysis and Understanding (Hubert Curien Lab, FR) 6 / 31
IntroductionRelated WorkMethodology
Results and Discussion
BackgroundApplicationsProblem Definition
Detection of Main Symmetry Axis
Axis Legend: Strong, Weak
M. ELAWADY, C. BARAT, C. DUCOTTET and P. COLANTONI Image Analysis and Understanding (Hubert Curien Lab, FR) 7 / 31
IntroductionRelated WorkMethodology
Results and Discussion
Intensity-based MethodsEdge-based Methods
Table of Contents
1 IntroductionBackgroundApplicationsProblem Definition
2 Related WorkIntensity-based MethodsEdge-based Methods
3 MethodologyMotivationAlgorithm Details
4 Results and Discussion
M. ELAWADY, C. BARAT, C. DUCOTTET and P. COLANTONI Image Analysis and Understanding (Hubert Curien Lab, FR) 8 / 31
IntroductionRelated WorkMethodology
Results and Discussion
Intensity-based MethodsEdge-based Methods
Baseline and its Successors I
The general scheme (Loy and Eklundh 2006 [5]) consists of:
Example:
1Second figure from [5]
M. ELAWADY, C. BARAT, C. DUCOTTET and P. COLANTONI Image Analysis and Understanding (Hubert Curien Lab, FR) 9 / 31
IntroductionRelated WorkMethodology
Results and Discussion
Intensity-based MethodsEdge-based Methods
Baseline and its Successors II
Disadvantages:
Depending mainly on the properties of hand-crafted features (i.e. SIFT).
For example: (smooth objects with noisy background)little feature points =⇒ lost symmetry.
(Mo and Draper 2011 [6]) proposed refinements in the general scheme in:
1 Selecting all symmetry candidate pairs instead of finding closest matchesfor each point.
2 Using less complex hough voting scheme.
M. ELAWADY, C. BARAT, C. DUCOTTET and P. COLANTONI Image Analysis and Understanding (Hubert Curien Lab, FR) 10 / 31
IntroductionRelated WorkMethodology
Results and Discussion
Intensity-based MethodsEdge-based Methods
State of Art
Instead of SIFT, the general idea (Cicconet et al. 2014 [7]) is extracting aregular set of wavelet segments with local edge amplitude and orientation.
Disadvantages:
Lacking neighborhood’s information inside the feature representation.
Depending on the scale parameter of the edge detector.
For example: (high texture objects with noisy background)inferior symmetrical info =⇒ incorrect symmetry.
M. ELAWADY, C. BARAT, C. DUCOTTET and P. COLANTONI Image Analysis and Understanding (Hubert Curien Lab, FR) 11 / 31
IntroductionRelated WorkMethodology
Results and Discussion
MotivationAlgorithm Details
Table of Contents
1 IntroductionBackgroundApplicationsProblem Definition
2 Related WorkIntensity-based MethodsEdge-based Methods
3 MethodologyMotivationAlgorithm Details
4 Results and Discussion
M. ELAWADY, C. BARAT, C. DUCOTTET and P. COLANTONI Image Analysis and Understanding (Hubert Curien Lab, FR) 12 / 31
IntroductionRelated WorkMethodology
Results and Discussion
MotivationAlgorithm Details
Proposed Idea
Investigating Cicconet’s edge features [7] within Loy’s scheme [5] byadding neighboring-pixel information.
Contributions:
1 Introducing a new local edge descriptor.
2 Using multiscale edge extraction exploiting the full resolution image.
3 Solving the orientation discontinuity problem in the voting space.
4 Introducing a symmetry dataset based on aesthetic analysis.
M. ELAWADY, C. BARAT, C. DUCOTTET and P. COLANTONI Image Analysis and Understanding (Hubert Curien Lab, FR) 13 / 31
IntroductionRelated WorkMethodology
Results and Discussion
MotivationAlgorithm Details
Symmetry Detection Algorithm
Main Steps:
(1) Mul�scale Edge Segment Extrac�on
(2) Triangula�on based on Local Symmetry Weights:
• Geometry Edge Orienta�ons (Cic)• Local Texture Histogram (Loy)
(3) Vo�ng Space for Peak Detec�on with Handling Orienta�on Discon�nuity.
θ
ρ0
π
M. ELAWADY, C. BARAT, C. DUCOTTET and P. COLANTONI Image Analysis and Understanding (Hubert Curien Lab, FR) 14 / 31
IntroductionRelated WorkMethodology
Results and Discussion
MotivationAlgorithm Details
Multiscale Edge Segment Extraction I
A feature point p and its local edge characteristics (Jp, τp) are extractedwithin each cell using a Morlet wavelet ψk,σ of constant scale σ andvarying orientation {τk , k = 1 . . . n}.
M. ELAWADY, C. BARAT, C. DUCOTTET and P. COLANTONI Image Analysis and Understanding (Hubert Curien Lab, FR) 15 / 31
IntroductionRelated WorkMethodology
Results and Discussion
MotivationAlgorithm Details
Multiscale Edge Segment Extraction II
Jk(p) denote the modulus of wavelet coefficients at point p, in whichlocal edge characteristics Jp and τp are obtained by seeking the maximumwavelet response and orientation over all orientations.
M. ELAWADY, C. BARAT, C. DUCOTTET and P. COLANTONI Image Analysis and Understanding (Hubert Curien Lab, FR) 16 / 31
IntroductionRelated WorkMethodology
Results and Discussion
MotivationAlgorithm Details
Multiscale Edge Segment Extraction III
Histogram count at a given orientation τk is:
hp(k) =∑
r∈N(p)
Jrδφk−φr (1)
where φk and φr are angles associated with τk and τr , and δx is theKronecker delta. hp is subsequently `1 normalized and circular shifted soas the first bin corresponds to τp.
0 36 72 108 1440
0.5
1
1.5
2
2.5
3
3.5
4
4.5#106
Magnitude Histogram
108 144 0 36 720
0.1
0.2
0.3
0.4
0.5
0.6
Histogram Count (hp)
0 36 72 108 1440
500
1000
1500
2000
2500
3000
Frequency Histogram
M. ELAWADY, C. BARAT, C. DUCOTTET and P. COLANTONI Image Analysis and Understanding (Hubert Curien Lab, FR) 17 / 31
IntroductionRelated WorkMethodology
Results and Discussion
MotivationAlgorithm Details
Multiscale Edge Segment Extraction IV
In most images, relevant information about the visual content mayappear at different scales. Feature points are computed with respect to aset of regular grids at different scales and a corresponding set of waveletscales {σl , l = 1..m}.
M. ELAWADY, C. BARAT, C. DUCOTTET and P. COLANTONI Image Analysis and Understanding (Hubert Curien Lab, FR) 18 / 31
IntroductionRelated WorkMethodology
Results and Discussion
MotivationAlgorithm Details
Triangulation: Local Texture Histogram
(Textural Information) Symmetry degree of the two regions around p andq can be measured by comparing their corresponding local orientationhistogram hp and hq. Texture-based symmetry measure is given by:
dI (hp, h∗q) =
n∑k=1
min(hp(k), h∗q(k)) (2)
108 144 0 36 720
0.1
0.2
0.3
0.4
0.5
0.6
Histogram Count (hp)
72 36 0 144 1080
0.1
0.2
0.3
0.4
0.5
0.6
Histogram Count (hq*)
1 2 3 4 50
0.1
0.2
0.3
0.4
0.5
0.6
Histogram Intersection
M. ELAWADY, C. BARAT, C. DUCOTTET and P. COLANTONI Image Analysis and Understanding (Hubert Curien Lab, FR) 19 / 31
IntroductionRelated WorkMethodology
Results and Discussion
MotivationAlgorithm Details
Triangulation: Geometry Edge Orientation
(Edge Information) Pairwise symmetry coefficient f (p, q) is defined as [7]:
f (p, q) = |τqS(T⊥pq)τp| (3)
M. ELAWADY, C. BARAT, C. DUCOTTET and P. COLANTONI Image Analysis and Understanding (Hubert Curien Lab, FR) 20 / 31
IntroductionRelated WorkMethodology
Results and Discussion
MotivationAlgorithm Details
Triangulation: Symmetry Weights
Given a pair of feature points (p, q), the candidate axis T⊥pq perpendicularto (pq) is parametrized by the orientation of its normal θpq and itsdistance to the origin ρpq.
Mirror symmetry histogram HS(ρ, θ) is defined as the sum of thecontribution of all pairs of feature points such as:
H(cx , cy , θ) =∑p,qp 6=q
JpJqf (p, q)dI (hp, hq)δ(cx ,cy )− p+q2δθ−θpq (4)
HS(ρ, θ) =∑cx ,cy
H(cx , cy , θ)δρ−ρpq (5)
M. ELAWADY, C. BARAT, C. DUCOTTET and P. COLANTONI Image Analysis and Understanding (Hubert Curien Lab, FR) 21 / 31
IntroductionRelated WorkMethodology
Results and Discussion
MotivationAlgorithm Details
Voting Space and Peak DetectionA1 A2 A3 A4 A5
B
A1 A2 A3 A4 A5
B
BA1 A2 A3 A4 A5
M. ELAWADY, C. BARAT, C. DUCOTTET and P. COLANTONI Image Analysis and Understanding (Hubert Curien Lab, FR) 22 / 31
IntroductionRelated WorkMethodology
Results and Discussion
Table of Contents
1 IntroductionBackgroundApplicationsProblem Definition
2 Related WorkIntensity-based MethodsEdge-based Methods
3 MethodologyMotivationAlgorithm Details
4 Results and Discussion
M. ELAWADY, C. BARAT, C. DUCOTTET and P. COLANTONI Image Analysis and Understanding (Hubert Curien Lab, FR) 23 / 31
IntroductionRelated WorkMethodology
Results and Discussion
Algorithm Evaluation
From Real-World Images Competition CVPR 2013 [10], a symmetrydetection is correct if: (1) θ < 15◦ and (2) d < 0.2 ∗min(lenGT , lenR).
RGT d
θ
M. ELAWADY, C. BARAT, C. DUCOTTET and P. COLANTONI Image Analysis and Understanding (Hubert Curien Lab, FR) 24 / 31
IntroductionRelated WorkMethodology
Results and Discussion
Quantitative Results��������uD(t
M
CM
iM
cM
OM
uM
rM
�M
M
M
CMM
���� �� �� iMCM �� iMCC �� iMCc ���iMCr
���iMMr
��iMCC
���iMCO
���iMCr
M. ELAWADY, C. BARAT, C. DUCOTTET and P. COLANTONI Image Analysis and Understanding (Hubert Curien Lab, FR) 25 / 31
IntroductionRelated WorkMethodology
Results and Discussion
Qualitative Results on PSU Datasets
(http://www.flickr.com/), around 200 images from PSU symmetrydetection challenges [9, 10] in ECCV2010, CVPR2011 and CVPR2013.
Legend: Groundtruth, Our2016, Loy2006, Mo2011, Cic2014
M. ELAWADY, C. BARAT, C. DUCOTTET and P. COLANTONI Image Analysis and Understanding (Hubert Curien Lab, FR) 26 / 31
IntroductionRelated WorkMethodology
Results and Discussion
Qualitative Results on AVA Dataset
(http://www.dpchallenge.com/), around 250 images from AestheticVisual Analysis “AVA” [8] with our global-axis symmetry groundtruth.
Legend: Groundtruth, Our2016, Loy2006, Mo2011, Cic2014
M. ELAWADY, C. BARAT, C. DUCOTTET and P. COLANTONI Image Analysis and Understanding (Hubert Curien Lab, FR) 27 / 31
IntroductionRelated WorkMethodology
Results and Discussion
Conclusion
Summary:1 A reliable global symmetry detection is developed among variants of visual cues.
2 A groundtruth of global symmetry axis is introduced and extracted from largescale Aesthetic Visual Analysis (AVA) dataset.
Future work:1 Real-world images is required to handle with large degrees of perspective view.
2 The proposed detection can be improved to avoid over-extended axes.
3 A stable balance measure can be introduced to describe the existence and degreeof global axes inside an image.
4 Possibility of integration within retrieval systems of visual arts.
M. ELAWADY, C. BARAT, C. DUCOTTET and P. COLANTONI Image Analysis and Understanding (Hubert Curien Lab, FR) 28 / 31
IntroductionRelated WorkMethodology
Results and Discussion
References I
[1] V. Bairagi, “Symmetry-based biomedical image compression,” Journal of digitalimaging, pp. 1–9, 2015.
[2] L. Yang, J. Liu, and X. Tang, “Depth from water reflection,” Image Processing,IEEE Transactions on, vol. 24, no. 4, pp. 1235–1243, 2015.
[3] C. L. Teo, C. Fermuller, and Y. Aloimonos, “Detection and segmentation of 2dcurved reflection symmetric structures,” in Proceedings of the IEEE InternationalConference on Computer Vision, pp. 1644–1652, 2015.
[4] S. Zhao, Y. Gao, X. Jiang, H. Yao, T.-S. Chua, and X. Sun, “Exploringprinciples-of-art features for image emotion recognition,” in Proceedings of theACM International Conference on Multimedia, pp. 47–56, ACM, 2014.
[5] G. Loy and J.-O. Eklundh, “Detecting symmetry and symmetric constellations offeatures,” in Computer Vision–ECCV 2006, pp. 508–521, Springer, 2006.
[6] Q. Mo and B. Draper, “Detecting bilateral symmetry with feature mirroring,” inCVPR 2011 Workshop on Symmetry Detection from Real World Images, 2011.
M. ELAWADY, C. BARAT, C. DUCOTTET and P. COLANTONI Image Analysis and Understanding (Hubert Curien Lab, FR) 29 / 31
IntroductionRelated WorkMethodology
Results and Discussion
References II
[7] M. Cicconet, D. Geiger, K. C. Gunsalus, and M. Werman, “Mirror symmetryhistograms for capturing geometric properties in images,” in Computer Visionand Pattern Recognition (CVPR), 2014 IEEE Conference on, pp. 2981–2986,IEEE, 2014.
[8] N. Murray, L. Marchesotti, and F. Perronnin, “Ava: A large-scale database foraesthetic visual analysis,” in Computer Vision and Pattern Recognition (CVPR),2012 IEEE Conference on, pp. 2408–2415, IEEE, 2012.
[9] I. Rauschert, K. Brocklehurst, S. Kashyap, J. Liu, and Y. Liu, “First symmetrydetection competition: Summary and results,” tech. rep., Technical ReportCSE11-012, Department of Computer Science and Engineering, ThePennsylvania State University, 2011.
[10] J. Liu, G. Slota, G. Zheng, Z. Wu, M. Park, S. Lee, I. Rauschert, and Y. Liu,“Symmetry detection from realworld images competition 2013: Summary andresults,” in Computer Vision and Pattern Recognition Workshops (CVPRW),2013 IEEE Conference on, pp. 200–205, IEEE, 2013.
M. ELAWADY, C. BARAT, C. DUCOTTET and P. COLANTONI Image Analysis and Understanding (Hubert Curien Lab, FR) 30 / 31
IntroductionRelated WorkMethodology
Results and Discussion
Questions?
M. ELAWADY, C. BARAT, C. DUCOTTET and P. COLANTONI Image Analysis and Understanding (Hubert Curien Lab, FR) 31 / 31