Ellipse Fitting forComputer VisionImplementation and Applications

Synthesis Lectures onComputer Vision

EditorsGérardMedioni,University of Southern CaliforniaSvenDickinson,University of Toronto

Synthesis Lectures on Computer Vision is edited by Gérard Medioni of the University of SouthernCalifornia and Sven Dickinson of the University of Toronto. e series publishes 50- to 150 pagepublications on topics pertaining to computer vision and pattern recognition. e scope will largelyfollow the purview of premier computer science conferences, such as ICCV, CVPR, and ECCV.Potential topics include, but not are limited to:

• Applications and Case Studies for Computer Vision

• Color, Illumination, and Texture

• Computational Photography and Video

• Early and Biologically-inspired Vision

• Face and Gesture Analysis

• Illumination and Reflectance Modeling

• Image-Based Modeling

• Image and Video Retrieval

• Medical Image Analysis

• Motion and Tracking

• Object Detection, Recognition, and Categorization

• Segmentation and Grouping

• Sensors

• Shape-from-X

• Stereo and Structure from Motion

• Shape Representation and Matching

iii

• Statistical Methods and Learning

• Performance Evaluation

• Video Analysis and Event Recognition

Ellipse Fitting for Computer Vision: Implementation and ApplicationsKenichi Kanatani, Yasuyuki Sugaya, and Yasushi Kanazawa2016

Background Subtraction: eory and PracticeAhmed Elgammal2014

Vision-Based InteractionMatthew Turk and Gang Hua2013

Camera Networks: e Acquisition and Analysis of Videos over Wide AreasAmit K. Roy-Chowdhury and Bi Song2012

Deformable Surface 3D Reconstruction from Monocular ImagesMathieu Salzmann and Pascal Fua2010

Boosting-Based Face Detection and AdaptationCha Zhang and Zhengyou Zhang2010

Image-Based Modeling of Plants and TreesSing Bing Kang and Long Quan2009

Copyright © 2016 by Morgan & Claypool

All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted inany form or by any means—electronic, mechanical, photocopy, recording, or any other except for brief quotationsin printed reviews, without the prior permission of the publisher.

Ellipse Fitting for Computer Vision: Implementation and Applications

Kenichi Kanatani, Yasuyuki Sugaya, and Yasushi Kanazawa

www.morganclaypool.com

ISBN: 9781627054584 paperbackISBN: 9781627054980 ebook

DOI 10.2200/S00713ED1V01Y201603COV008

A Publication in the Morgan & Claypool Publishers seriesSYNTHESIS LECTURES ON COMPUTER VISION

Lecture #8Series Editors: Gérard Medioni, University of Southern California

Sven Dickinson, University of TorontoSeries ISSNPrint 2153-1056 Electronic 2153-1064

Ellipse Fitting forComputer VisionImplementation and Applications

Kenichi KanataniOkayama University, Okayama, Japan

Yasuyuki SugayaToyohashi University of Technology, Toyohashi, Aichi, Japan

Yasushi KanazawaToyohashi University of Technology, Toyohashi, Aichi, Japan

SYNTHESIS LECTURES ON COMPUTER VISION #8

CM&

cLaypoolMorgan publishers&

ABSTRACTBecause circular objects are projected to ellipses in images, ellipse fitting is a first step for 3-D analysis of circular objects in computer vision applications. For this reason, the study of el-lipse fitting began as soon as computers came into use for image analysis in the 1970s, but itis only recently that optimal computation techniques based on the statistical properties of noisewere established. ese include renormalization (1993), which was then improved as FNS (2000)and HEIV (2000). Later, further improvements, called hyperaccurate correction (2006), HyperLS(2009), and hyper-renormalization (2012), were presented. Today, these are regarded as the mostaccurate fitting methods among all known techniques. is book describes these algorithms aswell implementation details and applications to 3-D scene analysis.

We also present general mathematical theories of statistical optimization underlying all el-lipse fitting algorithms, including rigorous covariance and bias analyses and the theoretical accu-racy limit. e results can be directly applied to other computer vision tasks including computingfundamental matrices and homographies between images.

is book can serve not simply as a reference of ellipse fitting algorithmsfor researchers, but also as learning material for beginners who want to start com-puter vision research. e sample program codes are downloadable from the web-site: https://sites.google.com/a/morganclaypool.com/ellipse-fitting-for-computer-vision-implementation-and-applications/.

KEYWORDSgeometric distance minimization, hyperaccurate correction, HyperLS, hyper-renormalization, iterative reweight, KCR lower bound, maximum likelihood, renor-malization, robust fitting, Sampson error, statistical error analysis, Taubin method

vii

ContentsPreface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi

1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

1.1 Ellipse Fitting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 Representation of Ellipses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.3 Least Squares Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.4 Noise and Covariance Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41.5 Ellipse Fitting Approaches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61.6 Supplemental Note . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2 Algebraic Fitting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

2.1 Iterative Reweight and Least Squares . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112.2 Renormalization and the Taubin Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122.3 Hyper-renormalization and HyperLS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152.5 Supplemental Note . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

3 Geometric Fitting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

3.1 Geometric Distance and Sampson Error . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193.2 FNS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203.3 Geometric Distance Minimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213.4 Hyperaccurate Correction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233.5 Derivations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243.6 Supplemental Note . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

4 Robust Fitting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

4.1 Outlier Removal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 314.2 Ellipse-specific Fitting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 324.3 Supplemental Note . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

viii

5 Ellipse-based 3-DComputation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 375.1 Intersections of Ellipses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 375.2 Ellipse Centers, Tangents, and Perpendiculars . . . . . . . . . . . . . . . . . . . . . . . . . . 385.3 Perspective Projection and Camera Rotation . . . . . . . . . . . . . . . . . . . . . . . . . . . 405.4 3-D Reconstruction of the Supporting Plane . . . . . . . . . . . . . . . . . . . . . . . . . . . 435.5 Projected Center of Circle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 445.6 Front Image of the Circle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 455.7 Derivations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 475.8 Supplemental Note . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

6 Experiments and Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 556.1 Ellipse Fitting Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 556.2 Statistical Accuracy Comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 566.3 Real Image Examples 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 596.4 Robust Fitting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 596.5 Ellipse-specific Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 596.6 Real Image Examples 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 616.7 Ellipse-based 3-D Computation Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . 626.8 Supplemental Note . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

7 Extension andGeneralization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 677.1 Fundamental Matrix computation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67

7.1.1 Formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 677.1.2 Rank Constraint . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 707.1.3 Outlier Removal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71

7.2 Homography Computation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 727.2.1 Formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 727.2.2 Outlier Removal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76

7.3 Supplemental Note . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77

8 Accuracy of Algebraic Fitting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 798.1 Error Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 798.2 Covariance and Bias . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 808.3 Bias Elimination and Hyper-renormalization . . . . . . . . . . . . . . . . . . . . . . . . . . 828.4 Derivations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 838.5 Supplemental Note . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90

ix

9 MaximumLikelihood andGeometric Fitting . . . . . . . . . . . . . . . . . . . . . . . . . . . 939.1 Maximum Likelihood and Sampson Error . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 939.2 Error Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 949.3 Bias Analysis and Hyperaccurate Correction . . . . . . . . . . . . . . . . . . . . . . . . . . . 969.4 Derivations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 969.5 Supplemental Note . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101

10 eoretical Accuracy Limit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10310.1 KCR Lower Bound . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10310.2 Derivation of the KCR Lower Bound . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10410.3 Expression of the KCR Lower Bound . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10710.4 Supplemental Note . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108

Answers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111

Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119

Authors’ Biographies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125

Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127

xi

PrefaceBecause circular objects are projected to ellipses in images, ellipse fitting is a first step for 3-D anal-ysis of circular objects in computer vision applications. For this reason, the study of ellipse fittingbegan as soon as computers came into use for image analysis in the 1970s. e basic principle wasto compute the parameters so that the sum of squares of expressions that should ideally be zerois minimized, which is today called least squares or algebraic distance minimization. In the 1990s,the notion of optimal computation based on the statistical properties of noise was introducedby researchers including the authors. e first notable example was the authors’ renormalization(1993), which was then improved as FNS (2000) and HEIV (2000) by researchers in Australiaand the U.S. Later, further improvements, called hyperaccurate correction (2006), HyperLS (2009),and hyper-renormalization (2012), were presented by the authors. Today, these are regarded as themost accurate fitting methods among all known techniques. is book describes these algorithmsas well as underlying theories, implementation details, and applications to 3-D scene analysis.

Most textbooks on computer vision begin with mathematical fundamentals followed by theresulting computational procedures. is book, in contrast, immediately describes computationalprocedures after a short statement of the purpose and the principle. e theoretical backgroundis briefly explained as Comments. us, readers need not worry about mathematical details, whichoften annoy those who only want to build their vision systems. Rigorous derivations and detailedjustifications are given later in separate sections, but they can be skipped if the interest is not intheories. Sample program codes of the authors are provided via the website¹ of the publisher. Atthe end of each chapter is given a section called Supplemental Note, describing historical back-grounds, related issues, and reference literature.

Chapters 1–4 specifically describe ellipse fitting algorithms. Chapter 5 discusses 3-D anal-ysis of circular objects in the scene extracted by ellipse fitting. In Chapter 6, performance com-parison experiments are conducted among the methods described in Chapters 1–4. Also, somereal image applications of the 3-D analysis of Chapter 5 are shown. In Chapter 7, we point outhow procedures of ellipse fitting can straightforwardly be extended to fundamental matrix andhomography computation, which play a central role in 3-D analysis by computer vision. Chap-ters 8 and 9 give general mathematical theories of statistical optimization underlying all ellipsefitting algorithms. Finally, Chapter 10 gives a rigorous analysis of the theoretical accuracy limit.However, beginners and practice-oriented readers can skip these last three chapters.

e authors used the materials in this book as student projects for introductory computervision research at Okayama University, Japan, and Toyohashi University of Technology, Japan.By implementing the algorithms themselves, students can learn basic programming know-hows¹https://sites.google.com/a/morganclaypool.com/ellipse-fitting-for-computer-vision-implementation-and-applications/

xii PREFACE

and also understand the theoretical background of vision computation as their interest deepens.We are hoping that this book can serve not simply as a reference of ellipse fitting algorithms forresearchers, but also as learning material for beginners who want to start computer vision research.

e theories in this book are the fruit of the authors’ collaborations and interactions withtheir colleagues and friends for many years. e authors thank Takayuki Okatani of TohokuUniversity, Japan, Mike Brooks and Wojciech Chojnacki of the University of Adelaide, Aus-tralia, Peter Meer of Rutgers University, U.S., Wolfgang Förstner, of the University of Bonn,Germany, Michael Felsberg of Linköping University, Sweden, Rudolf Mester of the Universityof Frankfurt, Germany, Prasanna Rangarajan of Southern Methodist University, U.S., Ali Al-Sharadqah of University of East Carolina, U.S., and Alexander Kukush of the University of Kiev,Ukraine. Special thanks are to (late) Professor Nikolai Chernov of the University of Alabamaat Birmingham, U.S., without whose inspiration and assistance this work would not have beenpossible.

Kenichi Kanatani, Yasuyuki Sugaya, and Yasushi KanazawaMarch 2016

1

C H A P T E R 1

Introductionis chapter describes the basic mathematical formulation to be used in subsequent chapters forfitting an ellipse to observed points. e main focus is on the description of statistical propertiesof noise in the data in terms of covariance matrices. We point out that two approaches exist forellipse fitting: “algebraic” and “geometric.” Also, some historical background is mentioned, andrelated mathematical topics are discussed.

1.1 ELLIPSE FITTING

Ellipse fitting means fitting an ellipse equation to points extracted from an image. is is oneof the fundamental tasks of computer vision for various reasons. First, we observe many circularobjects in man-made scenes indoors and outdoors, and a circle is projected as an ellipse in cameraimages. If we extract elliptic segments, say, by an edge detection filter, and fit an ellipse equationto them, we can compute the 3-D position of the circular object in the scene (we will discusssuch applications in Chapter 5). Figure 1.1a shows edges extracted from an indoor scene, usingan edge detection filter. is scene contains many elliptic arcs, as indicated there. Figure 1.1bshows ellipses fitted to them superimposed on the original image. We observe that fitted ellipsesare not necessarily exact object shapes, in particular when the observed arc is only a small part ofthe circumference or when it is continuously connected to a non-elliptic segment (we will discussthis issue in Chapter 4).

(a) (b)

Figure 1.1: (a) An edge image and selected elliptic arcs. (b) Ellipses are fitted to the arcs in (a) and superimposedon the original image.

2 1. INTRODUCTION

Ellipse fitting is also used for detecting not only circular objects in the image but also ob-jects of approximately elliptic shape, e.g., human faces. An important application of ellipse fittingis camera calibration for determining the position and internal parameters of a camera by takingimages of a reference pattern, for which circles are often used for the ease of image processing.Ellipse fitting is also important as a mathematical prototype of various geometric estimation prob-lems for computer vision. Typical problems include the computation of fundamental matrices andhomographies (we will briefly describe these in Chapter 7).

1.2 REPRESENTATIONOFELLIPSESe equation of an ellipse has the form

Ax2C 2Bxy C Cy2

C 2f0.Dx CEy/C f 20 F D 0; (1.1)

where f0 is a constant for adjusting the scale. eoretically, it can be 1, but for finite-lengthnumerical computation it should be chosen so that x=f0 and y=f0 have approximately the orderof 1; this increases the numerical accuracy, avoiding the loss of significant digits. In view of this,we take the origin of the image xy coordinate system at the center of the image, rather than theupper-left corner as is customarily done, and take f0 to be the length of the side of a square whichwe assume to contain the ellipse to be extracted. For example, if we know that an ellipse exists ina 600 � 600 pixel region, we let f0 = 600. Since Eq. (1.1) has scale indeterminacy, i.e., the sameellipse is represented if A, B , C , D, E, and F are simultaneously multiplied by a nonzero constant,we need some kind of normalization. Various types of normalizations have been considered in thepast, including

F D 1; (1.2)

AC C D 1; (1.3)

A2C B2

C C 2CD2

CE2C F 2

D 1; (1.4)

A2C B2

C C 2CD2

CE2D 1; (1.5)

A2C 2B2

C C 2D 1; (1.6)

AC � B2D 1: (1.7)

Among these, Eq. (1.2) is the simplest and most familiar one, but Eq. (1.1) with F = 1 cannotexpress an ellipse that passes through the origin .0; 0/. Equation (1.3) remedies this. Each of theabove normalization equations has its own reasoning, but in this book we adopt Eq. (1.4) (seeSupplemental Note (page 6) below for the background).

1.3. LEAST SQUARESAPPROACH 3

If we define the 6-D vectors

ξ D

0BBBBBBB@

x2

2xy

y2

2f0x

2f0y

f 20

1CCCCCCCA ; θ D

0BBBBBBB@

A

B

C

D

E

F

1CCCCCCCA ; (1.8)

Eq. (1.4) can be written as.ξ;θ/ D 0; (1.9)

where and hereafter we denote the inner product of vectorsa and b by .a; b/. Since the vector θ inEq. (1.9) has scale indeterminacy, it must be normalized in correspondence with Eqs. (1.2)–(1.7).Note that the left sides of Eqs. (1.2)–(1.7) can be seen as quadratic forms in A, ..., F ; Eqs. (1.2)and (1.3) are linear equations, but we may regard them as F 2 = 1 and .AC C /2 = 1, respectively.Hence, Eqs. (1.2)–(1.7) are all written in the form

.θ;Nθ/ D 1; (1.10)

for some normalization matrix N . e use of Eq. (1.4) corresponds to N = I (the identity), inwhich case Eq. (1.10) is simply kθk = 1, i.e., normalization to unit norm.

1.3 LEAST SQUARESAPPROACHFitting an ellipse in the form of Eq. (1.1) to a sequence of points .x1; y1/, ..., .xN ; yN / in thepresence of noise (Fig. 1.2) is to find A, B , C , D, E, and F such that

Ax2˛ C 2Bx˛y˛ C Cy2

˛ C 2f0.Dx˛ CEy˛/C f 20 F � 0; ˛ D 1; :::; N: (1.11)

If we write ξ˛ for the value obtained by replacing x and y in the 6-D vector ξ of Eq. (1.8) by x˛

and y˛, respectively, Eq. (1.11) can be equivalently written as

.ξ˛;θ/ � 0; ˛ D 1; :::; N: (1.12)

(x , y )α α

Figure 1.2: Fitting an ellipse to a noisy point sequence.

4 1. INTRODUCTION

Our task is to compute such a unit vector θ. e simplest and the most naive method is thefollowing least squares.

Procedure 1.1 (Least squares)

1. Compute the 6 � 6 matrix

M D1

N

NX˛D1

ξ˛ξ>˛ : (1.13)

2. Solve the eigenvalue problemMθ D �θ; (1.14)

and return the unit eigenvector θ for the smallest eigenvalue �.

Comments. is is a straightforward generalization of line fitting to a point sequence (,! Prob-lem 1.1); we minimize the sum of the squares

J D1

N

NX˛D1

.ξ˛;θ/2D

1

N

NX˛D1

θ>ξ˛ξ>˛ θ D .θ;

� 1

N

NX˛D1

ξ˛ξ>˛

�θ/ D .θ;Mθ/; (1.15)

subject to kθk = 1. As is well known in linear algebra, the minimum of this quadratic form inθ is given by the unit eigenvector θ of M for the smallest eigenvalue. Equation (1.15) is oftencalled the algebraic distance, and Procedure 1.1 is also known as algebraic distance minimization. Itis sometimes calledDLT (direct linear transformation). Since the computation is very easy and thesolution is immediately obtained, this method has been widely used. However, when the inputpoint sequence covers only a small part of the ellipse circumference, it often produces a small andflat ellipse very different from the true shape (we will see such examples in Chapter 6). Still, thisis a prototype of all existing ellipse fitting algorithms. How we can improve this method is themain theme of this book.

1.4 NOISEANDCOVARIANCEMATRICESe reason for the poor accuracy of Procedure 1.1 is that the properties of image noise are notconsidered; for accurate fitting, we need to take the statistical properties of noise into consider-ation. Suppose the data x˛ and y˛ are disturbed from their true values Nx˛ and Ny˛ by �x˛ and�y˛:

x˛ D Nx˛ C�x˛; y˛ D Ny˛ C�y˛: (1.16)

Substituting this into ξ˛, we can write

ξ˛ DNξ˛ C�1ξ˛ C�2ξ˛; (1.17)

1.4. NOISEANDCOVARIANCEMATRICES 5

where Nξ˛ is the value of ξ˛ obtained by replacing x˛ and y˛ by their true values Nx˛ and Ny˛, re-spectively, while �1ξ˛ and �2ξ˛ are, respectively, the first-order noise term (the linear expressionin �x˛ and �y˛) and the second-order noise term (the quadratic expression in �x˛ and �y˛).From Eq. (1.8), we obtain the following expressions:

�1ξ˛ D

0BBBBBBB@

2 Nx˛�x˛

2�x˛ Ny˛ C 2 Nx˛�y˛

2 Ny˛�y˛

2f0�x˛

2f0�y˛

0

1CCCCCCCA ; �2ξ˛ D

0BBBBBBB@

�x2˛

2�x˛�y˛

�y2˛

0

0

0

1CCCCCCCA : (1.18)

We regard the noise terms �x˛ and �y˛ as random variables and define the covariance matrix ofξ˛ by

V Œξ˛� D EŒ�1ξ˛�1ξ>˛ �; (1.19)

where EŒ � � denotes expectation over the noise distribution. If we assume that �x˛ and �y˛ aresampled from independent Gaussian distributions of mean 0 and standard deviation � , we obtain

EŒ�x˛� D EŒ�y˛� D 0; EŒ�x2˛� D EŒ�y2

˛� D �2; EŒ�x˛�y˛� D 0: (1.20)

Substituting Eq. (1.18) and using this relationship, we obtain the covariance matrix in Eq. (1.19)in the following form:

V Œξ˛� D �2V0Œξ˛�; V0Œξ˛� D 4

0BBBBBBB@

Nx2˛ Nx˛ Ny˛ 0 f0 Nx˛ 0 0

Nx˛ Ny˛ Nx2˛ C Ny

2˛ Nx˛ Ny˛ f0 Ny˛ f0 Nx˛ 0

0 Nx˛ Ny˛ Ny2˛ 0 f0 Ny˛ 0

f0 Nx˛ f0 Ny˛ 0 f 20 0 0

0 f0 Nx˛ f0 Ny˛ 0 f 20 0

0 0 0 0 0 0

1CCCCCCCA : (1.21)

Since all the elements of V Œξ˛� have themultiple �2, we factor it out and call V0Œξ˛� the normalizedcovariance matrix. We also call the standard deviation � the noise level . e diagonal elements ofthe covariance matrix V Œξ˛� indicate the noise susceptibility of each component of ξ˛, and theoff-diagonal elements measure their pair-wise correlation.

e covariance matrix of Eq. (1.19) is defined in terms of the first-order noise term �1ξ˛

alone. It is known that incorporation of the second-order term �2ξ˛ has little influence overfinal results. is is because �2ξ˛ is very small as compared with �1ξ˛. Note that the elementsof V0Œξ˛� in Eq. (1.21) contain true values Nx˛ and Ny˛. ey are replaced by observed values x˛

and y˛ in actual computation. It is known that this replacement has practically no effect in thefinal results.

6 1. INTRODUCTION

1.5 ELLIPSE FITTINGAPPROACHESIn the following chapters, we describe typical ellipse fitting methods that incorporate the abovenoise properties. We will see that all the methods we consider do not require knowledge of the noiselevel � , which is very difficult to estimate in real problems. e qualitative properties of noise areall encoded in the normalized covariance matrix V0Œξ˛�, which gives sufficient information fordesigning high accuracy fitting schemes. In general terms, there exist two approaches for ellipsefittings: algebraic and geometric.

Algebraic methods: We solves some algebraic equation for computing θ. e resulting solutionmay or may not minimize some cost function. In other words, the equation need not havethe form of rθJ = 0 for some cost function J . Rather, we can modify the equation in anyway so that the resulting solution θ is as close to its true value Nθ as possible. us, our taskis to find a good equation to solve. To this end, we need detailed statistical error analysis.

Geometric methods: We minimize some cost function J . Hence, the solution is uniquely deter-mined once the cost J is defined. us, our task is to find a good cost to minimize. For this,we need to consider the geometry of the ellipse and the data points. We also need to devisea convenient minimization algorithm, since minimization of a given cost is not always easy.

e meaning of these two approaches will be better understood by seeing the actual proceduresdescribed in the subsequent chapters. ere are, however, a lot of overlaps between the two ap-proaches.

1.6 SUPPLEMENTALNOTEe study of ellipse fitting began as soon as computers came into use for image analysis in the1970s. Since then, numerous fitting techniques have been proposed, and even today new methodsappear one after another. Since we are unable cite all the literature, we mention only some ofthe earlest work: Albano [1974], Bookstein [1979], Cooper and Yalabik [1979], Gnanadesikan[1977], Nakagawa and Rosenfeld [1979], Paton [1970]. Beside fitting an ellipse to data points,a voting scheme called Hough transform for accumulating evidences in the parameter space wasalso studied as a means of ellipse fitting [Davis, 1989].

In the 1990s, a paradigm shift occurred. It was first thought that the purpose of ellipsefitting was to find an ellipse that approximately passes near observed points. However, some re-searchers, including the authors, turned their attention to finding an ellipse that exactly passesthrough the true points that would be observed in the absence of noise. us, the problem turnedto a statistical problem for estimating the true points subject to the constraint that they are on someellipse. It follows that the goodness of the fitted ellipse is measured not by how close it is to theobserved points but by how close it is to the true shape. is type of paradigm shift has also oc-curred in other problems including fundamental matrix and homography computation for 3-D

1.6. SUPPLEMENTALNOTE 7

analysis. Today, statistical analysis is one of the main tools for accurate geometric computationfor computer vision.

e ellipse fitting techniques discussed in this book are naturally extended to general curvesand surfaces in a general space in the form

�1�1 C �2�2 C � � � C �n�n D 0; (1.22)

where �1, ..., �n are functions of coordinates x1, x2, ..., and �1, ..., �n are unknowns to be de-termined. is equation is written in the form of .ξ;θ/ = 0 in terms of the vector ξ = .�i / ofobservations and the vector θ = .�i / of unknowns. en, all techniques and analysis for ellipsefitting can apply. Evidently, Eq. (1.22) includes all polynomial curves in 2-D and all polynomialsurfaces in 3-D, but all algebraic functions also can be expressed in the form of Eq. (1.22) aftercanceling denominators. For general nonlinear surfaces, too, we can usually write the equationin the form of Eq. (1.22) after an appropriate reparameterization, as long as the problem is toestimate the “coefficients” of linear/nonlinear terms. Terms that are not multiplied by unknowncoefficients are regarded as being multiplied by 1, which is also regarded as an unknown. eresulting set of coefficients can be viewed as a vector of unknown magnitude, or a “homogeneousvector,” and we can write the equation in the form .ξ;θ/ = 0. us, the theory in this book haswide applicability beyond ellipse fitting.

In Eq. (1.1), we introduce the scaling constant f0 to make x=f0 and y=f0 have the orderof 1, and this also make the vector ξ in Eq. (1.8) have magnitude O.f 2

0 / so that ξ=f 20 is ap-

proximately a unit vector. e necessities and effects of such scaling for numerical computationis discussed by Hartley [1997] in relation to fundamental matrix computation (we will discussthis in Chapter 7), which is also a fundamental problem of computer vision and has the samemathematical structure as ellipse fitting. In this book, we introduce the scaling constant f0 andtake the coordinate origin at the center of the image based on the same reasoning.

e normalization using Eq. (1.2) was adopted by Albano [1974], Cooper and Yalabik[1979], and Rosin [1993]. Many authors used Eq. (1.4), but some authors preferred Eq. (1.5)[Gnanadesikan, 1977]. e use of Eq. (1.6) was proposed by Bookstein [1979], who argued thatit leads to “coordinate invariance” in the sense that the ellipse fitted by least squares after the co-ordinate system is translated and rotated is the same as the originally fitted ellipse translated androtated accordingly. In this respect, Eqs. (1.3) and (1.7) also have that invariance. e normaliza-tion using Eq. (1.7) was proposed by Fitzgibbon et al. [1999] so that the resulting least-squaresfit is guaranteed to be an ellipse, while other equations can theoretically produce a parabola or ahyperbola (we will discuss this in Chapter 4).

Today, we need not worry about the coordinate invariance, which is a concern of the past.As long as we regard ellipse fitting as statistical estimation and use Eq. (1.10) for normalization,all statistically meaningful methods are automatically invariant to the choice of the coordinatesystem. is is because the normalization matrix N in Eq. (1.10) is defined as a function of thecovariance matrix V Œξ˛� of Eq. (1.19). If we change the coordinate system, e.g., adding transla-tion, rotation, and other arbitrary mapping, the covariance matrix V Œξ˛� defined by Eq. (1.19)

8 1. INTRODUCTION

also changes, and the fitted ellipse after the coordinate change using the transformed covariancematrix is the same as the ellipse fitted in the original coordinate system using the original covari-ance matrix and transformed afterwards.

e fact that the all the normalization equations of Eqs. (1.2)–(1.7) are written in the formof Eq. (1.10) poses an interesting question: What N is the “best,” if we are to minimize thealgebraic distance of Eq. (1.15) subject to .θ;Nθ/ = 1? is problem was studied by the authors’group [Kanatani and Rangarajan, 2011, Kanatani et al., 2011, Rangarajan and Kanatani, 2009],and the matrix N that gives rise to the highest accuracy was found after a detailed error analysis.e method was named HyperLS (this will be described in the next chapter).

Minimizing the sum of squares in the form of Eq. (1.15) is a natural idea, but read-ers may wonder why we minimize

PN˛D1.ξ˛;θ/2. Why not minimize, say, the absolute sumPN

˛D1 j.ξ˛;θ/j or the maximum maxN˛D1 j.ξ˛;θ/j? is is the issue of the choice of the norm. A

class of criteria, called Lp-norms, exist for measuring the magnitude of an n-D vector x = .xi /:

kxkp �� nX

iD1

jxi jp�1=p

: (1.23)

e L2-norm, or the square norm,

kxk2 �

vuut nXiD1

jxi j2; (1.24)

is widely used for linear algebra. If we let p!1 in Eq. (1.23), it approaches

kxk1 �nmax

iD1jxi j; (1.25)

called the L1-norm, or the maximum norm, where components with large jxi j have a dominanteffect and those with small jxi j are ignored. Conversely, if let p ! 0 in Eq. (1.23), those com-ponents with large jxi j are ignored. e L1-norm

kxk1 �nX

iD1

jxi j (1.26)

is often called the average norm, because it effectively measures the average .1=N /Pn

iD1 jxi j. Inthe limit of p! 0, we obtain

kxk0 � jfi j jxi j ¤ 0gj; (1.27)

where the right side means the number of nonzero components. is is called the L0-norm, orthe Hamming distance.

Least squares for minimizing the L2-norm is the most widely used approach for statisti-cal optimization for two reasons. One is the computational simplicity: differentiation of a sum

1.6. SUPPLEMENTALNOTE 9

of squares leads to linear expressions, so the solution is immediately obtained by solving linearequations or an eigenvalue problem. e other reason is that it corresponds tomaximum likelihoodestimation (we will discuss this in Chapter 9), provided the discrepancies to be minimized arisefrom independent and identical Gaussian noise. Indeed, the scheme of least squares was inventedby Gauss, who introduced the “Gaussian distribution,” which he himself called the “normal dis-tribution,” as the standard noise model (physicists usually use the former term, while statisticiansprefer the latter). In spite of added computational complexity, however, minimization of the Lp-norm for p < 2, typically p = 1, has its merit, because the effect of those terms with large absolutevalues are suppressed. is suggests that terms irrelevant for estimation, called outliers, are auto-matically ignored. Estimation methods that are not very susceptible to outliers are said to berobust (we will discuss robust ellipse fitting in Chapter 4). For this reason, L1-minimization isfrequently used in some computer vision applications.

PROBLEMS1.1. Fitting a line in the form n1x C n2y C n3f0 = 0 to points .x˛; y˛/, ˛ = 1, ..., N , can be

viewed as the problem for computing n = .ni /, which can be normalized to a unit vector,such that .ξ˛;n/ � 0, ˛ = 1, .., N , with

ξ˛ D

0@ x˛

y˛

f0

1A ; n D

0@ n1

n2

n3

1A : (1.28)

(1) Write down the least-squares procedure for this computation.(2) If the noise terms �x˛ and �y˛ are sampled from independent Gaussian distribu-

tions of mean 0 and variance �2, how is their covariance matrix V Œξ˛� defined?