Joint optical ﬂow estimation, segmentation, and 3D...

www.elsevier.com/locate/cviu

Computer Vision and Image Understanding 103 (2006) 89–100

Joint optical flow estimation, segmentation,and 3D interpretation with level sets

H. Sekkati, A. Mitiche *

Institut National de la recherche Scientifique, INRS-EMT, Place Bonaventure, 800, de la Gauchetiere Ouest, Suite 6900, Montreal, Que., Canada H5A1K6

Received 8 July 2005; accepted 16 November 2005Available online 5 January 2006

Abstract

This paper describes a variational method with active curve evolution and level sets for the estimation, segmentation, and 3D inter-pretation of optical flow generated by independently moving rigid objects in space. Estimation, segmentation, and 3D interpretation areperformed jointly. Segmentation is based on an estimate of optical flow consistent with a single rigid motion in each segmentation region.The method, which allows both viewing system and viewed objects to move, results in three steps iterated until convergence: (a) evolutionof closed curves via level sets and, in each region of the segmentation, (b) linear least squares computation of the essential parameters ofrigid motion, (c) estimation of optical flow consistent with a single rigid motion. The translational and rotational components of rigidmotion and regularized relative depth are recovered analytically for each region of the segmentation from the estimated essential param-eters and optical flow. Several examples with real image sequences are provided which verify the validity of the method.� 2005 Elsevier Inc. All rights reserved.

Keywords: 3D structure; 3D motion; Segmentation; Optical flow; Variational method; Level sets

A major branch of picture processing deals with
image analysis or scene analysis; . . . the desired outputis a description of the given picture or scene.Segmentation is basically a process of pixel classifica-
tion; the picture is segmented into subsets by assigningthe individual pixels to classes . . . The approach dis-cussed . . . minimizes the expected classification error, . . .,more generally, (an) expected cost . . . It should be point-ed out that if range or velocity information is availablefor each pixel, obtained by special sensors or derivedfrom stereopairs or image sequences, we can use thisinformation as a basis for segmenting the picture.(At) an edge, the gray level is relatively consistent ineach of two adjacent, extensive regions, and changesabruptly as the border between the regions is crossed . . .this is a special case of pixel classification.

1077-3142/$ - see front matter � 2005 Elsevier Inc. All rights reserved.

doi:10.1016/j.cviu.2005.11.002

* Corresponding author.E-mail addresses: [email protected] (H. Sekkati), mitiche@emt.

inrs.ca (A. Mitiche).

Azriel Rosenfeld in A. Rosenfeld and A. C. Kak Digital

picture processing, Academic Press, Second edition,1982.

Foreword

Azriel Rosenfeld founded the field of image process-ing four decades ago. He identified and investigated fun-damental subjects which are still the focus of intenseresearch. The book Digital picture processing byA. Rosenfeld and A. C. Kak, published in 1976, wasstructured based on these subjects and describedresearch in a textbook style. This made its contentsinformative, accessible, and attractive so as to bestirextensive and sustained research. This paper deals withimage segmentation/edge-detection and description, sub-jects that the quotes from the book highlight. The basisfor segmentation/edge-detection is the velocity field

derived from an image sequence, and the saught descrip-tion is the three-dimensional (3D) structure and motionof the objects in the observed scene. The velocity fieldand its 3D interpretation are derived from cost function

mailto:[email protected]

mailto:mitiche@emt. inrs.ca

mailto:mitiche@emt. inrs.ca

90 H. Sekkati, A. Mitiche / Computer Vision and Image Understanding 103 (2006) 89–100

minimization, using some of the latest analysis tools: thevariational formalism and active curves via level sets.

x

y

O

image plane

X

Y

foca

l len

gth

= f

Z

optic

al ax

is

Ωω1

ω2

ω3

p(x,y)

P(X,Y,Z)

.

.

t1

t2

t3

Fig. 1. The viewing system is modeled by an orthonormal coordinatesystem and central projection through the origin.

1. Introduction

Optical flow analysis plays fundamental roles in com-puter vision. For instance, it serves movement detection,tracking, segmentation, and 3D interpretation [1,2].Optical flow 3D interpretation is of considerable interestin applications such as robot autonomous navigation,virtual and augmented reality, medical diagnosis, andsurveillance.

The primary focus of this study is dense 3D interpreta-tion of optical flow of rigid objects moving independentlyrelative to a viewing system. Within the general contextof 3D interpretation of image sequences, sparse interpreta-tion, which seeks depth and 3D motion from views of asparse set of points in space, has been the subject of a sub-stantial number of well documented studies [3–5,2,6].Dense optical flow interpretation has been investigated rel-atively little because it is considerably more complex, par-ticularly when the viewing system and the viewed objectsare allowed to move, [7–25].

Most current methods of dense optical flow interpreta-tion consider the problem of a viewing system moving ina stationary environment [7–19]. In this case, the cameramotion is the single 3D motion to recover and movingobject segmentation is no longer an issue. In spite of thesesimplifications, interpretation remains complicated byimage noise and depth discontinuities. Such complicationsoccur in optical flow and disparity estimation as well andhave been resolved by modeling and anisotropic diffusionwithin a variational framework [26]. The variational for-malism allows the coding of models, assumptions, and con-straints in a single functional to minimize. This leads toefficient, tractable algorithms [27]. The variational methodsof optical flow 3D interpretation in [19,8,12] have not fullyexploited the formalism.

A few studies have addressed the problem when boththe viewing system and the viewed objects are allowedto move independently [20–22,24,23,25]. In this case, seg-mentation of the image domain into regions correspond-ing to differently moving objects in space is essential.This complicates the problem significantly. Non variation-al methods have used traditional grouping processes forsegmentation, such as region growing [22] and clustering[20]. Variational formulations have been investigated in[24,23,25]. In [24,23], the methods are not based on activecurves/level sets and do not address segmentation explic-itly. The active curves/level sets method in [25] statesthe problem as 3D-motion segmentation with simulta-neous 3D-motion and dense depth estimation within eachregion of the segmentation.

This study investigates a new level set method for jointestimation, segmentation, and 3D interpretation of theoptical flow generated by multiple rigid objects in spacemoving independently relative to the viewing system.

Optical flow is estimated so as to correspond to a singlerigid 3D motion in each segmentation region via theessential parameters representation of rigid motion[28,29,2]. The formulation results in an algorithm wherethree steps are interated until convergence: (a) evolutionof closed curves via level sets and, in each segmentationregion, (b) least squares computation of the essentialparameters of rigid motion and (c) estimation of opticalflow consistent with a single rigid 3D motion. The trans-lational and rotational components of rigid motion andregularized relative depth are recovered analytically foreach segmentation region from the estimated essentialparameters and optical flow. Several examples with realimage sequences are provided to verify the validity ofthe method.

2. Basic models

In this section, we describe a constraint which relatesoptical flow to the representation of rigid 3D motion byessential parameters. This constraint will be used in theformulation to compute an estimate of optical flow con-form to a single rigid 3D motion in each segmentationregion. We also describe a mapping between the interi-ors fRck

gN�1k¼1 of a family of N � 1 closed plane curves

fckgN�1k¼1 and regions fRkgN

k¼1 which always forms a par-tition of the image domain. This mapping will be usedin the formulation to guarantee that the computed seg-mentation corresponds to a partition of the imagedomain.

2.1. Optical flow and essential parameters of rigid 3D motion

Let T = (t1, t2, t3), x = (x1,x2,x3) be the translationaland rotational components of the motion of a rigid bodyB in space. Let p be the image of a point P 2 B (refer to

H. Sekkati, A. Mitiche / Computer Vision and Image Understanding 103 (2006) 89–100 91

Fig. 1). Finally, let (x,y) be the coordinates of p andw = (u,v) the optical flow at (x,y) (the optical velocityof p) at some instant of observation. Optical flow and thecomponents T, x of the rigid motion of B are related bythe following homogeneous linear equation [28,29,2]:

d � e ¼ 0; ð1Þwhere d is a vector of dimension nine whose componentsdepend on image coordinates and optical flow:

d ¼ ðx2; y2; f 2; 2xy; 2xf ; 2yf ;�fv; fu;�uy þ vxÞ ð2Þand e is the vector of essential parameters related to T andx by

t1 ¼ e7; t2 ¼ e8; t3 ¼ e8;

2x1t1 ¼ e1 � e2 � e3;

2x2t2 ¼ e2 � e3 � e1;

2x3t3 ¼ e3 � e2 � e1;

x2t1 þ x1t2 ¼ 2e4;

x1t3 þ x3t1 ¼ 2e5;

x2t3 þ x3t2 ¼ 2e6. ð3Þ

2.2. A partition of the image domain from closed curves

In segmentation by active curve evolution, the standardrepresentation of a region maps the region to the interior ofa closed simple curve. A problem arises when two or morecurves intersect because the membership of the points inthe intersection becomes ambiguous. It is essential that asegmentation algorithm yields a partition of the imagedomain, albeit at convergence [30,31–33]. Here, we defineregions from curves so as to guarantee a partition [33]. Thisdefinition is as follows. Refer to Fig. 2 for an illustration.

Let fckgN�1k¼1 be a family of closed plane curves, and let

Rckdesignate the interior of ck,k = 1, . . .,N � 1. We define

region R1 as Rc1, i.e., the interior of c1. Region R2 of the

segmentation, however, is identified with Rcc1\ Rc2

, ratherthan Rc2

as in the standard correspondence. Similarly,

R2

R3

R1

γ1 γ3

γ2R4

Fig. 2. Representation of a partition of the image domain by explicitcorrespondence between regions of segmentation and regions defined byclosed plane curves (illustration for four regions).

region R3 of the segmentation is defined as Rcc1\ Rc

c2\ Rc3

and, more generally, Rk ¼ Rcc1\ Rc

c2\ � � � \ Rc

ck�1\

Rck; k 6 N � 1. The last region, RN, is defined as in the

standard correspondence by RN ¼ ð[N�1k¼1 RkÞc, i.e., the com-

plement of the union of the other regions. Therefore, theresulting family fRkgN

k¼1 is a partition of the image domainfor any given family of closed curves fckg

N�1k¼1 .

3. Formulation

The problem of segmenting optical flow into N regions,each corresponding to a single rigid 3D object, can be stat-ed as follows: minimize

EðfckgN�1k¼1 Þ ¼

XN

k¼1

ZRk

u� � �u�k� �2 þ v� � �v�k

� �2� �

dx

þ kXN�1

k¼1

Zck

ds; ð4Þ

where the family of regions fRkgNk¼1 is defined from closed

curves fckgN�1k¼1 as in Section 2.2; (u*, v*) is a motion field

consistent, in each region, with a single rigid motion de-scribed by essential parameters and computed jointly withthe essential parameters by

ðu�; v�; ekÞ ¼ arg minu;v;ek

ZRk

ðd � ekÞ2 þ lðrI � wþ I tÞ2h

þmðgðkrukÞ þ gðkrvkÞÞ�dx; k ¼ 1; . . . ;N

ð5Þ

and, �u�kð�v�kÞ is the mean of u* (v*) in region Rk, k = 1, . . .,N,

�u�k ¼1R

Rkdx

ZRk

u�dx

�v�k ¼1R

Rkdx

ZRk

v�dx.

Function g in (5) is of class C2. With the quadratic func-tion, we have the Horn-and-Shunck regularization term.This function can also be chosen to preserve motionboundaries within each region, as with the Aubert–Deri-che–Kornprobst function gðsÞ ¼ 2

ffiffiffiffiffiffiffiffiffiffiffiffi1þ s2p

� 2 [26].This formulation can be explained as follows. The

purpose is 3D interpretation of the optical flow inducedby 3D objects simultaneously and independently moving.First, this requires the segmentation of the objects. Sec-ond, it requires the optical flow. Third, the optical flowmust be related to the variables of the 3D interpretationwithin each region of the segmentation. These require-ments are all embedded in Functionals (4) (segmenta-tion) and (5) (estimation of optical flow and 3Dinterpretation) of the formulation. The two functionalsare tightly linked because the formulation seeks simulta-neously a segmentation and an estimate of optical flowconsistent with a single rigid body motion in each regionof the segmentation. There are two important observa-tions to make:


(A) In image and optical flow segmentation [34,35], afunctional such as (4) is sometimes referred to as a piece-wise constant segmentation functional . It is the simplestform of the general linear parametric functional [36]. Thefocus of such formulations is not on computing an accurateparametric representation of an image or a flow field, buton partitioning the image domain into regions which (areassumed to) differ by the parameters of the representation.This is the case in this formulation which seeks a segmen-tation under the assumption that the desired regions, i.e.,where each region corresponds to a single rigid object inspace, differ by their average optical velocity. This assump-tion is generally valid and, if necessary, an affine model [35]can be used without affecting the segmentation paradigm.However, 3D interpretation requires both an accurate opti-cal flow estimate [2] and a segmentation where each regioncorresponds to a single rigid object in space. These require-ments justify Functional (5). For each region of the seg-mentation, its minimization yields an estimate of opticalflow which is smooth (the third term of the integrand), con-forms to the spatio-temporal data (second term), and isconsistent with a single rigid body motion described byits essential parameters (the first term). Optical flow andthe essential parameters are estimated jointly. The contri-bution of the 3D interpretation term (the first term) to boththe segmentation and the estimation of optical flow will beillustrated in the experiments (Section 6). Note thatsmoothness defined by total variation as in (5) cannot bea basis for segmentation when regions are explicitly refer-enced in the objective functional as in (4).

(B) In regard to joint optical flow estimation and seg-mentation, this formulation is most related to the one in[35], and can be seen as a generalization because it alsojointly estimates 3D structure and motion. Other notabledifferences are: with this formulation (a) the estimated opti-cal flow is consistent with rigid 3D motion, (b) each regionof the segmentation is consistent with a single 3D rigidbody motion, (c) segmentation into N regions uses N � 1curves as described in Section 2.2, whereas in [35] it useslogN curves following the construction in [34]. Our con-struction has the advantage of being straightforward toimplement for an arbitrary number of regions [33]. Finally,(d) the method in [35] uses a single functional, whereas thismethod affords more flexibility with two different function-als, one for segmentation, the other for optical flow estima-tion (and 3D interpretation).

In regard to 3D interpretation, this formulation is mostrelated to the active curves/level sets formulation in [25].The formulation in [25] uses a term of conformity to databased on the Horn-and-Schunck optical flow constraintin which it substitutes for optical velocity its expressionin terms of depth and the rigid 3D motion componentsof translation and rotation. Therefore, one significant dif-ference is that optical flow estimation in this formulationreplaces the estimation of depth in [25]. This has a majorimplication computationally because optical flow involvessparse systems of linear equations which, although large,

can be solved quite efficiently by Gauss–Seidel iterationsor the semi-quadratic algorithm, whereas the estimationof depth involves large systems of nonlinear equationssolved by gradient descent. Another difference is that seg-mentation in [25] is based directly on 3D motion, ratherthan on optical flow constrained by 3D motion. On allthe examples we ran, both methods produce similar resultsbut this method executes significantly faster.

4. Algorithm

The formulation can be implemented by an algorithmwhich iterates three consecutive steps: computation of theessential parameters for each region, estimation of the opti-cal flow in each region, and curve evolution.

4.1. Initialization

The segmentation curves and the optical flow are initial-ized. The optical flow is initialized to zero. Alternatively, itcan be initialized with the result of a few iterations of theHorn-and-Schunck algorithm. An initial partition isdefined (Section 2.2) using closed curves arbitrarily placedover the image domain.

4.2. Step 1: essential parameters

With the optical flow and the segmentation curves fixed,Functionals (5) are each minimized with respect to theessential parameters. Because the essential parameters areconstant in each region, this reduces their estimation, upto a scale factor, to a linear least squares fit within eachregion. This can be done efficiently by the singular valuedecomposition method [37].

4.3. Step 2: optical flow estimation

With the curves of the segmentation and essentialparameters fixed, Functionals in (5) are minimized withrespect to optical flow. The corresponding Euler–Lagrangeequations for region Rk, k = 1, . . .,N, are

ðfek;8� yek;9Þd � ekþlIxðrI �wþ I tÞ�mdiv g0ðkrukÞkruk ru

� �¼ 0;

ð�fek;7þ xek;9Þd � ekþlIyðrI �wþ I tÞ� mdiv g0ðkrvkÞkrvk rv

� �¼ 0;

8><>:

ð6Þ

where ek, j designates the jth component of ek, the vector ofessential parameters corresponding to region Rk, andw = (u,v). When g is the quadratic function, we have a largesparse linear system equations which can be solved effi-ciently by Jacobi or Gauss–Seidel iterations, as with theHorn-and-Schunck algorithm. When g is the Aubert–Der-iche–Kornprobst function, we have a large sparse system ofnonlinear equations which can be solved efficiently by thehalf-quadratic algorithm [26]. In both cases, computationsare confined to the regions interior; this has the effect of


preserving 3D-motion boundaries because regions differ bytheir 3D-motion via the essential parameters.

4.4. Step 3: curves evolution

The curves evolution is governed by the Euler–Lagrangedescent equations corresponding to the minimization ofFunctional (4) with respect to the curves:

dck

ds¼ � dE

ock; k ¼ 1; . . . ;N � 1; ð7Þ

where each curve ck is embedded in a one-parameter familyof curves indexed by the (algorithmic) time parameter s.

With our representation of a partition from closedcurves (Section 2.2), and with optical flow and the essentialparameters fixed, the expression of the descent Eq. (7) are(see [33] for generic details):

dc1

ds ¼ �ðn1 � w1 þ kj1Þn1;

dc2

ds ¼ � vRc1ðn2 � w2Þ þ kj2

� �n2;

..

.

dckds ¼ � vRc

1. . . vRc

k�1ðnk � wkÞ þ kjk

� �nk;

..

.

dcN�1

ds ¼ � vRc1. . . vRc

N�2ðnN�1 � wN�1Þ þ kjN�1

� �nN�1;

8>>>>>>>>>>>>><>>>>>>>>>>>>>:

ð8Þwhere vR designates the characteristic function of region R,i.e., vR (x) = 1 if x 2 R and vR(x) = 0 otherwise; nk is theoutward unit normal function to ck and jk its mean curva-ture function. For k = 1,. . .,N, functions nk and wk are de-fined by

nk ¼ ðu� � �u�kÞ2 þ ðv� � �v�kÞ

2; k ¼ 1; . . . ;N ;

wk ¼ nkþ1vRckþ1þ nkþ2vRc

ckþ1vRckþ2

þ � � �

þ nN�1vRcckþ1� � � vRc

cN�2vRcN�1

þ nNvRcckþ1

. . . vRccN�2

vRccN�1

.

The corresponding level set evolution equations [38,33]are

oU1

os ¼ � n1 � w1 þ kjU1ð Þ j $U1 j;

oU2

os ¼ � vRcU1

ðn2 � w2Þ þ kjU2

� �j $U2 j;

..

.

oUkos ¼ � vRc

U1

. . . vRcUk�1

ðnk � wkÞ þ kjUk

� �j $Uk j;

..

.

oUN�1

os ¼ � vRcU1

. . . vRcUN�2

ðnN�1 � wN�1Þ þ kjUN�1

� �j $UN�1 j;

8>>>>>>>>>>>>><>>>>>>>>>>>>>:

ð9Þ

where Uk is the level set function corresponding to ck takenpositive inside ck; RUk ¼ fx 2 X j Ukðx; tÞ > 0g;jUk ¼ r � ðrUk= j rUk jÞ. Functions wk are given by

wk ¼ nkþ1vRUkþ1þ nkþ2vRc

Ukþ1

vRUkþ2þ . . .

þ nN�1vRcUkþ1

. . . vRcUN�2

vRUN�1

þ nNvRcUkþ1

. . . vRcUN�2

vRUcN�1

.

We summarize the algorithm as follows:

Initialize the curves and optical flow.Iterate

1. Compute the essential parameters in each region by lin-ear least-squares.

2. Estimate optical flow using (6).3. Evolve curves via level sets using (9).

Until convergence

In Functional (4), the term for region Rk in the sum cor-responding to the u (v) component of the motion field isproportional to the variance of the component in theregion. Therefore, assuming that the smoothing of motionwhich occurs at step 2 of the algorithm decreases this var-iance, Functional (4) will decrease at each iteration and thealgorithm will converge.

5. Recovery of structure

The components T, x of rigid motion for each seg-mentation region can be recovered analytically from theregion essential parameters. From (3), the translationalcomponent T = (t1, t2, t3), up to a sign and a positivescale factor, is given by t1 = e7, t2 = e8, t3 = e9. The rota-tional component is obtained, also analytically, from theother equations in (3), i.e., from the linear system ofequations

0 �e8 �e9

�e7 0 �e9

�e7 �e8 0e8

2e7

20

e9

20 e7

2

0 e9

2e8

2

0BBBBBBBB@

1CCCCCCCCA

x1

x2

x3

0B@

1CA ¼

e1

e2

e3

e4

e5

e6

0BBBBBBBB@

1CCCCCCCCA

.

Depth cannot be recovered if T = 0. When T „ 0, therelations [29]

u ¼ 1Z ðft1 � xt3Þ � xy

f x1 þ f 2þx2

f x2 � yx3;

v ¼ 1Z ðft2 � yt3Þ � f 2þy2

f x1 þ xyf x2 þ xx3

8<: ð10Þ

give (relative) depth in terms of (u,v) and (T,x):

Z¼ ðft1�xt3Þ2þðft2� yt3Þ2

ðuþ xyf x1� f 2þx2

f x2þ yx3Þ2þðvþ f 2þy2

f x1� xyf x2� xx3Þ2

!12

.

ð11Þ


Because the components of T appear in a ratio with depthin (10), translation and depth are recovered up to a com-mon scale factor, which is fixed when the essential param-eters are computed under a fixed norm constraint Eq. (1) ishomogeneous). Once depth is computed, the sign of T canbe adjusted, if necessary, to correspond to positive depth[28,2].

Note that the algorithm recovers a regularized estimateof depth because the estimated optical flow and essentialparameters from which it is computed (Eq. 11) are regular-ized solutions of functional minimization.

6. Experimental results

To evaluate the proposed method, we ran severalexperiments with real image sequences. Reconstructedobjects are displayed using triangulation-based surfacerendering with back-projection of the texture of the origi-nal input images. We also provide anaglyphs of stereo-

Fig. 3. (A) The first frame of the Berber sequence and the initial level set;(B) the computed segmentation; (C) the segmentation without the 3Dinterpretation term.

scopic images constructed from the estimated depth[25,39]. The results of the segmentation and optical flowestimation are shown and compared to those of othermethods. For each example, we experimented with differ-ent curve initializations by placing the curves so that theimage of a moving object is (a) completely inside a curve,(b) completely outside and, (c) partly inside/outside. In allthe examples, these initializations led to the same results.However, the algorithm convergence time varied withthe initialization.

This first example uses two consecutive frames of theBerber sequence of real images. The figurine rotates andmoves forward to the right. The scene surfaces are textured.The parameters are: m = 1, l = 103, and k = 10�2. The initialsegmentation curve is shown in Fig. 3A, superimposed on thefirst of the two frames used. The final segmentation is shownin Fig. 3B. Views of the reconstructed figurine, back-shadedusing the original texture, are shown in Figs. 4A–C. The esti-mated optical flow, and the optical flow computed by theHorn-and-Schunck and the Aubert–Deriche–Kornprobstalgorithms are displayed in Figs. 5A–C respectively. Themethods of Horn-and-Schunck and Aubert–Deriche–Kornprobst are benchmark variational methods which use,

Fig. 4. The reconstructed 3D structure of the moving figurine in theBerber sequence from different viewpoints.

A

B

C

Fig. 5. Optical flow estimated by (A) this method; (B) the Horn-and-Schunck method; (C) the Aubert–Deriche–Kornprobst method.

Fig. 6. (A) The first frame of Teddy1 sequence and the initial level set; (B)the computed segmentation; (C) the segmentation without the 3Dinterpretation term.

Fig. 7. The reconstructed 3D structure of the moving object from differentviewpoints.


respectively, isotropic and anisotropic regularization.Fig. 13A displays an anaglyph of a stereoscopic image con-structed from the first image and the estimated depth.

To evaluate the contribution of the 3D interpretationterm to the estimation of optical flow, we computed the dis-played frame difference (DFD) for this method and themethods of Horn and Schunck and Aubert, Deriche, Kor-nprobst. As can be seen in Table 1, the DFD for thismethod is the lowest. We also show the contribution of

Table 1The DFDs for the Berber sequence

Method DFD

Horn-and-Schunck 0.0345Aubert–Deriche–Kornprobst 0.0278This method 0.0123

Table 2The DFDs for the Teddy1 sequence

Method DFD

Horn-and-Schunck 0.0633Aubert–Deriche–Kornprobst 0.0428Method in [25] 0.0195This method 0.0205

A

B C

Fig. 8. Optical flow estimated by (A) this method; (B) the Horn-and- Schunck method; (C) the Aubert–Deriche–Kornprobst method.

Fig. 9. (A) The first of the two frames used of the Cylinder sequence with the initial curves; (B) the computed segmentation; (C) the segmentation withoutthe 3D interpretation term.



the 3D interpretation term to the segmentation. Fig. 3Cgives the segmentation obtained without the 3D interpreta-tion term. Although in one region the optical flow is consis-tent with a single rigid 3D motion, it is due to two differentmotions in the other (background and part of the object).

Fig. 10. The reconstructed 3D structure of the moving cylinder fromdifferent viewpoints.

Fig. 11. The reconstructed 3D structure of the moving box

By contrast, the 3D term in (5) constrained the optical flowto be consistent with a single rigid 3D motion in each seg-mentation region Fig. 3B.

In this second example (Teddy1 sequence, CarnegieMellon image database), the stuffed animal moves approx-imately laterally against a background which moves in theopposite direction. In contrast with the sequence of the pre-vious example, this sequence has large areas (on the object)with weak spatio-temporal variations, as well as smallareas without texture (eyes and nose tip of the object).The parameters in this experiment are set to m = 1,l = 0.5 · 103 and k = 10�2. The initial segmentation curveis shown in Fig. 6A superimposed on the first of the twoimages used in this example. The final segmentation isshown in Fig. 6B. Views of the reconstructed object sur-face, back-shaded using the original texture, are shown inFigs. 7A and B. Estimated optical flow, optical flow bythe Horn-and-Schunck algorithm and the Aubert–Deri-che–Kornprobst algorithm are displayed in Figs. 8A–C,respectively. Fig. 13B displays an anaglyph of a stereoscop-ic image constructed from the first image and the estimateddepth.

The DFDs for this method, the methods of Horn-and-Schunck and Aubert, Deriche, Kornprobst, andthe method in [25], are given in Table 2. The DFDs

shown with the background from different viewpoints.

A

B

C

Fig. 12. Optical flow estimated by (A) this method; (B) the Horn-and-Schunck method; (C) the Aubert–Deriche–Kornprobst method.


for this method and the method in [25] are about thesame and are lower than the other two. The segmenta-tion obtained without the 3D interpretation term isshown in Fig. 6C. Here, the segmentation is similar tothe one obtained with the inclusion of the 3D interpre-tation term.

This last example uses the Cylinder sequence (courtesyof S. Debrunner who constructed it and used it for sparse3D reconstruction [40]). This is also a sequence of realimages where a curved, cylinder-like object rotates aboutone degree per frame around a nearly vertical axis while

translating to the right with an image velocity of about0.14 pixel per frame. There is also a box moving right atan image rate of about 0.31 pixel per frame in front ofa flat backgroung also moving right at 0.14 pixel perframe. The parameter are m = 1, l = 104 and k = 10�1.The two initial segmentation curves are shown inFig. 9A, superimposed on the first of the two framesused. The challenge, here, is to segment the scene intotree regions because the 3D motion of box and back-ground are very close (they were considered as onemotion in [40]). The final segmentation is shown inFig. 9B. The reconstructed depth of the objects, triangu-lated and textured using the texture in the original images,is shown from different viewpoints in Figs. 10A–C and11A–C.

Note that these surfaces are smooth (regularizeddepth), and that the placement of boundaries is accurate.The estimated image motion by this method, by theHorn-and-Schunck algorithm, and the Aubert–Deriche–Kornprobst method are displayed in Figs. 12A–C,respectively. Fig. 13C displays an anaglyph of a stereo-scopic image constructed from the first image and theestimated depth.

The DFDs are shown in Table 3. As in the precedingexample, the 3D methods have similar DFDs, lowerthan the others. The segmentation without the 3D inter-pretation term is displayed in Fig. 9C. As in the firstexample, the exclusion of the 3D term from the formu-lation yields a segmentation where the optical flow esti-mate is not consistent with a single 3D motion in everyregion.

7. Conclusion

This paper investigated a level set method for the seg-mentation and 3D interpretation of the optical flow ofmultiple rigid objects moving independently in space.Estimation, segmentation, and 3D interpretation wereperformed jointly. This variational formulation saughtsimultaneously a segmentation into regions differing bytheir mean estimated optical flow, and a regularized esti-mate of optical flow consistent with both the Horn-and-Schunck optical flow constraint and a single rigid bodymotion in each region of the segmentation. The method,which allows both the viewing system and the viewedobjects to move, resulted in three steps interated untilconvergence: evolution of closed curves via level setsand, in region of segmentation, computation of essentialparameters of rigid motion by linear least squares, andestimation of optical flow consistent with a single rigidmotion. Following the segmentation, the translationaland rotational components of rigid motion and regular-ized relative depth are recovered analytically for eachregion of the segmentation from the estimated essentialparameters and optical flow. The validity of the methodhas been verified in several experiments with real imagesequences.

Fig. 13. Anaglyphs (to be viewed with red-blue glasses) of the sequences, (A) Berber; (B) Teddy1; and (C) Cylinder.

Table 3The DFDs for the Cylinder sequence

Method DFD

Horn-and-Schunck 0.0412Aubert–Deriche–Kornprobst 0.0325Method in [25] 0.0225This method 0.0275


Acknowledgment

This research was supported in part under NSERCGrant OGP0004234.

References

[1] K. Nakayama, J. Loomis, Optical velocity patterns, velocity-sensitiveneurons and space perception, Perception (3) (1974) 63–80.

[2] A. Mitiche, Computational Analysis of Visual Motion, Plenum Press,1994.

[3] J. Aggarwal, N. Nandhakumar, On the computation of motionfrom a sequence of images: A review, Proc. IEEE 76 (1988) 917–935.

[4] T. Huang, A. Netravali, Motion and structure from featurecorrespondences: A review, Proc. IEEE 82 (1994) 252–268.

[5] O. Faugeras, Three Dimensional Computer Vision: A GeometricViewpoint, MIT Press, 1993.

[6] A. Zisserman, R. Hartley, Multiple View Geometry in ComputerVision, Cambridge University Press, Cambridge, 2000.

[7] G. Adiv, Determining three-dimensional motion and structure fromoptical flow generated by several moving objects, IEEE Trans.Pattern Anal. Mach. Intell. 7 (4) (1985) 384–401.

[8] B. Horn, E. Weldon, Direct methods for recovering motion, Int. J.Comput. Vis. 2 (2) (1988) 51–76.

[9] S. Srinivasan, Extracting structure from optical flow using the fasterror search technique, Int. J. Comput. Vis. 37 (3) (2000) 203–230.

[10] Y. Hung, H. Ho, A kalman filter approach to direct depth estimationincorporating surface structure, IEEE Trans. Pattern Anal. Mach.Intell. 21 (6) (1999) 570–575.

[11] B. Shahraray, M. Brown, Robust depth estimation from optical flow,Int. Conf. Comput. Vis. (1988) 641–650.

[12] M. Taalebinezhaad, Direct recovery of motion and shape in thegeneral case by fixation, IEEE Trans. Pattern Anal. Mach. Intell. 14(8) (1992) 847–853.

[13] E. De Micheli, F. Giachero, Motion and structure from onedimensional optical flow, in: Proc. Internat. Conf. on ComputerVision and Pattern Recognition, 1994, pp. 962–965.

[14] N. Gupta, N. Kanal, 3D motion estimation from motion field, Artif.Intell. (78) (1995) 45–86.

[15] Y. Xiong S. Shafer, Dense structure from a dense optical flow, in:Proc. Internat. Conf. on Computer Vision and Image Understanding,1998, pp. 222–245.

[16] T. Brodsky, C. Fermuller, Y. Aloimonos, Structure from motion:Beyond the epipolar constraint, Int. J. Comput. Vis. 37 (3) (2000)231–258.

[17] H. Liu, R. Chellapa, A. Rosenfeld, A hierarchical approach forobtaining structure from two-frame optical flow, in: Proc. Workshopon Motion and Video Computing, 2002.

[18] D. Heeger, A. Jepson, Subspace methods for recovering rigidmotion I: algorithm and implementation, Int. J. Comput. Vis. 7 (2)(1992) 95–117.


[19] A. Bruss, B. Horn, Passive navigation, Comput. Graph. ImageProcess. (1983) 3–20.

[20] W. MacLean, A. Jepson, R. Frecher, Recovery of egomotion andsegmentation of independent object motion using the em algorithm,Proc. Br. Mach. Vis. Conf. BMVC 94 (1994) 13–16.

[21] S. Fejes, L. Davis, What can projections of flow fields tell us aboutvisual motion, Internat. Conf. on Computer Vision (1998) 979–986.

[22] J. Weber, J. Malik, Rigid body segmentation and shape descriptionfrom dense optical flow under weak perspective, IEEE Trans. PatternAnal. Mach. Intell. 19 (2) (1997) 139–143.

[23] H. Sekkati, A. Mitiche, Dense 3D interpretation of image sequences: avariational approach using anisotropic diffusion, Proc. Internat. Conf.on Image Analysis and Processing, Mantova, Italy, 2003, pp. 424–429.

[24] A. Mitiche, S. Hadjres, MDL estimation of a dense map of relativedepth and 3D motion from a temporal sequence of images, PatternAnal. Appl. (6) (2003) 78–87.

[25] H. Sekkati, A. Mitiche, Concurrent 3D-motion segmentation and 3Dinterpretation of temporal sequences of monocular images, IEEETrans. Image Process. (in press).

[26] R. Aubert, G. Deriche, P. Kornprobst, Computing optical flow viavariational techniques, SIAM J. Appl. Math. 60 (1) (1999) 156–182.

[27] G. Aubert, P. Kornprobst, Mathematical Problems in ImageProcessing, Springer, 2002.

[28] X. Zhuang, R. Haralick, Rigid body motion and the optical flowimage, Proc. First Int. Conf. on Artificial Intelligence Applications(1984) 366–375.

[29] H. Longuet-Higgins, K. Prazdny, The interpretation of a movingretinal image, Proc. R. Soc. London, B 208 (1981) 385–397.

[30] M. Rousson, R. Deriche, A variational framework for active andadaptive segmentation, Proc. IEEE Workshop Motion and VideoComputing (2002) 56–61.

[31] T. Chan, L. Vese, An active contour model without edges, Proc.Int. Conf. Scale-Space Theor. Comput. Vis. (1999) 141–151.

[32] C. Vazquez, A. Mitiche, I.B. Ayed, Image segmentation as regular-ized clustring: a fully global curve evolution method, Int. Conf. onImage Processing ICIP04 (2004) 3467–3470.

[33] A. Mansouri, A. Mitiche, C. Vazquez, Image partitioning by level setmultiregion competition, Int. Conf. on Image Processing, ICIP04(2004) 2721–2724.

[34] T. Chan, J. Shen, L. Vese, Variational PDE models in imageprocessing, Notices of the AMS 50 (1) (2033) 14–26.

[35] D. Cremers, S. Soatto, Motion competition: a variational approach topiecewise parametric motion segmentation, Int. J. Comput. Vis. 62 (3)(2005) 249–265.

[36] C. Vazquez, A. Mansouri, A. Mitiche, Approximation of images bybasis functions for multiple region segmentation with level sets, Int.Conf. on Image Processing, ICIP04 (2004) 549–552.

[37] G. Forsythe, M. Malcom, C. Moler, Computer Methods forMathematical Computations, Prentice-Hall, 1977.

[38] J. Sethian, Level Set Methods and Fast Marching Methods,Cambridge University Press, Cambridge, 1999.

[39] E. Dubois, A projection method to generate anaglyph stereo images,Proc. ICASP III (2001) 1661–1664.

[40] C. Debrunner, N. Ahuja, Segmentation and factorization-basedmotion and structure estimation for long image sequences, IEEETrans. Pattern Anal. Mach. Intell. 20 (2) (1998) 206–211.

Joint optical ﬂow estimation, segmentation, and 3D...

Documents

Transcript of Joint optical ﬂow estimation, segmentation, and 3D...