Motion from image and inertial measurements

Motion from image and inertial measurements

Dennis StrelowCarnegie Mellon University

Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 2

On the web

Related materials:these and related slidesrelated papersmoviesVRML models

at: http://www.cs.cmu.edu/~dstrelow/epson


Introduction (1)

From an image sequence, we can recover:6 degree of freedom (DOF) camera motionwithout knowledge of the camera’s surroundingswithout GPS


Introduction (2)

Fitzgibbon


Introduction (3)Potential applications include:

modeling from video

Yuji Uchida


Introduction (4)

micro air vehicles (MAVs)

AeroVironment Black Widow AeroVironment Microbat


Introduction (5)

rover navigation

Hyperion Nister, et al.


Introduction (6)

search and rescue robots

Rhex (movies: http://ai.eecs.umich.edu/Rhex/Movies)


Introduction (7)

NASA Personal Satellite Assistant (PSA)


Introduction (8)

For these problems, we want:6 DOF motionin unknown environmentswithout GPS or other absolute positioning


Introduction (8)

For these problems, we want:6 DOF motionin unknown environmentswithout GPS or other absolute positioningusing small, light, and cheap sensors


Introduction (8)

For these problems, we want:6 DOF motionin unknown environmentswithout GPS or other absolute positioningusing small, light, and cheap sensorsover the long term


Introduction (9)

Long-term motion estimation:absolute distance or time is longonly a small fraction of the scene is visible at any one time


Introduction (10)

given these requirements, cameras are promising sensors……and many algorithms for motion from images already exist


Introduction (11)

But, where are the systems for estimating the motion of:

over the long term?


Introduction (12)

…and for automatically modelingroomsbuildingscities

from a handheld camera?


Introduction (13)

Motion from images suffers from some long-standing difficulties

This work attacks these problems by…exploiting omnidirectional imagesexploiting image and inertial measurementsrobust image feature trackingrecognizing previously mapped locations


Outline

Motion from images refresher bundle adjustment difficultiesMotion from image and inertial measurementsRobust image feature trackingLong-term motion estimationConclusion


Motion from images: refresher (1)

A two-step process is common:sparse feature trackingestimation



A two-step process is common:sparse feature trackingestimation

Sparse feature tracking:inputs: raw imagesoutputs: projections



Template matching:correlation trackingLucas-Kanade (Lucas and Kanade, 1981)

Extraction and matching:Harris features (Harris, 1992)

Scale Invariant Feature Transform (SIFT) keypoints (Lowe, 2004)



The second step is estimation:inputs:

projectionsoutputs:

6 DOF camera position at the time of each image 3D position of each tracked point



bundle adjustment (various, 1950’s)

Kalman filtering (Broida, Chandrashekhar, and Chellappa, 1990)

variable state dimension filter (VSDF) (McLauchlan, 1996)

two- and three-frame methods(Hartley and Zisserman, 2000; Nister, et al. 2004)


Motion from images: bundle adjustment (1)

From tracking, we have the image locations xij for each point j and each image i



Suppose we also have estimates of:the camera rotation ρi and translation ti at time of each image3D point positions Xj of each tracked point

Then, we can compute reprojections:

))(( iji tXR



So, minimize:

with respect to all the ρi, ti, Xj

)),)(((,

image ijijji

i xtXRDE


Motion from images: difficulties (1)

Estimation step can be very sensitive to…incorrect or insufficient image feature tracking camera modeling and calibration errorsoutlier detection thresholdssequences with degenerate camera motions



Iterative batch methods have poor convergence or may fail to converge if:observations are missingthe initial estimate is poor



Recursive methods suffer from: poor prior assumptions on the motionpoor approximations in state error modeling



Resulting errors are: gross local errors long term drift



151 images, 23 pointsmanually corrected Lucas-Kanade



squares: ground truth points dash-dotted line: accurate estimate solid line: image-only, bundle adjustment estimate


Outline

Motion from imagesMotion from image and inertial measurements inertial sensors algorithms and results related workRobust image feature trackingLong-term motion estimationConclusion


Motion from image and inertial measurements: inertial sensors (1)

inertial sensors can be integrated to estimate six degree of freedom motion



But many applications require small, light, and cheap sensors



Integrating the outputs of these low grade sensors will produce drifting motion because of:noiseunmodeled nonlinearities



And, we can’t even integrate until we can separate the effects of…rotation ρgravity gacceleration aslowly changing bias banoise n

…in the accelerometer measurements



Image and inertial measurements are highly complementary

With inertial measurements we can:decrease sensitivity in image-only estimatesestablish two rotation angles without driftestablish the global scale



Image and inertial measurements are highly complementary

With inertial measurements we can:decrease sensitivity in image-only estimatesestablish two rotation angles without driftestablish the global scale

…even with our low-grade sensors



With image measurements, we can:reduce the drift in integrating inertial datadistinguish between…

rotation ρ gravity g acceleration a bias ba noise n

…in accelerometer measurements


Motion from image and inertial measurements: algorithms and results (1)

This work has developed both:batchrecursive

algorithms for motion from image and inertial measurements



Gyro measurements:

ω’, ω: measured and actual angular velocitybω: gyro bias

n: gaussian noise

nb '



Accelerometer measurements:

ρ: rotationa’, a: measured and actual accelerationg: gravity vectorba: accelerometer bias

n: gaussian noise

nbgaRa' aT )((



batch algorithm minimizes a combined error:

inertialimagecombined EEE



image term Eimage is the same as before



inertial error term Einertial is:

1

1n,translatio

1

1velocity,

1

1rotation,

ntranslatiovelocityrotationinertial

f

ii

f

ii

f

ii

E

E

E

EEEE




1

1n,translatio

1

1velocity,

1

1rotation,


f

ii

f

ii

f

ii

E

E

E

EEEE



,...))(,( 1n,translatio itii tItDE

timeτi-1 (time of image i - 1)

ti-1

ti

I(ti-1, …)

τi (time of image i)

tran

slat

ion

( : translation estimate for image i – 1)

( : translation estimate for image i)

( : translation integrated from previous estimate)



timeτ0

tran

slat

ion

τ1 τ2 τ5 τ3 τ4 τf-3 τf-2 τf-1

1

1n,translationtranslatio

f

iiEE



1

1n,translatio

1

1velocity,

1

1rotation,


f

ii

f

ii

f

ii

E

E

E

EEEE



It(τi-1, τi ,…, ti-1) depends on:

τi-1, τi (known)

all inertial measurements for times τi-1< τ < τi (known)ρi-1, ti-1gbω, bacamera linear velocities: vi



dash-dotted line: batch estimate from image and inertial solid line: image-only, bundle adjustment estimate squares: ground truth points



IEKF for the same sensors, unknowns dash-dotted line: batch estimate solid line: IEKF estimate



Difficulties with IEKF for this application:prior assumptions about motion smoothnesscannot model relative error between adjacent camera positions

So, converting the batch algorithm into a variable state dimension filter (VSDF) is a promising future direction



IEKF assumptions on motion smoothness dash-dotted line: batch estimate solid line: IEKF estimate

right: IEKF propagation variances too strict

left: IEKF propagation variances just right


Motion from image and inertial measurements

Recap:image, gyro, and accelerometer measurementsbatch algorithmrecursive algorithmexperiments

evaluate batch and recursive algorithms establish basic facts about motion from image and inertial measurements


Outline

Motion from imagesMotion from image and inertial measurementsRobust image feature tracking smalls in briefLong-term motion estimationConclusion


Robust image feature tracking: smalls in brief (1)

Lucas-Kanade has been the go-to feature tracker for shape-from-motion suitable for real-timesubpixel accuracygeneralheuristics for handling large image motions



Lucas-Kanade has been the go-to feature tracker for shape-from-motion suitable for real-timesubpixel accuracygeneralheuristics for handling large image motions…but not robust enough for “hands-free” motion estimation



smalls is a new feature tracker targeted at 6 DOF motion estimationcombines aspects of correlation tracking and “extract and match” trackersexploits the rigid scene assumptioneliminates the heuristics normally used with Lucas-KanadeSIFT is an enabling technology here



End analysis:allows hands-free SFM for many hard sequencescan still be defeated by repeated texture or lack of texture

Pointers to more information on smalls on the web page


Outline

Motion from imagesMotion from image and inertial measurementsRobust image feature trackingLong-term motion estimation proof of concept system experimentConclusion


Long-term motion estimation: proof of concept system (1)

Image-based motion estimates from any system will drift:if the features we see are always changinggiven sufficient timeif we don’t recognize when we’ve revisited a location



To limit drift:recognize when we’ve returned to a previous locationexploit the return

A proof of concept system demonstrates these capabilities



“smalls” tracker state: 2D feature history for images in I

variable state dimension filter (VSDF) state for images in I: 6 DOF camera positions, covariances for images in I 3D positions for features visible in I

SIFT keypoints for image in

system state S

image indices: I = {i1, …, in}



0 1 2 3 4 5 6 7 8

{0, 1}

{0} {0, 1, 2} {0, 1, …, 8}


rollback


0 1 2 3 4 5 6 7 8

{0, 1}

{0} {0, 1, 2} {0, 1, …, 8}

non-rollback States:


rollback


0 1 2 3 4 5 6 7 8

{0, 1}

{0} {0, 1, 2} {0, 1, …, 8}

8

non-rollback States:


rollback


0 1 2 3 4 5 6 7 8

{0, 1}

{0} {0, 1, 2}

8

{0, 1, 2, 3, 8}

non-rollback pruned States:


rollback


0 1 2 3 4 5 6 7 88 9 10 11

11 12 13 14

14 15 16 17

17 18 19 20

{0, …, 6, 11, 12, 17, …, 20}




When to “roll back”?examine the camera covariances for the current state and the candidate rollback state check the number of SIFT matchesextend from the candidate stateexamine the camera covariances for the current state and the resulting extended state


Long-term motion estimation: experiment (1)

CMU FRC highbay views; 945 images total



CMU FRC highbay



CMU FRC highbay

(first forward pass: images 0-213)



CMU FRC highbay




CMU FRC highbay

(first backward pass: images 214-380)



CMU FRC highbay

(second forward pass: images 381-493)



CMU FRC highbay

(second backward pass: images 494-609)



CMU FRC highbay

(third forward pass: images 610-762)



CMU FRC highbay

(third backward pass: images 763-944)


rollback


0 1 2 3 4 5 6 7 88 9 10 11

11 12 13 14

14 15 16 17

17 18 19 20


normally, the system produces a general tree of states



…0 1 2 3 4 5 6 713 14 15 14

16 17 18 17

non-rollback rollback pruned States:

for this example, the “rollback” states are restricted to the first forward pass

8 9

10 11 12 14

14213



movie…bottom half is smalls output:



movie…top half is motion estimates:


Outline

Motion from imagesMotion from image and inertial measurementsRobust image feature trackingLong-term motion estimationConclusion remaining issues some previous work


Conclusion: remaining issuesall: system is experimental, not optimized for speedimage and inertial: VSDF“smalls”: integration of gyro, more robustness to poor texture neededlong-term: “roll back” space, computation grow with sequence length


Conclusion: some previous work (1)1998-99 (CMU): trinocular stereo for Honda humanoid and Toyota highway obstacle detection1996-1998 (K2T, Inc.): architectural models from still images1996 (U. of Illinois): Masters thesis, visualizing fMRI data with virtual reality


Conclusion: some previous work (2)1995 (Los Alamos): automatically delineating rib cage volumes in CT volumes1994 (National Solar Observatory): tracking sunspot motion, measuring solar flare intensity1993 (U. of Nebraska): AVHRR satellite image restoration


Thanks!

Related materials:these and related slidesrelated papersmoviesVRML models

at:http://www.cs.cmu.edu/~dstrelow/epson

Motion from image and inertial measurements

Documents

Transcript of Motion from image and inertial measurements