Generating Seamless Stereo Mosaics from Aerial Video
-
Upload
kay-robinson -
Category
Documents
-
view
37 -
download
1
description
Transcript of Generating Seamless Stereo Mosaics from Aerial Video
![Page 1: Generating Seamless Stereo Mosaics from Aerial Video](https://reader036.fdocuments.us/reader036/viewer/2022062314/56812d9b550346895d92b86d/html5/thumbnails/1.jpg)
04/19/23 1
Generating Seamless Stereo
Mosaics from Aerial Video
Zhigang ZhuAllen R. Hanson, Harpal S. Bassali
Howard Schultz, Edward M. Riseman
Computer Vision Lab
Computer Science Department
University of Massachusetts at Amherst
http://www.cs.umass.edu/~zhu
![Page 2: Generating Seamless Stereo Mosaics from Aerial Video](https://reader036.fdocuments.us/reader036/viewer/2022062314/56812d9b550346895d92b86d/html5/thumbnails/2.jpg)
04/19/23 2
Introduction
ObjectivesDevelop methods to automatically generate geo-referenced stereo mosaics from video sequences
DefinitionsFree Mosaic: composite of video sequence by registering overlapping frames
• subject to drift relative to the terrain• constrains: pure rotation or planar scenes
Geo-Mosaic: use 3D instrumentation to constrain mosaic to world coordinates Stereo Mosaics : a pair of mosaics from a single camera
• seamless under motion parallax• preserve 3D information• can be viewed in 3D directly
![Page 3: Generating Seamless Stereo Mosaics from Aerial Video](https://reader036.fdocuments.us/reader036/viewer/2022062314/56812d9b550346895d92b86d/html5/thumbnails/3.jpg)
04/19/23 3
Important Issues in 3D Video Mosaicing
Representationcompact representation for a large-scale 3D sceneorthogonal (DEM), perspective (mosaic), parallel-perspective
Computationhow expensive are the computations of the algorithms?Goal: affordable, efficient and robust mosaicing
Accuracyaccurate for 3D viewing and 3D reconstruction ?
![Page 4: Generating Seamless Stereo Mosaics from Aerial Video](https://reader036.fdocuments.us/reader036/viewer/2022062314/56812d9b550346895d92b86d/html5/thumbnails/4.jpg)
04/19/23 4
Geometry, Representation and Properties
Sensor motion is pure translation
Sensor
Image Plane
“Right” Mosaic “Left” Mosaic
![Page 5: Generating Seamless Stereo Mosaics from Aerial Video](https://reader036.fdocuments.us/reader036/viewer/2022062314/56812d9b550346895d92b86d/html5/thumbnails/5.jpg)
04/19/23 5
Re-Organizing the images….
![Page 6: Generating Seamless Stereo Mosaics from Aerial Video](https://reader036.fdocuments.us/reader036/viewer/2022062314/56812d9b550346895d92b86d/html5/thumbnails/6.jpg)
04/19/23 6
Recovering Depth from Mosaics
parallel-perspective stereo mosaics
Depth accuracy independent of depth
Two views from different perspective stereo
P(X,Y,Z)
Height H from Laser Profiler
GPS
)(y
y
y
y
d
ydH
d
BFZ
disparity
baseline
Fixed !
displacement
![Page 7: Generating Seamless Stereo Mosaics from Aerial Video](https://reader036.fdocuments.us/reader036/viewer/2022062314/56812d9b550346895d92b86d/html5/thumbnails/7.jpg)
04/19/23 7
Stereo mosaics of Amazon rain forest
166-frame telephoto video sequence -> 7056*944 mosaics
a
b
c
![Page 8: Generating Seamless Stereo Mosaics from Aerial Video](https://reader036.fdocuments.us/reader036/viewer/2022062314/56812d9b550346895d92b86d/html5/thumbnails/8.jpg)
04/19/23 8
Stereo viewing
Red: Right view; Blue/Green: Left view
![Page 9: Generating Seamless Stereo Mosaics from Aerial Video](https://reader036.fdocuments.us/reader036/viewer/2022062314/56812d9b550346895d92b86d/html5/thumbnails/9.jpg)
04/19/23 9
Computation: how expensive in in real world
application?
Pro-processingArbitrary motion other than a 1D translation / 3D translationCamera orientation estimation and image rectification
Mosaicing Image sequence is not dense enough for seamless mosaicshow to generate parallel-perspective projection ?
3D recoveryhow expensive is the match in stereo mosaics?Baseline and epipolar geometry of stereo mosaics
Post-processingparallel-perspective 3D mosaics to an orthogonal DEM
![Page 10: Generating Seamless Stereo Mosaics from Aerial Video](https://reader036.fdocuments.us/reader036/viewer/2022062314/56812d9b550346895d92b86d/html5/thumbnails/10.jpg)
04/19/23 10
Step 1. Pro-ProcessingMotion estimation & rectification
Camera pose estimation using navigation instrumentation and bundle adjustment
only sparse tie points widely distributed in the two images are needed
Image rectificationtransformation on two narrow slices in each frame
Dominant motion direction
3D path of the camera: 3D rotation + 3D translation
Original image frames
Dominant motion direction Y
Rectified image frames
Path of the camera: 3D translation
![Page 11: Generating Seamless Stereo Mosaics from Aerial Video](https://reader036.fdocuments.us/reader036/viewer/2022062314/56812d9b550346895d92b86d/html5/thumbnails/11.jpg)
04/19/23 11
Step 2. 3D Mosaicingseamless mosaicing with motion parallax
Front slit : Both slits are sub-images of m columns (m>=1)Rear slit
Perspective image
Multi-perspective mosaics
Parallel-perspective mosaics
Rays of left view
Left view mosaic
……
Rays of right view
Right view mosaic
……
Rays of right view
Right view mosaic
……
Rays of left view
Left view mosaic
……
View interpolationRay interpolation
(1)
(2)
In a multi-perspective projection mosaic, each sub-image is full perspective, but sub-images from different frames will have different viewpoints. This may cause seams in the mosaic due to motion parallax.
Geometric Seams -Clearly Visible to Human Eyes, especially along depth boundaries
Introduce Error in Height Estimation
![Page 12: Generating Seamless Stereo Mosaics from Aerial Video](https://reader036.fdocuments.us/reader036/viewer/2022062314/56812d9b550346895d92b86d/html5/thumbnails/12.jpg)
04/19/23 12
PRISM parallel ray interpolation for stereo mosaicing
)(
)2/(
21
1
yyiy
xxxi
yy
yyi
tts
stt
syy
dytt
2
)2
( 11
yyii
y
y
xxii
dty
dy
s
sxtx
Interpolated view:
Mosaic coordinates:
- Take a slice of certain width from each frame- Perform local registration between the overlapping slices- Generate parallel interpolated views between two known views- Re-project the point back to the mosaic
(Tx, Ty)
IP of 1st fixed line
1st fixed line
y0= dy/2
(Tx+Sx, Ty+Sy)
IP of 2nd fixed line
2nd fixed line
(X,Y,Z)
(x1, y1)
(x2, y2)
IP of interpolated fixed line
(Txi,Tyi)
(xi, yi)
![Page 13: Generating Seamless Stereo Mosaics from Aerial Video](https://reader036.fdocuments.us/reader036/viewer/2022062314/56812d9b550346895d92b86d/html5/thumbnails/13.jpg)
04/19/23 13
Comparison : 2D mosaic & 3D mosaic
2D mosaic from dense image sequence ( 4-pixel interframe motion)2D mosaic from sparse image sequence ( 40-pixel interframe motion)
3D mosaic from sparse image sequence ( 40-pixel interframe motion)
![Page 14: Generating Seamless Stereo Mosaics from Aerial Video](https://reader036.fdocuments.us/reader036/viewer/2022062314/56812d9b550346895d92b86d/html5/thumbnails/14.jpg)
04/19/23 14
Step 3. 3D reconstructionepipolar geometry of stereo mosaics
epipolar curveylr
lrrlxlr dyy
yyyybxx
),(
Epipolar curve1D search
Near horizontal line
Coverged pairsmall search region
xl
yl
left mosaic
xr
yr
right mosaic
dy/2
(xl,yl)
(xl,yl
)
txl(yl)
(xr,yr
)txr(yr)- txl(yl)dy/2
txr(yr)
![Page 15: Generating Seamless Stereo Mosaics from Aerial Video](https://reader036.fdocuments.us/reader036/viewer/2022062314/56812d9b550346895d92b86d/html5/thumbnails/15.jpg)
04/19/23 15
Epipolar Geometry in Real Stereo mosaics
“Left” Mosaic
![Page 16: Generating Seamless Stereo Mosaics from Aerial Video](https://reader036.fdocuments.us/reader036/viewer/2022062314/56812d9b550346895d92b86d/html5/thumbnails/16.jpg)
04/19/23 16
Epipolar Geometry in Real Stereo mosaics
“Right” Mosaic
![Page 17: Generating Seamless Stereo Mosaics from Aerial Video](https://reader036.fdocuments.us/reader036/viewer/2022062314/56812d9b550346895d92b86d/html5/thumbnails/17.jpg)
04/19/23 17
Epipolar Geometry in Real Stereo mosaics
Depth Map: Brighter is higher elevation
![Page 18: Generating Seamless Stereo Mosaics from Aerial Video](https://reader036.fdocuments.us/reader036/viewer/2022062314/56812d9b550346895d92b86d/html5/thumbnails/18.jpg)
04/19/23 18
Step 4. Post-Processing from 3D mosaic to DEM
Just a transformation !!
(X,Y,Z) world coordinates(xl,yl) left mosaic coordinates(xr,yr) right mosaic coordinates
H : a reference height (depth)F: focal length of the camerady: distance between left and right slitsby: adaptive baseliney = yr - yl
2 ,)1( lr
ly
yy
F
HYx
d
y
F
HX
)1(
yy
y
d
yH
d
bHZ
![Page 19: Generating Seamless Stereo Mosaics from Aerial Video](https://reader036.fdocuments.us/reader036/viewer/2022062314/56812d9b550346895d92b86d/html5/thumbnails/19.jpg)
04/19/23 19
3D Rendering Result
![Page 20: Generating Seamless Stereo Mosaics from Aerial Video](https://reader036.fdocuments.us/reader036/viewer/2022062314/56812d9b550346895d92b86d/html5/thumbnails/20.jpg)
04/19/23 20
Motion Refinement for Geo-Mosaic- when geo-data is not accurate
Extended Kalman Filter (EKF) approach
Unconstrained
-Image Match
-Accumulating error
Ground TruthGeo Meaursement
- absolute error EKF EsimationFlying path
Time Update
“Predict” from
image registration
Measurement Update
“correct”
by geo-data Motion parameter
A: Warp matrix t : translation
![Page 21: Generating Seamless Stereo Mosaics from Aerial Video](https://reader036.fdocuments.us/reader036/viewer/2022062314/56812d9b550346895d92b86d/html5/thumbnails/21.jpg)
04/19/23 21
Geo-Reference Mosaicsframe-by frame mosaicing
Geo-mosaic
Free Mosaic
![Page 22: Generating Seamless Stereo Mosaics from Aerial Video](https://reader036.fdocuments.us/reader036/viewer/2022062314/56812d9b550346895d92b86d/html5/thumbnails/22.jpg)
04/19/23 22
Geo-Reference Mosaicsglobal warping from free mosaic
Free mosaic
Geo-mosaic from free mosaic
square
straight
Frame-by-frame geo-mosaic
diamond
curve
![Page 23: Generating Seamless Stereo Mosaics from Aerial Video](https://reader036.fdocuments.us/reader036/viewer/2022062314/56812d9b550346895d92b86d/html5/thumbnails/23.jpg)
04/19/23 23
Methods Summarydistribute the computations in four steps
Pro-processingMotion estimation: sparse tie points distirbuted in entire frames
Rectification and Mosaicing (PRISM)
Process two narrow slices
3D recovery (Terrest)stereo match only in two mosaics
Post-processingjust a coordinate transformation
![Page 24: Generating Seamless Stereo Mosaics from Aerial Video](https://reader036.fdocuments.us/reader036/viewer/2022062314/56812d9b550346895d92b86d/html5/thumbnails/24.jpg)
04/19/23 24
Accuracy of 3D from Stereo Mosaics
Adaptive baselines and fixed disparity - uniform depth resolution
Ray interpolation between two successive views is similar to image rectification
3D recovery accuracy is comparable to that of a perspective stereo with an optimal baseline
![Page 25: Generating Seamless Stereo Mosaics from Aerial Video](https://reader036.fdocuments.us/reader036/viewer/2022062314/56812d9b550346895d92b86d/html5/thumbnails/25.jpg)
04/19/23 25
Next Steps
Camera calibration and bundle adjustmentsGeometric and Photometric SeamlessnessUsing Structure Information and video –photo matchingError analysis in geo-referenced mosaic and 3D reconstruction using ground truth