1 DIRS Meeting, February 8 th, 2008 Prudhvi Gurram 3D Scene Reconstruction from Aerial Video Prudhvi...
-
Upload
gwen-perry -
Category
Documents
-
view
220 -
download
0
Transcript of 1 DIRS Meeting, February 8 th, 2008 Prudhvi Gurram 3D Scene Reconstruction from Aerial Video Prudhvi...
1
DIRS Meeting, February 8th, 2008Prudhvi Gurram
3D Scene Reconstruction from Aerial Video
Prudhvi Gurram, Eli Saber, Harvey RhodyChester F. Carlson Center for Imaging Science
Rochester Institute of Technology
DIRS Group Meeting
February 8th, 2008
2
DIRS Meeting, February 8th, 2008Prudhvi Gurram
Outline
Motivation Block Diagram Preprocessing Ray Interpolation Object Modeling Conclusions and Future Work
3
DIRS Meeting, February 8th, 2008Prudhvi Gurram
Outline
Motivation Block Diagram Preprocessing Ray Interpolation Object Modeling Conclusions and Future Work
4
DIRS Meeting, February 8th, 2008Prudhvi Gurram
Motivation Objective
Extraction of 3D geometry of a scene from multi-modal data sets Possible Approaches
Manual Interpretation of Stereo Imagery (Very intensive and time consuming for large areas in the order of days or even months)
Automated processing of video frames to build stereo mosaics for the extraction of 3D geometry
Combine this with information from LIDAR to improve the accuracy of the 3D Scene.
Combine the 3D coordinates with material properties from Hyperspectral imaging to render a 3D Scene which conforms both geometrically and radiometrically to real world
5
DIRS Meeting, February 8th, 2008Prudhvi Gurram
High-Resolution VideoHigh-Resolution VideoLidar DataLidar Data
Spectral ImagerySpectral Imagery
Spectrally-Accurate Spectrally-Accurate Scene ModelScene Model
Motivation
Rapidly construct radiometrically-correct scene models based on
multi-sensor data for use in DIRSIG synthetic scene generation
6
DIRS Meeting, February 8th, 2008Prudhvi Gurram
Outline
Motivation Block Diagram Preprocessing Ray Interpolation Object Modeling Conclusions and Future Work
7
DIRS Meeting, February 8th, 2008Prudhvi Gurram
Video Frames
Pre-processing of the video frames
Ray InterpolationObject Extraction
and Modeling 3D Structure
Exterior Orientation (EO) and Interior Orientation (IO)
parameters
Orientation-corrected video frames
Stereo Mosaics
Block Diagram
8
DIRS Meeting, February 8th, 2008Prudhvi Gurram
Outline
Motivation Block Diagram Preprocessing Ray Interpolation Object Modeling Conclusions and Future Work
9
DIRS Meeting, February 8th, 2008Prudhvi Gurram
Pre-processing of Video Frames Correct the orientation of the frames so that all the frames have same
orientation as the first frame. Observed motion parallax of objects is due to translational motion of
camera only. Effects of translational motion of camera in Z direction are lost during
digitization process.
)()( TPRTPRRRP worldworld
A world point can be expressed in camera coordinate system with Rotation matrix R and camera center at T as
worldP
10
DIRS Meeting, February 8th, 2008Prudhvi Gurram
Pre-processing (Contd…)
)( 11TPKP worldim
Considering the first frame as reference frame (rotation matrix is an identity matrix)
The image coordinates in any frame i are transformed by matrix A, to observe only translational motion in the sensor
)(11'iworldimiimim TPKPKKRAPP
iii
TPKR
k
ky
kx
P worldim
The image coordinates of this point are given by (Interior Orientation parameters are embedded in matrix K
11
DIRS Meeting, February 8th, 2008Prudhvi Gurram
Outline
Motivation Block Diagram Preprocessing Ray Interpolation Object Modeling Conclusions and Future Work
12
DIRS Meeting, February 8th, 2008Prudhvi Gurram
Introduction
Why are we using Parallel Ray Interpolation? To convert the view from perspective view to parallel-perspective view To use motion parallax information (while creating mosaics) To make the stereo mosaics seamless
Perspective view Parallel view
13
DIRS Meeting, February 8th, 2008Prudhvi Gurram
Introduction - AnimatedPerspective ViewParallel ViewParallel View from Perspective View
Using Fixed Lines
14
DIRS Meeting, February 8th, 2008Prudhvi Gurram
Ray InterpolationViewpoint 2Viewpoint 1
InterpolatedViewpoint
Image (Mosaic) Plane
Point in the image planefrom viewpoint 1
Point in the image planefrom viewpoint 2
Point in the image plane from the interpolated viewpoint
Acknowledgement:
Zhigang Zhu et al.,
City College of New York,
New York City, NY
15
DIRS Meeting, February 8th, 2008Prudhvi Gurram
Fast PRISM Algorithm
Block Diagram for building Left or Right Stereo Mosaic
Fixed Line
Fixed Line
Matching CurveControl Points
PRISM
Matching CurveControl Points
Stitching CurveControl Points
on Mosaic
Fixed LineControl Points
Fixed LineControl Points
Global Transformation
Global Transformation
Fixed LineControl Points
on Mosaic
Fixed LineControl Points
On Mosaic
DestinationTriangles
SourceTriangles
SourceTriangles
DestinationTriangles
AffineTransformation
AffineTransformation
Stitched SlicesIn Mosaic
Frame K
Frame K+1
Preparing Input Imagesfor Geometric Transformation
Geometric Transformations
Transformed Image DataInput Video FramesColor Code
16
DIRS Meeting, February 8th, 2008Prudhvi Gurram
Geometry
Frame 1 Frame 2
17
DIRS Meeting, February 8th, 2008Prudhvi Gurram
Geometry
Fixed Lines
Image Frame
18
DIRS Meeting, February 8th, 2008Prudhvi Gurram
Geometry
Frame 1:
Frame 2:
Fixed Line
Fixed Line
Overlapped Region
19
DIRS Meeting, February 8th, 2008Prudhvi Gurram
Geometry
Frame 1:
Frame 2:
Source Triangles
20
DIRS Meeting, February 8th, 2008Prudhvi Gurram
Interpolating points along the stitching line
Viewpoint 2Viewpoint 1InterpolatedViewpoint
Image (Mosaic) Plane
Point in the image planefrom viewpoint 1
Point in the image planefrom viewpoint 2
Point in the image plane from the interpolated viewpoint
21
DIRS Meeting, February 8th, 2008Prudhvi Gurram
GeometryDestination Trianglesin the Left Mosaic
22
DIRS Meeting, February 8th, 2008Prudhvi Gurram
Motion Parallax
Frame 1 Frame 2Interpolated Frame(before triangulation)
23
DIRS Meeting, February 8th, 2008Prudhvi Gurram
Interpolation
Frame 1 Frame 2Overlay of Frames 1 and 2Interpolation
24
DIRS Meeting, February 8th, 2008Prudhvi Gurram
Interpolation
Frame 1 Frame 2Interpolation
25
DIRS Meeting, February 8th, 2008Prudhvi Gurram
Triangulation Problem
Frame 1 Frame 2Interpolated Frame(after triangulation)
26
DIRS Meeting, February 8th, 2008Prudhvi Gurram
Triangulation Artifacts
Before After
27
DIRS Meeting, February 8th, 2008Prudhvi Gurram
Modified Triangulation
Make sure that none of the triangles include regions with different motion parallax
Find edges of different regions and align the sides of triangles with the edges
But aerial images are too noisy to obtain continuous edges around different objects
Use segmentation The inner boundary of each segment forms an edge
of a region/object
28
DIRS Meeting, February 8th, 2008Prudhvi Gurram
Overlapped region
Frame 1:
Frame 2:
29
DIRS Meeting, February 8th, 2008Prudhvi Gurram
Segmented images
Segmented Frame 1:
Segmented Frame 2:
30
DIRS Meeting, February 8th, 2008Prudhvi Gurram
Frame k
Segments in Overlapped Region
One of the segments
Significant points using Convex Hull around the segment
Triangulation
31
DIRS Meeting, February 8th, 2008Prudhvi Gurram
Frame k+1Matching curve
The other part of the segment between matching curve and fixed line
Triangulation
32
DIRS Meeting, February 8th, 2008Prudhvi Gurram
Mosaic
“Orphan” Pixels Orphan pixels filled
• Using a constraint inherent in the Parallel-Perspective view
• Parallel view in dominant motion direction and Perspective view in the other direction
• Do not consider motion parallax along x direction
X
Y
Direction
33
DIRS Meeting, February 8th, 2008Prudhvi Gurram
Results – Set 1Motion
Parallax
Frame 1 Frame 2
Fast PRISM Modified PRISM
34
DIRS Meeting, February 8th, 2008Prudhvi Gurram
Results – Set 2
Fast PRISM
Modified PRISM
Visual Artifact
Corrected by our method
DIR cm
X 0.1
Y 4
Z 0
X
Y
Camera Motion
Direction
Z
35
DIRS Meeting, February 8th, 2008Prudhvi Gurram
Results – Set 3
Visual artifact Corrected by our method
RIT imagery – Collected by WASP Lite at an altitude of 3000ft and Frames captured at 3Hz frequency
1 8 14 39 48 78
Frames
36
DIRS Meeting, February 8th, 2008Prudhvi Gurram
Stereo Mosaic – Fast PRISM
37
DIRS Meeting, February 8th, 2008Prudhvi Gurram
Stereo Mosaic – Modified PRISM
Publications:1. P. Gurram, E. Saber, and H. Rhody, “A Novel Triangulation Method for Building Parallel-Perspective
Stereo Mosaics”, Proceedings of Electronic Imaging Symposium, SPIE, San Jose, CA, January 2007.2. P. Gurram, E. Saber, and H. Rhody, “Segment-based Mesh Design for Building Parallel-Perspective
Stereo Mosaics”, To be submitted to IEEE Transactions on Image Processing
38
DIRS Meeting, February 8th, 2008Prudhvi Gurram
Outline
Motivation Block Diagram Preprocessing Ray Interpolation Object Modeling Conclusions and Future Work
Stereo Geometry
39
DIRS Meeting, February 8th, 2008Prudhvi Gurram
Image Courtesy:Z. Zhu, A. Hanson and E. Riseman, “Generalized Parallel-Perspective Stereo Mosaics from Airborne Video,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 26, pp. 226 – 237, 2004.
Find disparity between the points
lr yy ,
lr yyy Disparity
)1(yd
yHZ
Distance from the
Sensor to the point
HZ
Height of the point from ground
yd
yHh
– the coordinate of any pixel in the right and left mosaics in the dominant motion direction respectively
– the baseline of the stereo mosaicsyd
41
DIRS Meeting, February 8th, 2008Prudhvi Gurram
3D from Passive Video
Left mosaic, right mosaic and Nadir mosaic are built by controlling the angle of the parallel view of the mosaics
1 5 10 15 20 25 30
Image Plane
Scene
Viewpoints
Left mosaic Nadir mosaic Right mosaic
Sensor motion
42
DIRS Meeting, February 8th, 2008Prudhvi Gurram
Object extraction and modeling
Nadir mosaic Segmentation
Tree regions
Building surfaces extraction using height information
Each surface
Boundary of each surface
Right mosaic
Left mosaic
Plane fit for each surface using disparity between
mosaics
Corners through Curvature Scale Space
Edges through line fit between corners
CAD model of each building
DTM
Digital Elevation Model (DEM)
Reconstructed scene
100 200 300 400 500 600 700 800 900 1000 1100
50
100
150
200
250
300
350
400
450
500
43
DIRS Meeting, February 8th, 2008Prudhvi Gurram
Results
… …Raw Video Frames
1 2 20 29
After Pre-processing
… …
Video collected over Center for Imaging Science,RIT using WASP LT at an altitude of 1000 ft and with an overlap of about 90% to 98% between frames
44
DIRS Meeting, February 8th, 2008Prudhvi Gurram
Stereo Mosaics
Nadir
Left Right
Stereo Pair
45
DIRS Meeting, February 8th, 2008Prudhvi Gurram
Object extraction
100 200 300 400 500 600 700 800 900 1000 1100
50
100
150
200
250
300
350
400
450
500
Nadir Mosaic Tree regions
Segmented Nadir Mosaic
46
DIRS Meeting, February 8th, 2008Prudhvi Gurram
Object modeling
Noise
Solar shadow
Problems in 3D model of a
building due to solar shadow and noise in
images
Hypotheses of symmetry in the building and flat surfaces on the
roof of the building
Reconstructed building from different perspectivesPublications:
1. P. Gurram, E. Saber, and H. Rhody, “A Novel Triangulation Method for Building Parallel-Perspective Stereo Mosaics”, Proceedings of Electronic Imaging Symposium, SPIE, San Jose, CA, January 2007.
2. P. Gurram, S. Lach, E. Saber, H. Rhody, and J. Kerekes, “3D Scene Reconstruction through a Fusion of Passive Video and Lidar Imagery”, Proceedings of 36th Applied Imagery and Pattern Recognition Workshop (AIPR'06), Washington, D. C., 2006
47
DIRS Meeting, February 8th, 2008Prudhvi Gurram
Situation 1: Edges not Visible in Video
Solar shadow
Noise
There is no information in these cases as one planar surface merges with a neighboring surface at a different height during segmentation
Video
Lidar
Raw Lidar CAD Model from Lidar
Good Edges and Planes
48
DIRS Meeting, February 8th, 2008Prudhvi Gurram
Situation 2: Lidar Data is Undersampled
The reconstruction begins to break down• Edges misrepresented or missed altogether
• Segmentation Fails
Video
Good Edges, Corners, and Planes
Lidar
49
DIRS Meeting, February 8th, 2008Prudhvi Gurram
Situation 3: Textureless Non-planar Surfaces
No variation of disparity on the textureless surface
Fit a spherical model to the data using Levenberg-Marquardt algorithm
Video
Lidar
RIT Observatory
50
DIRS Meeting, February 8th, 2008Prudhvi Gurram
Outline
Motivation Block Diagram Preprocessing Ray Interpolation Object Modeling Conclusions and Future Work
51
DIRS Meeting, February 8th, 2008Prudhvi Gurram
Conclusions and Future Work
Verification process is only as accurate as camera parameters are.
Model is being overfitted to the data. Optimize the model according to the
uncertainty in data using Bayesian networks.
Evidence from other modalities like lidar data can be used to refine the model.
52
DIRS Meeting, February 8th, 2008Prudhvi Gurram
Bayesian or Belief network
Image Understanding
Algorithms(Visual and Lidar)
Control System
Belief Network
53
DIRS Meeting, February 8th, 2008Prudhvi Gurram
An example of belief network
Region
Buildings Trees Terrain
Feature 1
Feature 2Feature 3
Feature 4 Feature 5
Feature op.
Feature op. Feature op.
Feature op. Feature op.
Known StructureKnown DataLearning from data
54
DIRS Meeting, February 8th, 2008Prudhvi Gurram
Thank you!