Slides: Real-Time Spherical Videos from a Fast Rotating Camera
-
Upload
frank-nielsen -
Category
Art & Photos
-
view
136 -
download
1
description
Transcript of Slides: Real-Time Spherical Videos from a Fast Rotating Camera
Real-Time Spherical Videos from a Fast Rotating Camera
Frank Nielsen, Alexis Andre and Shigeru Tajima
Sony Computer Science Laboratories Inc, Japan
Ecole Polytechnique, France
June 26th , 2008
Outline• Problem• Related Work• Two Prototypes:
–Line Sensor•Workflow•Results
–Area Sensor•Workflow•Results
Three core problems
• We want to generate (1) full spherical (2) videos at (3) high framerates (>25fps).
Main Contribution
• Framework for acquisition of full spherical videos with a special tailored fast-rotating camera
• High quality smooth image (no seams)
Problem 1: Full-Spherical Images
• Two traditional ways:–Stitching of a small number of images
taken with a fish eye lens (Quicktime VR® )
–Succession of 1-pixel wide vertical strips (Spheron™ )
Inherent problems
• Use of step motors for automatic acquisition– Slow– Need to rewind (avoid cable twist)
• Rough calibration is ok: a slight seam is acceptable (for discrete stitch)
Problem 2: Full-Spherical Videos
• Use of multiple cameras (Sony Fourth View)– Parallax problems
• Virtual panoramic videos (Agarwala et al.)– Wide field of view– Very low frame rate (wrt. the rotation speed)
Our approach
• Unique camera, rotating very fast
(1800rpm – 30fps)– No parallax problems (MCOP image)– No calibration between various cameras
• Connection– Slip rings -> no need for a step motor– Gigabit Ethernet connection with remote control
Challenge
• 1D rotation angles unknown:– Register the data: we need to find the
rotation angle:
– Then, extract frames every revolution
θ(t) ?
θ(t) = t mod(2π)
System Setup
Slip rings for Ethernet Connection
Camera Platform
Movie No parallax barrier
Line Area Sensor
• Black and white CCD • very fast sensor (67000 lines per second)• Smooth spherical image
(not unique center of projection, MCOP)
-> calibration not required
• Only 1/3 of the vertical axis (lens/sensor)• Choosing appropriate lens (tailored lens)
Raw Data: Plain unrolling
Acquisition speed may not be constant
Raw Data
Raw Data
One round One Round
Frame Registration
• Based on current data and past data
• We look for similar bands of pixels
(Dynamic programming-like search)– Only works if most of the scene is static
Registration
SSD
Score
ReferencePoint
CurrentPoint
Registration
SSD
Score
ReferencePoint
CurrentPoint
Registration
SSD
Score
ReferencePoint
CurrentPoint
Registration
SSD, or normalized mutual information, etc.
Registration
Minimum: One round!
Registration
Best Match
ReferencePoint
CurrentPoint
Registration
NextPoint Search Zone (expand)
Registration: Frame Extraction
Yields potential for super-resolution (non-uniform sampling)
Result movie (cropped vertically)
Results
• Rotation speed might not be uniform across one revolution:– Slight jittering/tripod
• Changes in speed affect resolution– In a production setting, speed must be controlled
carefully.
Issues
• Not robust to dynamic scenes
• Fails on homogenous scenes (ambiguity)
• Use of a small visual hint help registration– Bright LED at a specific position (visual pollution)
Area Sensor
• Align the nodal point with the rotation axis to avoid parallax (similar to spherical images problems)
• 24-bit color VGA frames• horizontal resolution fixed• need to control shutter time and/or rotation
speed to get enough overlap
Key difference: Camera can be calibrated exactly on its nodal point(multi-camera cannot)
“Traditional Approach”
• Stitch frames: need to register each frame with the previous ones
• Horizontal shift depends on the rotation speed and on the shutter time
• But resolution is fixed (programmable ROI)
Workflow for 2D stitching
Registration for two frames
Defish (=map to env. map.)
Registration: crude w/o blending
Actually perform only this in coordinate systems (no intermediate images)
Obvious seam
Smooth blending
Better methods: Laplacian multi-spectral, Poisson blending, etc.(but we are concerned with real-time systems…)
Panoramic roll: never-ending sequence
Artefact comes from the model of fish-eye projection. Calibrate rayel!
Results (movie)
Inherent Problems
• Lens imperfections / resolution decrease near the edges of the lens
• Perfect fisheye model or calibration is needed• Motion Blur
Imperfect Fisheye Model
• Slight seam– On static images: can be ignored
(error propagated to the edges of the image)– On movies: disturbing moving seam
Motion Blur
• Lighting conditions force us to use long exposure time (wrt. the hardware):– Motion blur appears– Ideally horizontal motion blur– At the desired speeds, unrecoverable blur
(horizontal) Motion blur
0rpm 150rpm 250rpm
Summary
• Path to high-frame rate full-spherical videos is promising– Better hardware is needed– H/W technologies is already here
• Classical framework– Established techniques– Optimization available
Future Work
• Horzontal deblur
• Registration in more dynamic scenes
• ROI selection
• Applications for AI applications
ETVC08Emerging Trends in
Visual Computing
Information geometry
Computer vision
Computer graphics
Machine learning
Computational Geometry
Registration is free