Post on 25-Aug-2018
Computational Photography and Capture:
Time-Lapse VideoAnalysis & Editing
Gabriel Brostow & Tim Weyrich
TA: Frederic Besse
Week Date Topic Hours
1 12-Jan Introduction to Computational Photography and Capture 1
1 14-Jan Intro + More on Cameras, Sensors and Color 2
2 19-Jan No lecture! (Go capture bracketed photos?) -
2 21-Jan Blending, Compositing, Poisson Editing 2
3 26-Jan Time-Lapse and Seam Carving 1
3 28-Jan Warping, Morphing, Mosaics and Panoramas 2
4 02-Feb High-Dynamic-Range Imaging and Tone Mapping 1
4 04-Feb Hybrid Images, Flash and Multi-Flash Photography 2
5 09-Feb Colourisation and Colour Transfer 1
5 11-Feb Image Inpainting and Texture Synthesis 2
7 23-Feb Video Based Rendering of Scenes I 1
7 25-Feb Video Based Rendering of Scenes II 2
8 02-Mar Video Texture Synthesis 1
8 04-Mar Video Sprites 2
9 09-Mar Deblurring/Dehazing and Coded Aperture Imaging 1
9 11-Mar Image-based Rendering 2
10 16-Mar Motion Capture guest lecture by Doug Griffin
10 18-Mar Capturing Geometry with Active Lighting 2
11 23-Mar Intrinsic Images 1
11 25-Mar Dual Photography and Reflectance Analysis 2
Today’s schedule
• Computational Time-Lapse Video
– Bennett & McMillan, Siggraph 2007
• Factored Time-Lapse Video
– Sunkavalli, Matusik, Pfister, Rusinkiewicz
Siggraph 2007
• Video Synopsis & Indexing,
– Pritch, Rav-Acha, Peleg, ICCV 2007
Time-lapse Photography
• Definition: when frames are captured at a lower rate than the rate at which they will ultimately be played back.
• Compare to
– Slow-motion
– Bullet-time
Time-lapse Filmmaker’s Challenges
Time-lapse Filmmaker’s Challenges
• Lighting changes (strobing)
• High frequency motion (missed action)
– Saturation & exposure
• Camera motion: intentional or not
• Triggering / data storage / camera safety
• Useful? Editable?
Does the old definition of time-lapse still hold?
(Quick Overview of)Computational Time-Lapse Video
Bennett & McMillan
Siggraph 2007
(Project web page)
Virtual Shutter
Sampling in 1D
• Uniform
• Interpolate linearly+ piece-wise
• Same # of samples, but where needed!
• Optimum polygonal approximation of digitized curves, Perez & Vidal, PRL’94
– Use dynamic programming to find global min.
Completely User-controllable:Min-Change, Min-Error, Median, etc
Factored Time-Lapse Video
Sunkavalli, Matusik, Pfister, Rusinkiewicz
SIGGRAPH 2007
Project web page
Outdoors: More Than Just Color vs Illumination?
Original No Shadows
Given:Daytime, under clear-sky conditions
Outdoors: More Than Just Color vs Illumination?
Original No Shadows
Given:Daytime, under clear-sky conditions
Wishlist
Remove shadowsRender new shadows
Change albedoChange global illumination
Outdoors: More Than Just Color vs Illumination?
Original No Shadows
Lighting due to sun? sky?
Surface Normals?
Formulation
• size: width x height x time
• F: xyt volume of frames over time
• Isky: Accumulated intensity due to sky-light
• Isun: Accumulated intensity due to sun-light
• Ssun: Binary; is a pixel in shadow
Intuition
Separate Sky first
• Photoshop!
– Just to pick “non-surfaces”
• When is sun’s contribution == 0?
– Leaving just sky’s contribution
• “Definitely” shadow: median of darkest 20%, then threshold at 1.5x
Bilateral Filter “Definitely” Shadow
How Did Skylight Change?
• Isky: Accumulated intensity due to sky-light
• Wsky: Sky-light image
• Hsky: Sky-light basis curve (1D function!!)
– Whole scene got brighter/darker together
• Blue: skylight curve (same one)
Factorization
• Decompose appearance into
– per-pixel W
– H curve
• Alternating Constrained Least Squares (ACLS):
– H(t) is held fixed while W is optimized using least squares, then vice versa
ACLS: Alternating Constrained Least Squares
• Inverse shade trees for non-parametric material representation and editing,
– Lawrence et al., Siggraph 2006, code online
Original and Isky
Isky and Reconstruction
So Far - Explained Sky
Now Decompose Sun’s Contributions
• Isun: Accumulated intensity due to sun-light
• Wsun: Sun-light image, per-pixel weights
• Hsun: Sun-light basis curve
• : Shift-map; per pixel offset in time
• Isun: Accumulated intensity due to sun-light
• Wsun: Sun-light image, per-pixel weights
• Hsun: Sun-light basis curve
• : Shift-map; per pixel offset in time
• Isun: Accumulated intensity due to sun-light
• Wsun: Sun-light image, per-pixel weights
• Hsun: Sun-light basis curve
• : Shift-map; per pixel offset in time
Total sunlight contribution
Sunlight image
(like on moon)Shift map Sunlight
Factorize Again!
• For each image, subtract off Isky(t)
• Ssun: Sky-light mask, serves as C, confidence map
• ACLS again
– But add 3rd update phase, searching for
that minimizes reconstruction error
of scaled + offset H(t) vs. F(t)
Green: sunlight curve
(same one, but time-shifted)
Isun(t)
Wsun
Reconstruction
Does It Work?
Video: FactoredTimeLapseV
What have we gained?
• Can
– Remove shadows
– Change albedo
– Change amount of global illumination
• Compression
• Render new shadows:
– We have 1 component of the per-pixel Normals, N
Estimated Normals (1D)
Manipulating Normals
(View Results of)Video Synopsis & Indexing
Pritch, Rav-Acha, Peleg
ICCV 2007, PAMI 2008
(Project web page)
HDR Time-Lapse
• Essentially started in 2006 (manually at first)
• More are popping up online