Computational Photography and Capture - UCL€¦ · Computational Photography and Capture:...

50
Computational Photography and Capture: Time-Lapse Video Analysis & Editing Gabriel Brostow & Tim Weyrich TA: Frederic Besse

Transcript of Computational Photography and Capture - UCL€¦ · Computational Photography and Capture:...

Page 1: Computational Photography and Capture - UCL€¦ · Computational Photography and Capture: Time-Lapse Video Analysis & Editing Gabriel Brostow & Tim Weyrich TA: Frederic Besse

Computational Photography and Capture:

Time-Lapse VideoAnalysis & Editing

Gabriel Brostow & Tim Weyrich

TA: Frederic Besse

Page 2: Computational Photography and Capture - UCL€¦ · Computational Photography and Capture: Time-Lapse Video Analysis & Editing Gabriel Brostow & Tim Weyrich TA: Frederic Besse

Week Date Topic Hours

1 12-Jan Introduction to Computational Photography and Capture 1

1 14-Jan Intro + More on Cameras, Sensors and Color 2

2 19-Jan No lecture! (Go capture bracketed photos?) -

2 21-Jan Blending, Compositing, Poisson Editing 2

3 26-Jan Time-Lapse and Seam Carving 1

3 28-Jan Warping, Morphing, Mosaics and Panoramas 2

4 02-Feb High-Dynamic-Range Imaging and Tone Mapping 1

4 04-Feb Hybrid Images, Flash and Multi-Flash Photography 2

5 09-Feb Colourisation and Colour Transfer 1

5 11-Feb Image Inpainting and Texture Synthesis 2

7 23-Feb Video Based Rendering of Scenes I 1

7 25-Feb Video Based Rendering of Scenes II 2

8 02-Mar Video Texture Synthesis 1

8 04-Mar Video Sprites 2

9 09-Mar Deblurring/Dehazing and Coded Aperture Imaging 1

9 11-Mar Image-based Rendering 2

10 16-Mar Motion Capture guest lecture by Doug Griffin

10 18-Mar Capturing Geometry with Active Lighting 2

11 23-Mar Intrinsic Images 1

11 25-Mar Dual Photography and Reflectance Analysis 2

Page 3: Computational Photography and Capture - UCL€¦ · Computational Photography and Capture: Time-Lapse Video Analysis & Editing Gabriel Brostow & Tim Weyrich TA: Frederic Besse

Today’s schedule

• Computational Time-Lapse Video

– Bennett & McMillan, Siggraph 2007

• Factored Time-Lapse Video

– Sunkavalli, Matusik, Pfister, Rusinkiewicz

Siggraph 2007

• Video Synopsis & Indexing,

– Pritch, Rav-Acha, Peleg, ICCV 2007

Page 4: Computational Photography and Capture - UCL€¦ · Computational Photography and Capture: Time-Lapse Video Analysis & Editing Gabriel Brostow & Tim Weyrich TA: Frederic Besse

Time-lapse Photography

• Definition: when frames are captured at a lower rate than the rate at which they will ultimately be played back.

• Compare to

– Slow-motion

– Bullet-time

Page 5: Computational Photography and Capture - UCL€¦ · Computational Photography and Capture: Time-Lapse Video Analysis & Editing Gabriel Brostow & Tim Weyrich TA: Frederic Besse

Time-lapse Filmmaker’s Challenges

Page 6: Computational Photography and Capture - UCL€¦ · Computational Photography and Capture: Time-Lapse Video Analysis & Editing Gabriel Brostow & Tim Weyrich TA: Frederic Besse

Time-lapse Filmmaker’s Challenges

• Lighting changes (strobing)

• High frequency motion (missed action)

– Saturation & exposure

• Camera motion: intentional or not

• Triggering / data storage / camera safety

• Useful? Editable?

Page 7: Computational Photography and Capture - UCL€¦ · Computational Photography and Capture: Time-Lapse Video Analysis & Editing Gabriel Brostow & Tim Weyrich TA: Frederic Besse

Does the old definition of time-lapse still hold?

Page 8: Computational Photography and Capture - UCL€¦ · Computational Photography and Capture: Time-Lapse Video Analysis & Editing Gabriel Brostow & Tim Weyrich TA: Frederic Besse

(Quick Overview of)Computational Time-Lapse Video

Bennett & McMillan

Siggraph 2007

(Project web page)

Page 9: Computational Photography and Capture - UCL€¦ · Computational Photography and Capture: Time-Lapse Video Analysis & Editing Gabriel Brostow & Tim Weyrich TA: Frederic Besse

Virtual Shutter

Page 10: Computational Photography and Capture - UCL€¦ · Computational Photography and Capture: Time-Lapse Video Analysis & Editing Gabriel Brostow & Tim Weyrich TA: Frederic Besse
Page 11: Computational Photography and Capture - UCL€¦ · Computational Photography and Capture: Time-Lapse Video Analysis & Editing Gabriel Brostow & Tim Weyrich TA: Frederic Besse

Sampling in 1D

• Uniform

• Interpolate linearly+ piece-wise

• Same # of samples, but where needed!

• Optimum polygonal approximation of digitized curves, Perez & Vidal, PRL’94

– Use dynamic programming to find global min.

Page 13: Computational Photography and Capture - UCL€¦ · Computational Photography and Capture: Time-Lapse Video Analysis & Editing Gabriel Brostow & Tim Weyrich TA: Frederic Besse

Completely User-controllable:Min-Change, Min-Error, Median, etc

Page 14: Computational Photography and Capture - UCL€¦ · Computational Photography and Capture: Time-Lapse Video Analysis & Editing Gabriel Brostow & Tim Weyrich TA: Frederic Besse
Page 15: Computational Photography and Capture - UCL€¦ · Computational Photography and Capture: Time-Lapse Video Analysis & Editing Gabriel Brostow & Tim Weyrich TA: Frederic Besse

Factored Time-Lapse Video

Sunkavalli, Matusik, Pfister, Rusinkiewicz

SIGGRAPH 2007

Project web page

Page 16: Computational Photography and Capture - UCL€¦ · Computational Photography and Capture: Time-Lapse Video Analysis & Editing Gabriel Brostow & Tim Weyrich TA: Frederic Besse

Outdoors: More Than Just Color vs Illumination?

Original No Shadows

Given:Daytime, under clear-sky conditions

Page 17: Computational Photography and Capture - UCL€¦ · Computational Photography and Capture: Time-Lapse Video Analysis & Editing Gabriel Brostow & Tim Weyrich TA: Frederic Besse

Outdoors: More Than Just Color vs Illumination?

Original No Shadows

Given:Daytime, under clear-sky conditions

Wishlist

Remove shadowsRender new shadows

Change albedoChange global illumination

Page 18: Computational Photography and Capture - UCL€¦ · Computational Photography and Capture: Time-Lapse Video Analysis & Editing Gabriel Brostow & Tim Weyrich TA: Frederic Besse

Outdoors: More Than Just Color vs Illumination?

Original No Shadows

Lighting due to sun? sky?

Surface Normals?

Page 19: Computational Photography and Capture - UCL€¦ · Computational Photography and Capture: Time-Lapse Video Analysis & Editing Gabriel Brostow & Tim Weyrich TA: Frederic Besse

Formulation

• size: width x height x time

• F: xyt volume of frames over time

• Isky: Accumulated intensity due to sky-light

• Isun: Accumulated intensity due to sun-light

• Ssun: Binary; is a pixel in shadow

Page 20: Computational Photography and Capture - UCL€¦ · Computational Photography and Capture: Time-Lapse Video Analysis & Editing Gabriel Brostow & Tim Weyrich TA: Frederic Besse

Intuition

Page 21: Computational Photography and Capture - UCL€¦ · Computational Photography and Capture: Time-Lapse Video Analysis & Editing Gabriel Brostow & Tim Weyrich TA: Frederic Besse
Page 22: Computational Photography and Capture - UCL€¦ · Computational Photography and Capture: Time-Lapse Video Analysis & Editing Gabriel Brostow & Tim Weyrich TA: Frederic Besse

Separate Sky first

• Photoshop!

– Just to pick “non-surfaces”

• When is sun’s contribution == 0?

– Leaving just sky’s contribution

Page 23: Computational Photography and Capture - UCL€¦ · Computational Photography and Capture: Time-Lapse Video Analysis & Editing Gabriel Brostow & Tim Weyrich TA: Frederic Besse

• “Definitely” shadow: median of darkest 20%, then threshold at 1.5x

Page 24: Computational Photography and Capture - UCL€¦ · Computational Photography and Capture: Time-Lapse Video Analysis & Editing Gabriel Brostow & Tim Weyrich TA: Frederic Besse

Bilateral Filter “Definitely” Shadow

Page 25: Computational Photography and Capture - UCL€¦ · Computational Photography and Capture: Time-Lapse Video Analysis & Editing Gabriel Brostow & Tim Weyrich TA: Frederic Besse
Page 26: Computational Photography and Capture - UCL€¦ · Computational Photography and Capture: Time-Lapse Video Analysis & Editing Gabriel Brostow & Tim Weyrich TA: Frederic Besse

How Did Skylight Change?

• Isky: Accumulated intensity due to sky-light

• Wsky: Sky-light image

• Hsky: Sky-light basis curve (1D function!!)

– Whole scene got brighter/darker together

Page 27: Computational Photography and Capture - UCL€¦ · Computational Photography and Capture: Time-Lapse Video Analysis & Editing Gabriel Brostow & Tim Weyrich TA: Frederic Besse

• Blue: skylight curve (same one)

Page 28: Computational Photography and Capture - UCL€¦ · Computational Photography and Capture: Time-Lapse Video Analysis & Editing Gabriel Brostow & Tim Weyrich TA: Frederic Besse

Factorization

• Decompose appearance into

– per-pixel W

– H curve

• Alternating Constrained Least Squares (ACLS):

– H(t) is held fixed while W is optimized using least squares, then vice versa

Page 29: Computational Photography and Capture - UCL€¦ · Computational Photography and Capture: Time-Lapse Video Analysis & Editing Gabriel Brostow & Tim Weyrich TA: Frederic Besse

ACLS: Alternating Constrained Least Squares

• Inverse shade trees for non-parametric material representation and editing,

– Lawrence et al., Siggraph 2006, code online

Page 31: Computational Photography and Capture - UCL€¦ · Computational Photography and Capture: Time-Lapse Video Analysis & Editing Gabriel Brostow & Tim Weyrich TA: Frederic Besse

Original and Isky

Page 32: Computational Photography and Capture - UCL€¦ · Computational Photography and Capture: Time-Lapse Video Analysis & Editing Gabriel Brostow & Tim Weyrich TA: Frederic Besse

Isky and Reconstruction

Page 33: Computational Photography and Capture - UCL€¦ · Computational Photography and Capture: Time-Lapse Video Analysis & Editing Gabriel Brostow & Tim Weyrich TA: Frederic Besse

So Far - Explained Sky

Page 34: Computational Photography and Capture - UCL€¦ · Computational Photography and Capture: Time-Lapse Video Analysis & Editing Gabriel Brostow & Tim Weyrich TA: Frederic Besse

Now Decompose Sun’s Contributions

• Isun: Accumulated intensity due to sun-light

• Wsun: Sun-light image, per-pixel weights

• Hsun: Sun-light basis curve

• : Shift-map; per pixel offset in time

Page 35: Computational Photography and Capture - UCL€¦ · Computational Photography and Capture: Time-Lapse Video Analysis & Editing Gabriel Brostow & Tim Weyrich TA: Frederic Besse
Page 36: Computational Photography and Capture - UCL€¦ · Computational Photography and Capture: Time-Lapse Video Analysis & Editing Gabriel Brostow & Tim Weyrich TA: Frederic Besse

• Isun: Accumulated intensity due to sun-light

• Wsun: Sun-light image, per-pixel weights

• Hsun: Sun-light basis curve

• : Shift-map; per pixel offset in time

Page 37: Computational Photography and Capture - UCL€¦ · Computational Photography and Capture: Time-Lapse Video Analysis & Editing Gabriel Brostow & Tim Weyrich TA: Frederic Besse

• Isun: Accumulated intensity due to sun-light

• Wsun: Sun-light image, per-pixel weights

• Hsun: Sun-light basis curve

• : Shift-map; per pixel offset in time

Page 38: Computational Photography and Capture - UCL€¦ · Computational Photography and Capture: Time-Lapse Video Analysis & Editing Gabriel Brostow & Tim Weyrich TA: Frederic Besse

Total sunlight contribution

Sunlight image

(like on moon)Shift map Sunlight

Page 39: Computational Photography and Capture - UCL€¦ · Computational Photography and Capture: Time-Lapse Video Analysis & Editing Gabriel Brostow & Tim Weyrich TA: Frederic Besse

Factorize Again!

• For each image, subtract off Isky(t)

• Ssun: Sky-light mask, serves as C, confidence map

• ACLS again

– But add 3rd update phase, searching for

that minimizes reconstruction error

of scaled + offset H(t) vs. F(t)

Page 40: Computational Photography and Capture - UCL€¦ · Computational Photography and Capture: Time-Lapse Video Analysis & Editing Gabriel Brostow & Tim Weyrich TA: Frederic Besse

Green: sunlight curve

(same one, but time-shifted)

Page 41: Computational Photography and Capture - UCL€¦ · Computational Photography and Capture: Time-Lapse Video Analysis & Editing Gabriel Brostow & Tim Weyrich TA: Frederic Besse

Isun(t)

Wsun

Page 42: Computational Photography and Capture - UCL€¦ · Computational Photography and Capture: Time-Lapse Video Analysis & Editing Gabriel Brostow & Tim Weyrich TA: Frederic Besse

Reconstruction

Page 43: Computational Photography and Capture - UCL€¦ · Computational Photography and Capture: Time-Lapse Video Analysis & Editing Gabriel Brostow & Tim Weyrich TA: Frederic Besse

Does It Work?

Page 44: Computational Photography and Capture - UCL€¦ · Computational Photography and Capture: Time-Lapse Video Analysis & Editing Gabriel Brostow & Tim Weyrich TA: Frederic Besse

Video: FactoredTimeLapseV

Page 45: Computational Photography and Capture - UCL€¦ · Computational Photography and Capture: Time-Lapse Video Analysis & Editing Gabriel Brostow & Tim Weyrich TA: Frederic Besse

What have we gained?

• Can

– Remove shadows

– Change albedo

– Change amount of global illumination

• Compression

• Render new shadows:

– We have 1 component of the per-pixel Normals, N

Page 46: Computational Photography and Capture - UCL€¦ · Computational Photography and Capture: Time-Lapse Video Analysis & Editing Gabriel Brostow & Tim Weyrich TA: Frederic Besse

Estimated Normals (1D)

Page 47: Computational Photography and Capture - UCL€¦ · Computational Photography and Capture: Time-Lapse Video Analysis & Editing Gabriel Brostow & Tim Weyrich TA: Frederic Besse

Manipulating Normals

Page 48: Computational Photography and Capture - UCL€¦ · Computational Photography and Capture: Time-Lapse Video Analysis & Editing Gabriel Brostow & Tim Weyrich TA: Frederic Besse

(View Results of)Video Synopsis & Indexing

Pritch, Rav-Acha, Peleg

ICCV 2007, PAMI 2008

(Project web page)

Page 49: Computational Photography and Capture - UCL€¦ · Computational Photography and Capture: Time-Lapse Video Analysis & Editing Gabriel Brostow & Tim Weyrich TA: Frederic Besse

What to do with blob-tracks?

(Project web page)

Page 50: Computational Photography and Capture - UCL€¦ · Computational Photography and Capture: Time-Lapse Video Analysis & Editing Gabriel Brostow & Tim Weyrich TA: Frederic Besse

HDR Time-Lapse

• Essentially started in 2006 (manually at first)

• More are popping up online