Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive...
Transcript of Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive...
![Page 1: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/1.jpg)
Tutorial: Compressive Sensing of Video
Venue: CVPR 2012, Providence, Rhode Island, USA
June 16, 2012
Richard Baraniuk Mohit Gupta Aswin Sankaranarayanan Ashok Veeraraghavan
![Page 2: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/2.jpg)
Tutorial: Compressive Sensing of Video
Unit 3 – Compressive Video Sensing Systems
Ashok Veeraraghavan
![Page 3: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/3.jpg)
RECAP: Compressive Sensing
• Signal recovery via L1 optimization [Candes, Romberg, Tao; Donoho]
random measurements
sparse signal
nonzero entries
![Page 4: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/4.jpg)
RECAP: Compressive Sensing
• Signal recovery via iterative greedy algorithm
– (orthogonal) matching pursuit [Gilbert, Tropp]
– iterated thresholding [Nowak, Figueiredo; Kingsbury, Reeves; Daubechies, Defrise, De Mol; Blumensath, Davies; …]
– CoSaMP [Needell and Tropp]
random measurements
sparse signal
nonzero entries
![Page 5: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/5.jpg)
The Plan for the next hour
• Model Based Compressive Video Sensing – Coded Strobing for Periodic signals
– CS-Linear Dynamical Systems
– Linear Motion Model
• Compressive Sensing of Generic Videos – Single Pixel Video Camera
– Voxel Subsampling
– Voxel Multiplexing
![Page 6: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/6.jpg)
Model Based CS Video
• Periodic or Repetitive Videos
– CS Strobing
• Linear Dynamical Systems
– CS – LDS
• Linear Motion Model
– Multi-Exposure Video
![Page 7: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/7.jpg)
Coded Strobing Camera
Ashok Veeraraghavan
Dikpal Reddy
Ramesh Raskar
PAMI 2010
![Page 8: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/8.jpg)
8
Periodic Visual Signals
![Page 9: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/9.jpg)
9
Strobing: Applications
-Strobo-laryngoscopy
-Observing machine
parts, motors etc.
![Page 10: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/10.jpg)
11
Periodic Signals
Periodic signal with period P and band-limited to fMax.
The Fourier domain contains only terms that are multiples of fP=1/P.
4fP
3fP 0 fMax - fMax fP=1/P 2fP -fP -2fP -4fP -3fP
t
Periodic Signal (x(t)) with period P
P
![Page 11: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/11.jpg)
12
Normal Camera and Aliasing
0 fMax - fMax
Causes aliasing of high frequency information.
fS/2 = Sampling Frequency/2
Integrates out the high frequency information and this information is
irrevocably lost.
P = 10ms
TFrame= Frame Duration = 40ms
![Page 12: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/12.jpg)
14
High Speed Camera
0 fMax - fMax
High speed video camera is inefficient since its bandwidth is wasted on
the periodic signal since it is sparse in the Fourier domain
fP=1/P 2fP -fP -2fP -4fP -3fP 4fP 3fP
Nyquist Sampling of x(t) – When each period of x has very fine high
frequency variations Nyquist sampling rate is very high.
P = 10ms Ts = 1/(2 fMax)
![Page 13: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/13.jpg)
15
Traditional Strobing
When sampling rate of camera is low, generate beat frequencies of
the high speed periodic signal which now can be easily captured
P = 10ms
The light throughput is also very low in order to avoid blurring of the
features during the strobe time.
The period of the signal needs to be known a priori
t
![Page 14: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/14.jpg)
16
Coded Strobing
![Page 15: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/15.jpg)
17
Coded strobing: Time Domain
Proposed Design: In every exposure duration observe different linear
combinations of the underlying periodic signal.
Advantage of the design:
Exposure coding is independent of the frequency of the underlying periodic
signal.
Further, light throughput is on an average 50% which is far greater than
can be achieved via traditional strobing.
t P = 10ms
![Page 16: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/16.jpg)
18
Coded strobing: Frequency domain
Measure linear combinations of the Fourier components.
Decode by enforcing sparsity of the reconstructed signal in Fourier
domain.
4fP fP=1/P 2fP -fP -4fP
Measure Linear Combinations
Sparsity Enforcing Reconstruction Algorithm
0
Measure Linear Combinations
Sparsity Enforcing Reconstruction Algorithm
-3fP 3fP fMax - fMax
Measure Linear Combinations
Sparsity Enforcing Reconstruction Algorithm
![Page 17: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/17.jpg)
19
Signal model
Non-zero
elements
Signal Fourier Basis Basis Coeff
x = B s
b1 b2 bN
t
![Page 18: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/18.jpg)
20
Observation Model
y
Observed
Intensity Signal
C x =
Coded
Strobing
Frame 1
Frame M Frame Integration
Period TS
N unknowns
t
![Page 19: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/19.jpg)
21
Signal and Observation model
Mixing matrix ( satisfies
Restricted Isometry under
certain conditions )
Fourier coefficients
of the signal ( sparse
if periodic)
Observed
low-frame rate
video
A s y C B s = =
Signal Model:
x = B s
Observation Model:
y = C x
A of size M x N, M<<N
![Page 20: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/20.jpg)
22
Linear Inversion?
• y = A s highly underdetermined. M << N
• For example: Capture at 10 fps for 30s with 1 ms resolution has
M=300, N=30000
• Highly unstable in noise
• To reconstruct can we use extra information about the signal?
![Page 21: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/21.jpg)
24
Extra info: Periodic signal
Very few (K)
non-zero
elements
Sparse Basis Coeff
y = A s
3fP 0 fMax - fMax fP=1/P 2fP -fP -2fP -4fP -3fP
Mixing matrix Observed low
rate frames
![Page 22: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/22.jpg)
26
Compressed Sensing
Estimate sparse s that satisfies y = A s Solve
P1 is a linear program. Fast and robust algorithms available for solving P1
(P1) 1 2
min . .s s t y As
![Page 23: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/23.jpg)
27
Signal model and recovery • Signal Model
• Observation Model
• Compressed sensing
• Sparsity and location enforced recovery
• x = B s
• y = C x
• Solve
1 2
min . .s s t y As
![Page 24: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/24.jpg)
28
Periodic Signals
Periodic signal with period P and band-limited to fMax.
The Fourier domain contains only terms that are multiples of fP=1/P.
4fP
3fP 0 fMax - fMax fP=1/P 2fP -fP -2fP -4fP -3fP
t
Periodic Signal (x(t)) with period P
P
![Page 25: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/25.jpg)
Structure of Sparsity
Sparse Basis Coeff
y = A s Mixing matrix Observed low
rate frames
Fundamental frequency and its harmonics
![Page 26: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/26.jpg)
30
Sparsity and location
(P1t) 1 2
min . .s s t y As
• In addition to sparsity, use the location of the non-zero coefficients as constraint
• All the non-zero coefficients equally spaced at fP
• Solve
with truncated mixing matrix and basis coefficients
![Page 27: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/27.jpg)
31
Signal model and recovery • Signal Model
• Observation Model
• Compressed sensing
• Sparsity and location enforced recovery
• x = B s
• y = C x
• Solve
• Solve
1 2min . .s s t y As
1 2min . .s s t y As
![Page 28: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/28.jpg)
32
Implementation Captured at 10fps using PGR Dragonfly2
FLC Shutter
Can be strobed at 1ms resolution
Can flutter at 250us
![Page 29: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/29.jpg)
33
Example: Fractal simulation Original Video: Period 25ms Recovered Video
Captured by 10fps camera Captured by 10fps coded strobing
camera
![Page 30: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/30.jpg)
34
Rotating Mill Tool (Dragonfly2)
Normal Video: 10fps
Reconstructed Video
3000 RPM
Reconstructed Video
6000 RPM
Reconstructed Video
12000 RPM
Reconstructed Video
4500 RPM
![Page 31: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/31.jpg)
Compressive Sensing: Linear Dynamical Systems
Aswin Sankaranarayanan
Pavan Turaga
Richard Baraniuk
Rama Chellappa
![Page 32: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/32.jpg)
CSLDS: Video CS under LDS model
• Motivation
– Extensive use in dynamic textures, sequence of LDS
for activity analysis, etc…
– LDS == Frames of the video lie on a low dimensional subspace. Linearity applicable over short durations
• Benefits
– LDS is a parametric model. Video CS is equivalent to
parameter estimation
– Majority of the LDS parameters are static. This is
extremely helpful in achieving high compression at sensing
![Page 33: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/33.jpg)
Linear dynamical system example
Video sequence
Few frames
Six key images
All frames can be estimated using linear combinations of SIX images
![Page 34: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/34.jpg)
Observation model Data lies on a low dimensional subspace
C is a basis for the subspace. For uncompressed data, C is usually learnt
using PCA on training examples.
Terminology xt are the frames of the video (high dim) vt are the subspace coeffficients (low
dim)
LDS model
NdRvRx
Cvx
d
t
N
t
tt
,,
![Page 35: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/35.jpg)
Can we recover C and vt from yt ?
Very hard since the unknowns are bilinear
Compressive LDS
NMRy
vCAxAy
NdRvRx
Cvx
M
t
ttt
d
t
N
t
tt
,
,,
Digital micro-mirror
device
Photo-detector
![Page 36: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/36.jpg)
Challenges and solution
• Key challenge: Unknown variables appear as bilinear relationships – CS theory handles unknowns in a linear relationship
• Solution: A novel measurement model that reduces the bilinear problem to a sequence of linear/convex problems
– Stable reconstruction at a measurement rate proportional to the state space dimension.
Common measurements
Innovations measurements
Estimate state sequence using SVD
Model based CoSaMP for recovery of observation
matrix
Scene CS camera
Tx :1
t~
Ty :1
Ty :1~
Tv :1
C
tt vCxˆˆ Tx :1
ˆ
![Page 37: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/37.jpg)
Results
Ground truth Comp 20x Comp 50x SNR: 24.9 dB SNR: 20.4 dB
Ground truth Comp20x Comp50x SNR: 25.3 dB SNR:23.8 dB
![Page 38: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/38.jpg)
Amit Agrawal
Yi Xu
Ramesh Raskar
Linear Motion Model:
Invertible Motion Blur in Video
![Page 39: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/39.jpg)
Traditional Exposure Video
Fourier
Transform
Motion PSF
(Box Filter)
Information is lost
Exposure Time
Slide courtesy Agrawal
![Page 40: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/40.jpg)
Varying Exposure Video
Exposure Time Fourier
Transform
Slide courtesy Agrawal
![Page 41: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/41.jpg)
Varying Exposure Video
Exposure Time
Exposure Time Fourier
Transform
No common nulls
Slide courtesy Agrawal
![Page 42: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/42.jpg)
Varying Exposure Video
Exposure Time
Exposure Time
Exposure Time
Fourier
Transform
No common nulls
Slide courtesy Agrawal
![Page 43: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/43.jpg)
Varying Exposure Video = PSF Null-Filling
Joint Frequency
Spectrum
Preserves All
Frequencies
Fourier
Transform
Slide courtesy Agrawal
![Page 44: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/44.jpg)
Varying Exposure Video
Slide courtesy Agrawal
![Page 45: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/45.jpg)
Deblurred Result
Blurred Photos
Slide courtesy Agrawal
![Page 46: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/46.jpg)
Single Image Deblurring (SID)
Multiple Image Deblurring (MID) Slide courtesy Agrawal
![Page 47: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/47.jpg)
The Story so far
• Periodic or Repetitive Videos
– CS Strobing
• Linear Dynamical Systems
– CS – LDS
• Linear Motion Model
– Multi-Exposure Video
![Page 48: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/48.jpg)
How about general Videos?
• General Multiplexing
– Single Pixel Camera
• Voxel Subsampling
– Flexible Voxels
– Coded per-Pixel Exposure Video
• Voxel Multiplexing
– Programmable Pixel Compressive Camera
![Page 49: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/49.jpg)
Single pixel camera
• Each configuration of micro-mirrors yield ONE compressive measurement • Non-visible wavelengths
• Sensor material costly in IR/UV bands
• Light throughput
• Half the light in the scene is directed to the photo-detector
Digital micro-mirror
device
Photo-detector
![Page 50: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/50.jpg)
Single-Pixel Camera
target N=65536 pixels
M=1300 measurements (2%)
M=11000 measurements (16%)
M randomized measurements
![Page 51: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/51.jpg)
From Image to Video Sensing
• SPC works under the assumption of a static scene
– However, for dynamic scenes changes as we collect measurements
• Need to measure temporal events at their “information rate”
– Need to borrow richer models for videos and exploit them at sensing and reconstruction
![Page 52: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/52.jpg)
Single Pixel Camera in SWIR
Single pixel camera setup
Object
InGaAs Photo-detector (Short-wave IR)
SPC sampling rate: 10,000 sample/s
Number of compressive measurements: M = 16,384
Recovered video: N = 128 x 128 x 61 = 61*M
![Page 53: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/53.jpg)
CS-MUVI: IR spectrum
Joint work with Xu and Kelly
Recovered Video
initial estimate Upsampled
![Page 54: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/54.jpg)
CS-MUVI on SPC
Naïve frame-to-frame recovery
CS-MUVI
Joint work with Xu and Kelly
![Page 55: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/55.jpg)
Voxel Subsampling Compressive Video
Flexible Voxels for Content Aware Videography ECCV 2010
Mohit Gupta, Amit Agrawal, Ashok Veeraraghavan,
Srinivas Narasimhan
Video from a Single Exposure Coded Photograph ICCV 2011
Yasunobu Hitomi, Jinwei Gu, Mohit Gupta ,
Tomoo Mitsunaga, Shree K. Nayar
![Page 56: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/56.jpg)
Spatial resolution (pixels)
Tem
po
ral
reso
luti
on (
fps)
30
1M
1024x1024
Conventional video camera
500
64K
256x256
High-speed Cameras
Slide adapted from Ben-Ezra et al
Fundamental Resolution Trade-off
![Page 57: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/57.jpg)
68
Low-Frame Rate (30 fps) Camera
Large Motion Blur
Imaging Dynamic Scenes
High-Speed (480 fps) Camera
Low Spatial Resolution
![Page 58: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/58.jpg)
69
‘Motion Aware’ Camera
Variable Spatio-Temporal Resolution
![Page 59: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/59.jpg)
Spatio-Temporal Sampling
Thin and long voxels
High SR, Low TR
Fat and short voxels
Low SR, High TR
Flexible voxels
Motion Aware Video
x
t
y
Fast Moving
Slow
Moving
Static
![Page 60: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/60.jpg)
Sampling of the Space-Time Volume
Time
Camera
Sensor Plane
Integration Time
Conventional Sampling Scheme:
Frame 1 Frame 2 Frame N
Camera
Our Sampling Scheme: Frame 1 Frame 2 Frame N
![Page 61: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/61.jpg)
Multi-resolution Sampling
Captured Data Desired Spatio-temporal Interpretations
Re-binning
TR = 1X
SR = 1X
TR = 2X
SR = ½ X
TR = 4X
SR = ¼X
TR = Gain in Temporal Resolution SR = Gain in Spatial Resolution
Time
Spac
e
![Page 62: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/62.jpg)
Multi-resolution Sampling
Captured Data Different Spatio-temporal Interpretations
Re-binning
TR = 1X
SR = 1X
TR = 2X
SR = ½ X
TR = 4X
SR = ¼X
What is the correct firing order?
Re-binning
TR = 1X
SR = 1X
TR = 2X
SR = ½ X
TR = 4X
SR = ¼X
Unsampled Bins
![Page 63: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/63.jpg)
Computing the Firing Order
Captured Data Desired Spatio-temporal Interpretations
Integer Linear Program:
ij
Bp
p Bxij
1
KxK
p
p
2
1
1,0pxijB
K = Number of pixels in a group p = location index Bij = ith block in jth interpretation
Each Block contains exactly one measurement
Number of measurements = Number of pixels
![Page 64: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/64.jpg)
2 1 2
3 4
5 6
7 8 9 10
11 12
13 14
15 16
t
1
5
15
8
4
11
16
9 7 10
13 3 14
12 6
1 2
3 4
5 6
7 8
9 10
11 12
13 14
15 16 t
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
t
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
t
TR = 4, SR = 1/4 TR = 8, SR = 1/8 TR = 16, SR = 1/16
TR = 2, SR = 1/2 TR = 1, SR = 1/1
Firing Order and Reconstructions for a 4x4 Block
1
5
15
8
4
11
2 16
9 7 10
13 3 14
12 6
x
y
![Page 65: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/65.jpg)
Co-located Projector-Camera Setup
Image Plane
Projector Image
Plane
Camera
Scene
Beam
Splitter
Time
Pixel 1
Pixel 2
Pixel K
Camera Integration Time Projector Pattern
![Page 66: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/66.jpg)
Co-located Projector-Camera Setup
Image Plane
Projector Image
Plane
Camera
Scene
Beam
Splitter
![Page 67: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/67.jpg)
LCOS Hardware Prototype
78
![Page 68: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/68.jpg)
Multiple Balls Bouncing and Colliding (15 FPS)
Large Motion Blur
Close-up
![Page 69: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/69.jpg)
Reconstruction at 4X Temporal Resolution (60FPS)
Aliasing Artifacts on the Background
![Page 70: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/70.jpg)
Reconstruction at 16X Temporal Resolution (240FPS)
Aliasing Artifacts on the Background
![Page 71: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/71.jpg)
Motion-aware Video
Captured Frame Different Spatio-temporal Interpretations
Increasing Temporal Resolution
Motion
Analysis
Optical Flow Magnitudes
+ +
Motion-Aware Video
![Page 72: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/72.jpg)
Motion-aware Video
Simple Rebinning Motion-Aware Video
![Page 73: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/73.jpg)
Multiple Balls Bouncing
Input Sequence Motion-Aware Video
![Page 74: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/74.jpg)
Input Sequence Motion-Aware Video
Comparison
![Page 75: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/75.jpg)
Voxel Subsampling with Dictionary Learning?
Slide Courtesy Hitomi et. al.
![Page 76: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/76.jpg)
Exposure pattern
(white=on, black=off)
Original scene Input Coded image
. =
Pixel-wise Coded Exposure Sampling
Slide Courtesy Hitomi et. al.
![Page 77: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/77.jpg)
Dictionary learning
91
Learning video set 20 scenes, 8 spatial rotations,
Play forward and backward
Overcomplete dictionary bases
Slide Courtesy Hitomi et. al.
![Page 78: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/78.jpg)
92
Simulations
Slide Courtesy Hitomi et. al.
![Page 79: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/79.jpg)
Exposure Pattern (450x300x36)
Coded image (450x300)
Road Traffic
93
Original scene
(450x300x36)
Slide Courtesy Hitomi et. al.
![Page 80: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/80.jpg)
Grid sampling + interpolation (450x300x36)
Our method (450x300x36)
3D DCT representation + sparse reconstruction (450x300x36)
Thin-out movie (13x9x36 450x300x36)
Simulation Results
94 Slide Courtesy Hitomi et. al.
![Page 81: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/81.jpg)
95
Original scene
(384x216x36)
Coded image (384x216)
Dog’s Tongue
Slide Courtesy Hitomi et. al.
![Page 82: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/82.jpg)
3D DCT representation + sparse reconstruction (384x216x36)
Simulation Results
96
Thin-out movie (11x6x36 384x216x36)
Our method (384x216x36)
Grid sampling + interpolation (384x216x36)
Slide Courtesy Hitomi et. al.
![Page 83: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/83.jpg)
Experimental Results
![Page 84: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/84.jpg)
Falling Ball
Coded image
(320x320, 18ms)
Reconstructed Movie
(320x320x18)
Slide Courtesy Hitomi et. al.
![Page 85: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/85.jpg)
Reconstructed Movie
(320x320x18)
Coded image
(320x320, 18ms)
Fluid (milk) pouring
Slide Courtesy Hitomi et. al.
![Page 86: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/86.jpg)
Reconstructed Movie
(320x320x9)
Coded image
(320x320, 27ms)
Blinking Eye
Slide Courtesy Hitomi et. al.
![Page 87: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/87.jpg)
Voxel Multiplexing Compressive Video
Programmable Pixel Compressive Camera
CVPR 2011
Dikpal Reddy, Ashok Veeraraghavan, Rama Chellappa
![Page 88: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/88.jpg)
P2C2: Programmable Pixel Compressive Camera
Spatio-temporal mask modulates every pixel independently.
Mask has 1 megapixels and modulates at 200 fps.
Camera operating at 25 fps Fast moving scene 0.25 megapixel Sensor array
Captured low-res coded
frames from a 25fps camera
Optimization using
Brightness constancy &
Wavelet domain sparsity
Capture
Recovery
Camera operating at 25 fps
Captured low-res coded
frames from a 25fps camera
Video credit: TechImaging
![Page 89: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/89.jpg)
P2C2: Prototype
Camera Lens Pentax 35mm
LCOS Mirror SXGA-3DM
Objective Lens F=80 mm
D=40 mm Polarizing
beamsplitter
![Page 90: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/90.jpg)
P2C2: Programmable pixel compressive camera
)(D
Reconstruction
Algorithm
):,(:, tX
Camera operating at 25 fps
Modulation at 200 fps
):,(:,):,(:, tMtX
Integration Subsampling
spatially by 2
):,(:, ltY
Coded motion blur
Temporal superres
Spatial superres
8tL
2sL
Frame captured
by P2C2
Frame captured
by P2C2
![Page 91: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/91.jpg)
P2C2: Programmable pixel compressive camera
6x
=
4,1 tLsL
2y
Observations and unknowns
6mP2C2 captured frame
high speed frame
mask
![Page 92: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/92.jpg)
P2C2: Programmable pixel compressive camera
2
2xyEdata
xy
=
4,1 tLsL
Observations and unknowns
• Severely ill-conditioned system of equations (216 x 220)
![Page 93: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/93.jpg)
P2C2: Brightness constancy constraints
t=1 t=2 t=3 t=4
= 0
-1 1 1x
2x
3x
4x
Forward flow Forward flow Forward flow
1 -1 1 1x
2x
3x
1x
2x
4x
3x
1x
2x
0)()( 12 xxBrightness constancy
![Page 94: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/94.jpg)
P2C2: Brightness constancy constraints
2
2xEBC
t=1 t=2 t=3 t=4
= 0
0 x
1x
2x
3x
4x
Subpixel flow
Bilinear interpolation
Occluded pixel
Pixel occluded in t=4
Row removed
Backward flow
Forward flow Forward flow Forward flow
-1 1
-1 1
.1 .3 .2 .4 -1
1x
2x
3x
1x
2x
4x
3x
1x
2x
![Page 95: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/95.jpg)
P2C2: Transform sparsity
1
1
tspatial xE
Wavelet: DB6 DCT
tx1
DCT Image
tx
![Page 96: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/96.jpg)
P2C2: Dancers video
x y
Original high speed scene P2C2 capture Normal camera capture
Temporal superres
Spatial superres
8tL
1sL
Video credit: Photron
![Page 97: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/97.jpg)
If brightness constancy constraints available
• Solve
• But optical flow not available apriori
• Optical flow can be extracted from smooth, low spatial resolution
images
Estimate sub-frames without brightness constancy
constraints
P2C2: Cost function
T
t
t
spatialBCdata
xxxy
EEE
11
12
2
2
2min
min
![Page 98: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/98.jpg)
Recover sub-frames by solving
• Solve
• Significantly ill conditioned
• Set high-frequency coefficients to zero
P2C2: Initial estimate
T
t
t
spatialdata
xxyx
EE
11
12
2
0 minarg
min
0xpreserves motion
allowing OF to be
estimated
preserves motion
allowing OF to be
estimated
smooth, high
spatial frequency is
lost
![Page 99: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/99.jpg)
P2C2: Estimate Optical Flow
Recover flow from estimated sub-frames
• Perform consistency check using backward and forward flow
• Perform occlusion reasoning to prune the brightness
constancy constraints
)( 00 ivuabs )( ivuabs
OF from original high speed frames OF from initial recovered frames
![Page 100: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/100.jpg)
If brightness constancy constraints available
• Solve
• But optical flow not available apriori
• Optical flow can be extracted from smooth, low spatial resolution
images
Estimate sub-frames without brightness constancy
constraints
P2C2: Cost function
T
t
t
spatialBCdata
xxxy
EEE
11
12
2
2
2min
min
![Page 101: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/101.jpg)
Begin iteration (iteration index k)
• Estimate OF from
• Build BC constraint matrix
• Estimate sub-frames by minimizing
End iteration
P2C2: Algorithm
),( 11 kk vu
1k
kx
• Estimate sub-frames without using BC constraints 0x
1kx
totalE
![Page 102: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/102.jpg)
P2C2: Recovered Examples
Original high speed scene
Reconstructed scene
row
time
occlusion
Normal camera capture
occlusion
Row-Time slice
![Page 103: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/103.jpg)
P2C2: Recovered Examples
Original high speed scene
occlusion
row
time
Reconstructed scene Normal camera capture
Row-Time slice
![Page 104: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/104.jpg)
How about general Videos?
• General Multiplexing
– Single Pixel Camera
• Voxel Subsampling
– Flexible Voxels
– Coded per-Pixel Exposure Video
• Voxel Multiplexing
– Programmable Pixel Compressive Camera
![Page 105: Tutorial: Compressive Sensing of Videoas48/VideoCS2012Tutorial/Part3.AV.pdfTutorial: Compressive Sensing of Video Venue: CVPR 2012, ... • Linear Motion Model ... 2 1 2 4 3 5 6 8](https://reader033.fdocuments.us/reader033/viewer/2022051601/5add714d7f8b9a9a768ce723/html5/thumbnails/105.jpg)
• Motion Blur
– Flutter Shutter
– Parabolic Exposure
• Rolling Shutter Cameras
– Coded Rolling Shutter
• High Dimensional Visual Compressive Sensing
– Compressive Light Transport
– Compressive volumetric Scattering
– Reinterpretable Imaging
Whats next ?