Milanfar et al. EE Dept, UCSC 1 “Locally Adaptive Patch-based Image and Video Restoration”...
-
date post
20-Dec-2015 -
Category
Documents
-
view
214 -
download
0
Transcript of Milanfar et al. EE Dept, UCSC 1 “Locally Adaptive Patch-based Image and Video Restoration”...
Milanfar et al. EE Dept, UCSC 1
“Locally Adaptive Patch-based Image and Video Restoration”
Session I: Today (Mon) 10:30 – 1:00
Session II: Wed
Same Time, Same Room
Thank you for participating inand contributing to our mini-
symposium on
Milanfar et al. EE Dept, UCSC 2
Local Adaptivity + Patch-Based Approaches
• State of the Art Performance A Convergence of Ideas Extremely Popular
Milanfar et al. EE Dept, UCSC 3
Patchy the Pirate
Patch-based methods have become so popular in fact ….
Milanfar et al. EE Dept, UCSC 4
Multi-dimensional Kernel Regression for Video Processing
and Reconstruction
*Joint work with Hiro Takeda (UCSC), Mattan Protter and Michael Elad (Technion),
Peter van Beek (Sharp Labs of America)
SIAM Imaging Science Meeting, July 7, 2008
Peyman Milanfar*EE Department
University of California, Santa Cruz
Milanfar et al. EE Dept, UCSC 5
Outline
• Background and Motivation
• Classic Kernel Regression
• Data-Adaptive Regression • Adaptive Implicit-Motion Steering Kernel
(AIMS)
• Motion-Aligned Steering Kernel (MASK)
• Conclusions
Milanfar et al. EE Dept, UCSC 6
Summary
• Motivation:– Existing methods make strong assumptions
about signal and noise models.– Develop “universal”, robust methods based on
adaptive nonparametric statistics
• Goal:– Develop the adaptive Kernel Regression
framework for a wide class of problems, including video processing; producing algorithms competitive with state of the art.
Milanfar et al. EE Dept, UCSC 7
Outline
• Background and Motivation
• Classic Kernel Regression
• Data-Adaptive Regression• Adaptive Implicit-Motion Steering Kernel
(AIMS)
• Motion-Aligned Steering Kernel (MASK)
• Conclusions
Milanfar et al. EE Dept, UCSC 8
Kernel Regression Framework
• The data model
A sample
The regression function
Zero-mean, i.i.d noise (No other assump.)
The number of samplesThe sampling position
• The specific form of
may remain unspecified for now.
Milanfar et al. EE Dept, UCSC 9
• The data model
• Local representation (N-terms Taylor expansion)
• Note– With a polynomial basis, we only need to estimate the first unknown,
– Other localized representations are also possible, and may be advantageous.
Local Approximation in KR
Unknowns
Milanfar et al. EE Dept, UCSC 10
Optimization Problem• We have a local representation with respect to each sample:
• MinimizationN+1 terms
This term give the estimated
pixel value at x.
The regression order
The choice of the kernel function is
open, e.g. Gaussian.
Milanfar et al. EE Dept, UCSC 11
Locally Linear Estimator
• The optimization yields a pointwise estimator:
• The bias and variance are related to the regression order and the smoothing parameter:– Large N small bias and large variance
– Large h large bias and small variance
The weighted linear
combinations of the given data
Equivalent kernel
function
Kernelfunction
The smoothingparameter
The regressionorder
Milanfar et al. EE Dept, UCSC 12
Outline
• Background and Motivation
• Classic Kernel Regression
• Data-Adaptive Regression • Adaptive Implicit-Motion Steering Kernel
(AIMS)
• Motion-Aligned Steering Kernel (MASK)
• Conclusions
Milanfar et al. EE Dept, UCSC 13
(2D) Data-Adaptive Kernels
Classic kernel Data-adapted kernel
• Take not only spatial distances, but also radiometric distances (pixel value differences) into account
• Data-adaptive kernel function
• Yields locally non-linear estimators
Milanfar et al. EE Dept, UCSC 14
Simplest Case: Bilateral Kernels
= .
= .
= .
Low noise case
Spatial kernel
Radiometric kernel
Milanfar et al. EE Dept, UCSC 16
Better: Steering Kernel MethodLocal dominant
orientation estimate based
on local gradient covariance
H. Takeda, S. Farsiu, P. Milanfar, “Kernel Regression for Image Processing and Reconstruction”,IEEE Transactions on Image Processing, Vol. 16, No. 2, pp. 349-366, February 2007.
Milanfar et al. EE Dept, UCSC 17
Steering Kernel
• Kernel adapted to locally dominant structure
• The steering matrices scale, elongate, and rotate the kernel footprints locally.
Local dominant orientation estimation
Steering matrix
Elongate Rotate Scale
Milanfar et al. EE Dept, UCSC 19
Steering Kernel (Low Noise)
• Kernel weights and footprints:
Low noise case
FootprintsWeightsSteering kernel
as a function of xi with x held fixed
Steering kernel as a function of x with xi and
Hi held fixed
Milanfar et al. EE Dept, UCSC 20
Steering Kernel (High Noise)
High noise case
FootprintsWeightsSteering kernel
as a function of xi with x held fixed
Steering kernel as a function of x with xi and
Hi held fixed
• Steering approach provides stable weights even in the presence of significant noise.
• Kernel weights and footprints:
Milanfar et al. EE Dept, UCSC 21
Some Related (0th-order) Methods
• Non-Local Means (NLM)– A. Buades, B. Coll, and J. M. Morel. “A review of image denoising algorithms, with a new one.” Multiscale
Modeling & Simulation, 4(2):490-530, 2005.
• Optimal Spatial Adaptation (OSA)– C. Kervrann, J. Boulanger “Optimal spatial adaptation for patch-based image denoising.” IEEE Trans. on
Image Processing, 15(10):2866-2878, Oct 2006.
SKR
2 4 6 8 10 12 14 16 18 20
2
4
6
8
10
12
14
16
18
20
2 4 6 8 10 12 14 16 18 20
2
4
6
8
10
12
14
16
18
20
NLM OSA
Milanfar et al. EE Dept, UCSC 22
Adaptive Kernels for Interpolation
• When there are missing pixels:– We cannot have the radiometric distance.
– Using a “pilot” estimate, fill the missing pixels:
• Classic kernel regression
• Cubic or bilinear interpolation ??
?
?
?
?
?
?
?
??
??
?
?
?
Milanfar et al. EE Dept, UCSC 23
Outline
• Background and Motivation
• Classic Kernel Regression
• Data-Adaptive Regression • Regression in 3-D
• Adaptive Implicit-Motion Steering Kernel (AIMS)• Motion-Aligned Steering Kernel (MASK)
• Conclusions
Milanfar et al. EE Dept, UCSC 24
Kernel Regression in 3-D
• Setup is similar to 2-D, but…..
• Data samples come from various (nearby) frames
• Signal “structure” is now in 3-D
• We can perform– Denoising – Spatial Interpolation– Frame rate upconversion– Space-time super-resolution
Spatial gradients
Temporalgradients
Milanfar et al. EE Dept, UCSC 25
Kernel Regression in 3-D Cont.
• Two ways to proceed
– Adaptive Implicit-Motion Steering Kernel (AIMS)
• Roughly warp the data to “neutralize” large motions • Implicitly capture sub-pixel motions in 3-D Kernel
– Motion-Aligned Steering Kernel (MASK)
• Estimate motion with subpixel accuracy• Accurately warp the kernel (instead of the data)
Milanfar et al. EE Dept, UCSC 26
Outline
• Background and Motivation
• Classic Kernel Regression
• Data-Adaptive Regression • Regression in 3-D
• Adaptive Implicit-Motion Steering Kernel (AIMS)• Motion-Aligned Steering Kernel (MASK)
• Conclusions
Milanfar et al. EE Dept, UCSC 27
AIMS Kernel in 3-D
• Steering kernel visualization examples
A plane structure Steering kernel weights Isosurface
A tube structure
Milanfar et al. EE Dept, UCSC 28
AIMS Motion Compensation
• Large displacements make orientation estimation difficult.
• By neutralizing the large displacement, the steering kernel can effectively spread again.
Small motions Large motions
The local kernel effectively spread
along the local motion trajectory.
The local kernel effectively spread
along the local motion trajectory.
Shiftdown
Shiftup
The local kernel after motion
compensation.
Important: The compensation does not require subpixel accurate motion estimation, nor does it require interpolation
Milanfar et al. EE Dept, UCSC 29
AIMS Contains Implicit Motion“Small” motion vector
Optical flow equation
Assuming the patch moves
with approximate uniformity
Homogeneous Optical Flow Vector
(Eigenvalues of C)
Space-time gradientsof roughly compensated data
Milanfar et al. EE Dept, UCSC 30
AIMS Summary
• AIMS is a two-tiered approach.
1. Neutralize whole-pixel motions.
2. 3-D SKR with implicit subpixel motion information
Steering matrices estimated from the
motion compensated data in 3-D.
Milanfar et al. EE Dept, UCSC 32
Foreman Example
Lanczos(frame-by-frame upscaling)
AIMS
Factor of 2 upscaling
Input video(QCIF: 144 x 176 x 28)
Milanfar et al. EE Dept, UCSC 33
Spatial Upscaling Example
Input (200 x 200) Upscaled image by AIMS(multi-frame, 5 frames), 400x400
Milanfar et al. EE Dept, UCSC 34
Spatiotemporal Upscaling
Input video(200 x 200 x 20)
Single framesteering kernelregression(400 x 400 x 20)
Spatiotemporalclassic kernel
regression(400 x 400 x 40)
AIMSregression(400 x 400 x 40)
Milanfar et al. EE Dept, UCSC 35
Outline
• Background and Motivation
• Classic Kernel Regression
• Data-Adaptive Regression • Regression in 3-D
• Adaptive Implicit-Motion Steering Kernel (AIMS)• Motion-Aligned Steering Kernel (MASK)
• Conclusions
Milanfar et al. EE Dept, UCSC 36
Motion-Aligned Steering Kernel
• Motion is explicitly estimated to subpixel accuracy
• Kernel weights are aligned with the local motion vectors using warping/shearing
• The warped kernel acts directly on the data– Handles large and/or complex motions
“2-D motion-steered” (spatial) kernel
1-D (temporal) kernel
Accurate, explicit motion estimates
Milanfar et al. EE Dept, UCSC 37
Intuition Behind the MASK
2-D “motion-steered” (spatial) kernel 1-D (temporal) kernel
Milanfar et al. EE Dept, UCSC 38
The Shapes of MASK
• Spreads along spatial orientations and local motion vectors.
Local dataSlices of
MASK kernels
Milanfar et al. EE Dept, UCSC 40
A Comparison of AIMS and MASK
• Spin Calendar video
Input video(200 x 200 x 20)
AIMS(400 x 400 x 40)
MASK(400 x 400 x 40)
Milanfar et al. EE Dept, UCSC 41
A Comparison of AIMS and MASK
• Foreman video
Input video(QCIF: 144 x 176 x 28)
AIMS + BTV deblurring(CIF: 288 x 352 x 28)
MASK + BTV deblurring(CIF: 288 x 352 x 28)
Milanfar et al. EE Dept, UCSC 42
Conclusions
• We extended the 2-D kernel regression framework to 3-D.– Illustrated 2 distinct approaches
• AIMS: Avoids subpixel motion estimation, needs comp. for large motions
• MASK: Needs subpixel motion estimation, deals directly with large motions
– Which is better? Depends on the application.
• The overall 3-D SKR framework is simultaneously well-suited for spatial, temporal, and spatiotemporal– upscaling, denoising, blocking artifact removal, superresolution– not only in video but in general 3-D data sets.
• Future work– Integration of deblurring directly in the 3-D framework– Computational complexity