A Gentle Introduction to Bilateral Filtering and its ...yu/Teaching/CISC849_S15/handouts/bil… ·...
Transcript of A Gentle Introduction to Bilateral Filtering and its ...yu/Teaching/CISC849_S15/handouts/bil… ·...
A Gentle Introductionto Bilateral Filteringand its Applications
A Gentle Introductionto Bilateral Filteringand its Applications
Sylvain Paris – MIT CSAIL
Pierre Kornprobst – INRIA Odyssée
Jack Tumblin – Northwestern University
Frédo Durand – MIT CSAIL
• The bilateral filter is becoming in computational photography.
• Many applications with high quality results.
Photographic Style Transfer[Bae 06]Photographic Style Transfer[Bae 06]
input
Photographic Style Transfer[Bae 06]Photographic Style Transfer[Bae 06]
output
Tone Mapping[Durand 02]Tone Mapping[Durand 02]
HDR input
Tone Mapping[Durand 02]Tone Mapping[Durand 02]
output
input
Cartoon Rendition[Winnemöller 06]Cartoon Rendition[Winnemöller 06]
Cartoon Rendition[Winnemöller 06]Cartoon Rendition[Winnemöller 06]
output
6 papers atSIGGRAPH’07
6 papers atSIGGRAPH’07
Goal: Image SmoothingGoal: Image Smoothing
Split an image into:
• large-scale features, structure
• small-scale features, texture
input smoothed(structure, large scale)
residual(texture, small scale)
Gaussian Convolution
BLUR HALOS
Naïve Approach: Gaussian BlurNaïve Approach: Gaussian Blur
Impact of Blur and HalosImpact of Blur and Halos• If the decomposition introduces blur and
halos, the final result is corrupted.
Sample manipulation:increasing texture
(residual × 3)
input smoothed(structure, large scale)
residual(texture, small scale)
edge-preserving: Bilateral Filter
Bilateral Filter: no Blur, no HalosBilateral Filter: no Blur, no Halos
input
increasing texturewith Gaussian convolution
H A L O S
increasing texturewith bilateral filter
N O H A L O S
Notation and DefinitionsNotation and Definitions
• Image = 2D array of pixels
• Pixel = intensity (scalar) or color (3D vector)
• Ip = value of image I at position: p = ( px , py )
• F [ I ] = output of filter F applied to image I
x
y
Strategy for Smoothing ImagesStrategy for Smoothing Images
• Images are not smooth because adjacent pixels are different.
• Smoothing = making adjacent pixelslook more similar.
• Smoothing strategypixel → average of its neighbors
Box AverageBox Average
average
input
square neighborhood
output
sum overall pixels q
normalizedbox function
intensity atpixel q
result atpixel p
Equation of Box AverageEquation of Box Average
∑∈
−=S
IBIBAq
qp qp )(][ σ
0
Square Box Generates Defects Square Box Generates Defects
• Axis-aligned streaks
• Blocky results
input
output
unrelatedpixels
unrelatedpixels
relatedpixels
Box ProfileBox Profile
pixelposition
pixelweight
Strategy to Solve these ProblemsStrategy to Solve these Problems
• Use an isotropic (i.e. circular) window.
• Use a window with a smooth falloff.
box window Gaussian window
Gaussian BlurGaussian Blur
average
input
per-pixel multiplication
output*
input
box average
Gaussian blur
normalizedGaussian function
Equation of Gaussian BlurEquation of Gaussian Blur
( )∑∈
−=S
IGIGBq
qp qp ||||][ σ
Same idea: weighted average of pixels.
0
1
unrelatedpixels
unrelatedpixels
uncertainpixels
uncertainpixels
relatedpixels
Gaussian ProfileGaussian Profile
pixelposition
pixelweight
⎟⎟⎠
⎞⎜⎜⎝
⎛−= 2
2
2exp
21)(
σπσσxxG
size of the window
Spatial ParameterSpatial Parameter
( )∑∈
−=S
IGIGBq
qp qp ||||][ σ
small σ large σ
input
limited smoothing strong smoothing
How to set σHow to set σ
• Depends on the application.
• Common strategy: proportional to image size– e.g. 2% of the image diagonal
– property: independent of image resolution
Properties of Gaussian BlurProperties of Gaussian Blur
• Weights independent of spatial location
– linear convolution
– well-known operation
– efficient computation (recursive algorithm, FFT…)
Properties of Gaussian BlurProperties of Gaussian Blur
• Does smooth images
• But smoothes too much:edges are blurred.– Only spatial distance matters
– No edge term
input
output
( )∑∈
−=S
IGIGBq
qp qp ||||][ σspace
Blur Comes from Averaging across EdgesBlur Comes from Averaging across Edges
*
*
*
input output
Same Gaussian kernel everywhere.
Bilateral FilterNo Averaging across EdgesBilateral FilterNo Averaging across Edges
*
*
*
input output
The kernel shape depends on the image content.
[Aurich 95, Smith 97, Tomasi 98]
space weight
not new
range weight
I
new
normalizationfactor
new
Bilateral Filter Definition:an Additional Edge TermBilateral Filter Definition:an Additional Edge Term
( ) ( )∑∈
−−=S
IIIGGW
IBFq
qqpp
p qp ||||||1][rs σσ
Same idea: weighted average of pixels.
Illustration a 1D ImageIllustration a 1D Image
• 1D image = line of pixels
• Better visualized as a plot
pixelintensity
pixel position
space
Gaussian Blur and Bilateral FilterGaussian Blur and Bilateral Filter
space rangenormalization
Gaussian blur
( ) ( )∑∈
−−=S
IIIGGW
IBFq
qqpp
p qp ||||||1][rs σσ
Bilateral filter[Aurich 95, Smith 97, Tomasi 98]
space
spacerange
p
p
q
q
( )∑∈
−=S
IGIGBq
qp qp ||||][ σ
qqpp
Bilateral Filter on a Height FieldBilateral Filter on a Height Field
output input
( ) ( )∑∈
−−=S
IIIGGW
IBFq
qqpp
p qp ||||||1][rs σσ
pp
reproducedfrom [Durand 02]
Space and Range ParametersSpace and Range Parameters
• space σs : spatial extent of the kernel, size of the considered neighborhood.
• range σr : “minimum” amplitude of an edge
( ) ( )∑∈
−−=S
IIIGGW
IBFq
qqpp
p qp ||||||1][rs σσ
Influence of PixelsInfluence of Pixels
pp
Only pixels close in space and in range are considered.
space
range
σs = 2
σs = 6
σs = 18
σr = 0.1 σr = 0.25σr = ∞
(Gaussian blur)
input
Exploring the Parameter SpaceExploring the Parameter Space
σs = 2
σs = 6
σs = 18
σr = 0.1 σr = 0.25σr = ∞
(Gaussian blur)
input
Varying the Range ParameterVarying the Range Parameter
input
σr = 0.1
σr = 0.25
σr = ∞(Gaussian blur)
σs = 2
σs = 6
σs = 18
σr = 0.1 σr = 0.25σr = ∞
(Gaussian blur)
input
Varying the Space ParameterVarying the Space Parameter
input
σs = 2
σs = 6
σs = 18
How to Set the ParametersHow to Set the Parameters
Depends on the application. For instance:
• space parameter: proportional to image size– e.g., 2% of image diagonal
• range parameter: proportional to edge amplitude– e.g., mean or median of image gradients
• independent of resolution and exposure
A FewMore Advanced
Remarks
A FewMore Advanced
Remarks
Bilateral Filter Crosses Thin LinesBilateral Filter Crosses Thin Lines• Bilateral filter averages across
features thinner than ~2σs
• Desirable for smoothing: more pixels = more robust• Different from diffusion that stops at thin lines
close-up kernel
Iterating the Bilateral FilterIterating the Bilateral Filter
• Generate more piecewise-flat images
• Often not needed in computational photo.
][ )()1( nn IBFI =+
input
1 iteration
2 iterations
4 iterations
Bilateral Filtering Color ImagesBilateral Filtering Color Images
( ) ( )∑∈
−−=S
IIIGGW
IBFq
qqpp
p qp ||||||1][rs σσ
( ) ( )∑∈
−−=S
GGW
IBFq
qqpp
p CCCqp ||||||||1][rs σσ
For gray-level images
For color images
intensity difference
color difference
The bilateral filter isThe bilateral filter isextremely easy to adapt to your need.extremely easy to adapt to your need.
scalar
3D vector (RGB, Lab)
input
output
Hard to ComputeHard to Compute
• Nonlinear
• Complex, spatially varying kernels– Cannot be precomputed, no FFT…
• Brute-force implementation is slow > 10min
( ) ( )∑∈
−−=S
IIIGGW
IBFq
qqpp
p qp ||||||1][rs σσ
input smoothed(structure, large scale)
residual(texture, small scale)
edge-preserving: Bilateral Filter
Decomposition into Large-scale and Small-scale LayersDecomposition into Large-scale and Small-scale Layers
Weighted Average of PixelsWeighted Average of Pixels
space rangenormalization
( ) ( )∑∈
−−=S
IIIGGW
IBFq
qqpp
p qp ||||||1][rs σσ
space
range
p
q
• Depends on spatial distanceand intensity difference– Pixels across edges have almost influence
A Gentle Introductionto Bilateral Filteringand its Applications
A Gentle Introductionto Bilateral Filteringand its Applications
Efficient Implementationsof the Bilateral Filter
Sylvain Paris – MIT CSAIL
OutlineOutline
• Brute-force Implementation
• Separable Kernel [Pham and Van Vliet 05]
• Box Kernel [Weiss 06]
• 3D Kernel [Paris and Durand 06]
Brute-force ImplementationBrute-force Implementation
For each pixel pFor each pixel q
Compute
8 megapixel photo: 64,000,000,000,000 iterations!
V E R Y S L O W !V E R Y S L O W !More than 10 minutes per imageMore than 10 minutes per image
( ) ( )∑∈
−−=S
IIIGGW
IBFq
qqpp
p qp ||||||1][rs σσ
( ) ( ) qqpqp IIIGG ||||||rs
−− σσ
ComplexityComplexity
• Complexity = “how many operations are needed, how this number varies”
• S = space domain = set of pixel positions
• | S | = cardinality of S = number of pixels– In the order of 1 to 10 millions
• Brute-force implementation: )|(| 2SΟ
Better Brute-force ImplementationBetter Brute-force Implementation
Idea: Far away pixels are negligible
For each pixel pa. For each pixel q such that || p – q || < cte × σs
looking at all pixels looking at neighbors only
DiscussionDiscussion
• Complexity:
• Fast for small kernels: σs ~ 1 or 2 pixels
• BUT: slow for larger kernels
)|(| 2sσ×SΟ
neighborhood area
OutlineOutline
• Brute-force Implementation
• Separable Kernel [Pham and Van Vliet 05]
• Box Kernel [Weiss 06]
• 3D Kernel [Paris and Durand 06]
Separable KernelSeparable Kernel
• Strategy: filter the rows then the columns
• Two “cheap” 1D filters instead of an “expensive” 2D filter
[Pham and Van Vliet 05]
DiscussionDiscussion
• Complexity: – Fast for small kernels (<10 pixels)
• Approximation: BF kernel not separable– Satisfying at strong edges and uniform areas
– Can introduce visible streaks on textured regions
)|(| sσ×SΟ
input
brute-forceimplementation
separable kernelmostly OK,
some visible artifacts(streaks)
OutlineOutline
• Brute-force Implementation
• Separable Kernel [Pham and Van Vliet 05]
• Box Kernel [Weiss 06]
• 3D Kernel [Paris and Durand 06]
Box KernelBox Kernel
• Bilateral filter with a square box window
• The bilateral filter can be computed only from the list of pixels in a square neighborhood.
[Weiss 06]
( ) ( )∑∈
−−=S
IIIGBW
IYq
qqpp
p qp ||||||1][rs σσ
[Yarovlasky 85]
box window
( )∑∈
−=s
r||1][
σ
σB
IIIGW
IYq
qqpp
p
restrict the sum
independent of position q
Box KernelBox Kernel• Idea: fast histograms of square windows
[Weiss 06]
input:full histogram is known
update:add one line, remove one line
Tracking one window
Box KernelBox Kernel• Idea: fast histograms of square windows
[Weiss 06]
input:full histograms are known
update:add one line, remove one line,
add two pixels, remove two pixels
Tracking two windows at the same time
DiscussionDiscussion
• Complexity: – always fast
• Only single-channel images
• Exploit vector instructions of CPU
• Visually satisfying results (no artifacts)– 3 passes to remove artifacts due to
box windows (Mach bands)
)log|(| sσ×SΟ
1 iteration
3 iterations
input
brute-forceimplementation
box kernelvisually different,
yet no artifacts
OutlineOutline
• Brute-force Implementation
• Separable Kernel [Pham and Van Vliet 05]
• Box Kernel [Weiss 06]
• 3D Kernel [Paris and Durand 06]
3D Kernel3D Kernel
• Idea: represent image data such that the weights depend only on the distance between points
[Paris and Durand 06]
pixelintensity
pixel position
1D image
PlotI = f ( x )
far in range
close in space
1st Step: Re-arranging Symbols1st Step: Re-arranging Symbols
( ) ( )
( ) ( )∑
∑
∈
∈
−−=
−−=
S
S
IIGGW
IIIGGW
IBF
qqpp
qqqp
pp
qp
qp
||||||
||||||1][
rs
rs
σσ
σσ
( ) ( )
( ) ( ) 1||||||
||||||][
rs
rs
∑
∑
∈
∈
−−=
−−=
S
S
IIGGW
IIIGGIBFW
qqpp
qqqppp
qp
qp
σσ
σσ
Multiply first equation by Wp
1st Step: Summary1st Step: Summary
• Similar equations
• No normalization factor anymore
• Don’t forget to divide at the end
( ) ( )
( ) ( ) 1||||||
||||||][
rs
rs
∑
∑
∈
∈
−−=
−−=
S
S
IIGGW
IIIGGIBFW
qqpp
qqqppp
qp
qp
σσ
σσ
2nd Step: Higher-dimensional Space2nd Step: Higher-dimensional Space
pp
space
range
• “Product of two Gaussians” = higher dim. Gaussian
2nd Step: Higher-dimensional Space2nd Step: Higher-dimensional Space
pp
space
range
• 0 almost everywhere, I at “plot location”
2nd Step: Higher-dimensional Space2nd Step: Higher-dimensional Space
pp
• 0 almost everywhere, I at “plot location”
• Weighted average at each point = Gaussian blur
2nd Step: Higher-dimensional Space2nd Step: Higher-dimensional Space
pp
• 0 almost everywhere, I at “plot location”
• Weighted average at each point = Gaussian blur
• Result is at “plot location”
higher dimensional functions
Gaussian blur
division
slicing
Higherdimensional
Homogeneousintensity
Higherdimensional
Homogeneousintensity
New num. scheme:• simple operations• complex space
higher dimensional functions
Gaussian convolution
division
slicing
D O W N S A M P L E
U P S A M P L E
Heavilydownsampled
Heavilydownsampled
Strategy:downsampledconvolution
Conceptual view,not exactly the actual algorithm
Actual AlgorithmActual Algorithm
• Never compute full resolution– On-the-fly downsampling
– On-the-fly upsampling
• 3D sampling rate = ),,( rss σσσ
Pseudo-code: StartPseudo-code: Start
• Input– image I
– Gaussian parameters σs and σr
• Output: BF [ I ]
• Data structure: 3D arrays wi and w (init. to 0)
Pseudo-code: On-the-fly DownsamplingPseudo-code: On-the-fly Downsampling
• For each pixel
– Downsample:
– Update:
SYX ∈),(
⎟⎟⎠
⎞⎜⎜⎝
⎛⎥⎦
⎤⎢⎣
⎡⎥⎦
⎤⎢⎣
⎡⎥⎦
⎤⎢⎣
⎡=
rss
),(,,),,(σσσ
YXIYXzyx
1),,(),(),,(
=+=+
zyxwYXIzyxwi
D O W N S A M P L E
U P S A M P L E
[ ] = closest int.
Pseudo-code: ConvolvingPseudo-code: Convolving• For each axis , , and
– For each 3D point
• Apply a Gaussian mask ( 1 , 4 , 6 , 4 , 1 ) to wi and we.g., for the x axis:
wi’(x) = wi(x-2) + 4.wi(x-1) + 6.wi(x) + 4.wi(x+1) + wi(x+2)
xr yr zr
),,( zyx
D O W N S A M P L E
U P S A M P L E
Pseudo-code: On-the-fly UpsamplingPseudo-code: On-the-fly Upsampling
• For each pixel
– Linearly interpolate the values in the 3D arrays
SYX ∈),(
( )( )),(,,,einterpolat
),(,,,einterpolat),(][YXIYXwYXIYXwiYXIBF =
D O W N S A M P L E
U P S A M P L E
DiscussionDiscussion
• Complexity:
• Fast for medium and large kernels– Can be ported on GPU [Chen 07]: always very fast
• Can be extended to color images but slower
• Visually similar to brute-force computation
⎟⎟⎠
⎞⎜⎜⎝
⎛+
r2s
||||||σσRSSΟ
numberof pixels
numberof 3D cells
| R | : number of gray levels
input
brute-forceimplementation
3D kernelvisually similar
Running TimesRunning Times
separable kernel
3D kernel box kernel
brut
e fo
rce
How to Choose an Implementation?How to Choose an Implementation?
Depends a lot on the application. A few guidelines:
• Brute-force: tiny kernels or if accuracy is paramount
• Box Kernel: for short running times on CPU with any kernel size, e.g. editing package
• 3D kernel: – if GPU available
– if only CPU available: large kernels, color images, cross BF(e.g., good for computational photography)