Sampling and reconstruction - Hacettepe Üniversitesierkut/bco511.s12/w10...Hacettepe BCO511 Spring...
Transcript of Sampling and reconstruction - Hacettepe Üniversitesierkut/bco511.s12/w10...Hacettepe BCO511 Spring...
Hacettepe BCO511 Spring 2012 • Week10
Sampling and reconstruction
Week 10
1
Acknowledgement: The course slides are adapted from the slides prepared by Steve Marschner of Cornell University
Hacettepe BCO511 Spring 2012 • Week10
Sampled representations
• How to store and compute with continuous functions?
• Common scheme for representation: sampleswrite down the function’s values at many points
[FvD
FH fi
g.14
.14b
/ W
olbe
rg]
2
Hacettepe BCO511 Spring 2012 • Week10
Reconstruction
• Making samples back into a continuous functionfor output (need realizable method)
for analysis or processing (need mathematical method)
amounts to “guessing” what the function did in between
[FvD
FH fi
g.14
.14b
/ W
olbe
rg]
3
Hacettepe BCO511 Spring 2012 • Week10
Filtering
• Processing done on a functioncan be executed in continuous form (e.g. analog circuit)
but can also be executed using sampled representation
• Simple example: smoothing by averaging
4
Hacettepe BCO511 Spring 2012 • Week10
Roots of sampling
• Nyquist 1928; Shannon 1949famous results in information theory
• 1940s: first practical uses in telecommunications
• 1960s: first digital audio systems
• 1970s: commercialization of digital audio
• 1982: introduction of the Compact Discthe first high-profile consumer application
• This is why all the terminology has a communications or audio “flavor”early applications are 1D; for us 2D (images) is important
5
Hacettepe BCO511 Spring 2012 • Week10
Sampling in digital audio
• Recording: sound to analog to samples to disc
• Playback: disc to samples to analog to sound againhow can we be sure we are filling in the gaps correctly?
6
Hacettepe BCO511 Spring 2012 • Week10
Undersampling
• What if we “missed” things between the samples?
• Simple example: undersampling a sine waveunsurprising result: information is lost
surprising result: indistinguishable from lower frequency
also was always indistinguishable from higher frequencies
aliasing: signals “traveling in disguise” as other frequencies
7
Hacettepe BCO511 Spring 2012 • Week10
Undersampling
• What if we “missed” things between the samples?
• Simple example: undersampling a sine waveunsurprising result: information is lost
surprising result: indistinguishable from lower frequency
also was always indistinguishable from higher frequencies
aliasing: signals “traveling in disguise” as other frequencies
7
Hacettepe BCO511 Spring 2012 • Week10
Undersampling
• What if we “missed” things between the samples?
• Simple example: undersampling a sine waveunsurprising result: information is lost
surprising result: indistinguishable from lower frequency
also was always indistinguishable from higher frequencies
aliasing: signals “traveling in disguise” as other frequencies
7
Hacettepe BCO511 Spring 2012 • Week10
Undersampling
• What if we “missed” things between the samples?
• Simple example: undersampling a sine waveunsurprising result: information is lost
surprising result: indistinguishable from lower frequency
also was always indistinguishable from higher frequencies
aliasing: signals “traveling in disguise” as other frequencies
7
Hacettepe BCO511 Spring 2012 • Week10
Undersampling
• What if we “missed” things between the samples?
• Simple example: undersampling a sine waveunsurprising result: information is lost
surprising result: indistinguishable from lower frequency
also was always indistinguishable from higher frequencies
aliasing: signals “traveling in disguise” as other frequencies
7
Hacettepe BCO511 Spring 2012 • Week10
Undersampling
• What if we “missed” things between the samples?
• Simple example: undersampling a sine waveunsurprising result: information is lost
surprising result: indistinguishable from lower frequency
also was always indistinguishable from higher frequencies
aliasing: signals “traveling in disguise” as other frequencies
7
Hacettepe BCO511 Spring 2012 • Week10
Preventing aliasing
• Introduce lowpass filters:remove high frequencies leaving only safe, low frequencies
choose lowest frequency in reconstruction (disambiguate)
8
Hacettepe BCO511 Spring 2012 • Week10
Preventing aliasing
• Introduce lowpass filters:remove high frequencies leaving only safe, low frequencies
choose lowest frequency in reconstruction (disambiguate)
8
Hacettepe BCO511 Spring 2012 • Week10
Linear filtering: a key idea
• Transformations on signals; e.g.:bass/treble controls on stereo
blurring/sharpening operations in image editing
smoothing/noise reduction in tracking
• Key propertieslinearity: filter(f + g) = filter(f) + filter(g)
shift invariance: behavior invariant to shifting the input• delaying an audio signal• sliding an image around
• Can be modeled mathematically by convolution
9
Hacettepe BCO511 Spring 2012 • Week10
Convolution warm-up
• basic idea: define a new function by averaging over a sliding window
• a simple example to start off: smoothing
10
Hacettepe BCO511 Spring 2012 • Week10
Convolution warm-up
• basic idea: define a new function by averaging over a sliding window
• a simple example to start off: smoothing
10
Hacettepe BCO511 Spring 2012 • Week10
Convolution warm-up
• basic idea: define a new function by averaging over a sliding window
• a simple example to start off: smoothing
10
Hacettepe BCO511 Spring 2012 • Week10
Convolution warm-up
• basic idea: define a new function by averaging over a sliding window
• a simple example to start off: smoothing
10
Hacettepe BCO511 Spring 2012 • Week10
Convolution warm-up
• basic idea: define a new function by averaging over a sliding window
• a simple example to start off: smoothing
10
Hacettepe BCO511 Spring 2012 • Week10
Convolution warm-up
• Same moving average operation, expressed mathematically:
11
Hacettepe BCO511 Spring 2012 • Week10
Discrete convolution
• Simple averaging:
every sample gets the same weight
• Convolution: same idea but with weighted average
each sample gets its own weight (normally zero far away)
• This is all convolution is: it is a moving weighted average
12
Hacettepe BCO511 Spring 2012 • Week10
Filters
• Sequence of weights a[j] is called a filter
• Filter is nonzero over its region of supportusually centered on zero: support radius r
• Filter is normalized so that it sums to 1.0this makes for a weighted average, not just any
old weighted sum
• Most filters are symmetric about 0since for images we usually want to treat
left and right the same
a box !lter
13
Hacettepe BCO511 Spring 2012 • Week10
Convolution and filtering
• Can express sliding average as convolution with a box filter
• abox = […, 0, 1, 1, 1, 1, 1, 0, …]
14
Hacettepe BCO511 Spring 2012 • Week10
Example: box and step
15
Hacettepe BCO511 Spring 2012 • Week10
Example: box and step
15
Hacettepe BCO511 Spring 2012 • Week10
Example: box and step
15
Hacettepe BCO511 Spring 2012 • Week10
Example: box and step
15
Hacettepe BCO511 Spring 2012 • Week10
Example: box and step
15
Hacettepe BCO511 Spring 2012 • Week10
Convolution and filtering
• Convolution applies with any sequence of weights
• Example: bell curve (gaussian-like) […, 1, 4, 6, 4, 1, …]/16
16
Hacettepe BCO511 Spring 2012 • Week10
Convolution and filtering
• Convolution applies with any sequence of weights
• Example: bell curve (gaussian-like) […, 1, 4, 6, 4, 1, …]/16
16
Hacettepe BCO511 Spring 2012 • Week10
And in pseudocode…
17
Hacettepe BCO511 Spring 2012 • Week10
Discrete convolution
• Notation:
• Convolution is a multiplication-like operationcommutative
associative
distributes over addition
scalars factor out
identity: unit impulse e = […, 0, 0, 1, 0, 0, …]
• Conceptually no distinction between filter and signal
18
Hacettepe BCO511 Spring 2012 • Week10
Discrete filtering in 2D
• Same equation, one more index
now the filter is a rectangle you slide around over a grid of numbers
• Commonly applied to imagesblurring (using box, using gaussian, …)
sharpening (impulse minus blur)
• Usefulness of associativityoften apply several filters one after another: (((a * b1) * b2) * b3)
this is equivalent to applying one filter: a * (b1 * b2 * b3)
19
Hacettepe BCO511 Spring 2012 • Week10
And in pseudocode…
20
Hacettepe BCO511 Spring 2012 • Week10
original | box blur sharpened | gaussian blur[Philip Greenspun]
21
Hacettepe BCO511 Spring 2012 • Week10
original | box blur sharpened | gaussian blur[Philip Greenspun]
21
Hacettepe BCO511 Spring 2012 • Week10
Optimization: separable filters
• basic alg. is O(r2): large filters get expensive fast!
• definition: a2(x,y) is separable if it can be written as:
this is a useful property for filters because it allows factoring:
22
Hacettepe BCO511 Spring 2012 • Week10
Separable filtering
first, convolve with this
23
Hacettepe BCO511 Spring 2012 • Week10
Separable filtering
first, convolve with this
second, convolve with this
23
Hacettepe BCO511 Spring 2012 • Week10
Continuous convolution: warm-up
• Can apply sliding-window average to a continuous function just as welloutput is continuous
integration replaces summation
24
Hacettepe BCO511 Spring 2012 • Week10
Continuous convolution: warm-up
• Can apply sliding-window average to a continuous function just as welloutput is continuous
integration replaces summation
24
Hacettepe BCO511 Spring 2012 • Week10
Continuous convolution: warm-up
• Can apply sliding-window average to a continuous function just as welloutput is continuous
integration replaces summation
24
Hacettepe BCO511 Spring 2012 • Week10
Continuous convolution: warm-up
• Can apply sliding-window average to a continuous function just as welloutput is continuous
integration replaces summation
24
Hacettepe BCO511 Spring 2012 • Week10
Continuous convolution: warm-up
• Can apply sliding-window average to a continuous function just as welloutput is continuous
integration replaces summation
24
Hacettepe BCO511 Spring 2012 • Week10
Continuous convolution
• Sliding average expressed mathematically:
note difference in normalization (only for box)
• Convolution just adds weights
weighting is now by a functionweighted integral is like weighted average
again bounds are set by support of f(x)
25
Hacettepe BCO511 Spring 2012 • Week10
One more convolution
• Continuous–discrete convolution
used for reconstruction and resampling
26
Hacettepe BCO511 Spring 2012 • Week10
Continuous-discrete convolution
27
Hacettepe BCO511 Spring 2012 • Week10
Continuous-discrete convolution
27
Hacettepe BCO511 Spring 2012 • Week10
Continuous-discrete convolution
27
Hacettepe BCO511 Spring 2012 • Week10
Continuous-discrete convolution
27
Hacettepe BCO511 Spring 2012 • Week10
Continuous-discrete convolution
27
Hacettepe BCO511 Spring 2012 • Week10
Resampling
• Changing the sample ratein images, this is enlarging and reducing
• Creating more samples:increasing the sample rate
“upsampling”
“enlarging”
• Ending up with fewer samples:decreasing the sample rate
“downsampling”
“reducing”
28
Hacettepe BCO511 Spring 2012 • Week10
Resampling
• Reconstruction creates a continuous functionforget its origins, go ahead and sample it
29
Hacettepe BCO511 Spring 2012 • Week10
Resampling
• Reconstruction creates a continuous functionforget its origins, go ahead and sample it
29
Hacettepe BCO511 Spring 2012 • Week10
Resampling
• Reconstruction creates a continuous functionforget its origins, go ahead and sample it
29
Hacettepe BCO511 Spring 2012 • Week10
Resampling
• Reconstruction creates a continuous functionforget its origins, go ahead and sample it
29
Hacettepe BCO511 Spring 2012 • Week10
Resampling
• Reconstruction creates a continuous functionforget its origins, go ahead and sample it
29
Hacettepe BCO511 Spring 2012 • Week10
And in pseudocode…
30
Hacettepe BCO511 Spring 2012 • Week10
Cont.–disc. convolution in 2D
• same convolution—just two variables now
loop over nearby pixels, average using filter weight
looks like discrete filter,but offsets are not integersand filter is continuous
remember placement of filterrelative to grid is variable
31
Hacettepe BCO511 Spring 2012 • Week10
Cont.–disc. convolution in 2D
32
Hacettepe BCO511 Spring 2012 • Week10
Separable filters for resampling
• just as in filtering, separable filters are usefulseparability in this context is a statement about a continuous filter, rather than a
discrete one:
• resample in two passes, one resampling each row and one resampling each column
• intermediate storage required: product of one dimension of src. and the other dimension of dest.
• same yucky details about boundary conditions
33
Hacettepe BCO511 Spring 2012 • Week10
two-stage resampling using aseparable !lter
[Phi
lip G
reen
spun
]
34
Hacettepe BCO511 Spring 2012 • Week10
A gallery of filters
• Box filterSimple and cheap
• Tent filterLinear interpolation
• Gaussian filterVery smooth antialiasing filter
• B-spline cubicVery smooth
• Catmull-rom cubicInterpolating
• Mitchell-Netravali cubicGood for image upsampling
35
Hacettepe BCO511 Spring 2012 • Week10
Box filter
36
Hacettepe BCO511 Spring 2012 • Week10
Tent filter
37
Hacettepe BCO511 Spring 2012 • Week10
Gaussian filter
38
Hacettepe BCO511 Spring 2012 • Week10
B-Spline cubic
39
fB(x) =16
�⌅⇤
⌅⇥
�3(1� |x|)3 + 3(1� |x|)2 + 3(1� |x|) + 1 �1 ⇥ x ⇥ 1,
(2� |x|)3 1 ⇥ |x| ⇥ 2,
0 otherwise.
Hacettepe BCO511 Spring 2012 • Week10
Catmull-Rom cubic
40
fC(x) =12
�⌅⇤
⌅⇥
�3(1� |x|)3 + 4(1� |x|)2 + (1� |x|) �1 ⇥ x ⇥ 1,
(2� |x|)3 � (2� |x|)2 1 ⇥ |x| ⇥ 2,
0 otherwise.
Hacettepe BCO511 Spring 2012 • Week10
Michell-Netravali cubic
41
fM (x) =13fB(x) +
23fC(x)
=118
�⌅⇤
⌅⇥
�21(1� |x|)3 + 27(1� |x|)2 + 9(1� |x|) + 1 �1 ⇥ x ⇥ 1,
7(2� |x|)3 � 6(2� |x|)2 1 ⇥ |x| ⇥ 2,
0 otherwise.
Hacettepe BCO511 Spring 2012 • Week10
Effects of reconstruction filters
• For some filters, the reconstruction process winds up implementing a simple algorithm
• Box filter (radius 0.5): nearest neighbor samplingbox always catches exactly one input point
it is the input point nearest the output point
so output[i, j] = input[round(x(i)), round(y(j))]x(i) computes the position of the output coordinate i on the input grid
• Tent filter (radius 1): linear interpolationtent catches exactly 2 input points
weights are a and (1 – a)result is straight-line interpolation from one point to the next
42
Hacettepe BCO511 Spring 2012 • Week10
Properties of filters
• Degree of continuity
• Impulse response
• Interpolating or no
• Ringing, or overshoot
interpolating filter used for reconstruction
43
Hacettepe BCO511 Spring 2012 • Week10
Ringing, overshoot, ripples
• Overshootcaused by
negative filtervalues
• Ripplesconstant in,
non-const. out
ripple free when:
44
Hacettepe BCO511 Spring 2012 • Week10
Constructing 2D filters
• Separable filters (most common approach)
45
Hacettepe BCO511 Spring 2012 • Week10
Yucky details
• What about near the edge?the filter window falls off the edge of the image
need to extrapolate
methods:• clip filter (black)• wrap around• copy edge• reflect across edge• vary filter near edge
46
[Phi
lip G
reen
spun
]
Hacettepe BCO511 Spring 2012 • Week10
Yucky details
• What about near the edge?the filter window falls off the edge of the image
need to extrapolate
methods:• clip filter (black)• wrap around• copy edge• reflect across edge• vary filter near edge
46
[Phi
lip G
reen
spun
]
Hacettepe BCO511 Spring 2012 • Week10
Yucky details
• What about near the edge?the filter window falls off the edge of the image
need to extrapolate
methods:• clip filter (black)• wrap around• copy edge• reflect across edge• vary filter near edge
46
[Phi
lip G
reen
spun
]
Hacettepe BCO511 Spring 2012 • Week10
Yucky details
• What about near the edge?the filter window falls off the edge of the image
need to extrapolate
methods:• clip filter (black)• wrap around• copy edge• reflect across edge• vary filter near edge
46
[Phi
lip G
reen
spun
]
Hacettepe BCO511 Spring 2012 • Week10
Yucky details
• What about near the edge?the filter window falls off the edge of the image
need to extrapolate
methods:• clip filter (black)• wrap around• copy edge• reflect across edge• vary filter near edge
46
[Phi
lip G
reen
spun
]
Hacettepe BCO511 Spring 2012 • Week10
Yucky details
• What about near the edge?the filter window falls off the edge of the image
need to extrapolate
methods:• clip filter (black)• wrap around• copy edge• reflect across edge• vary filter near edge
46
[Phi
lip G
reen
spun
]
Hacettepe BCO511 Spring 2012 • Week10
Yucky details
• What about near the edge?the filter window falls off the edge of the image
need to extrapolate
methods:• clip filter (black)• wrap around• copy edge• reflect across edge• vary filter near edge
46
[Phi
lip G
reen
spun
]
Hacettepe BCO511 Spring 2012 • Week10
Yucky details
• What about near the edge?the filter window falls off the edge of the image
need to extrapolate
methods:• clip filter (black)• wrap around• copy edge• reflect across edge• vary filter near edge
46
[Phi
lip G
reen
spun
]
Hacettepe BCO511 Spring 2012 • Week10
Yucky details
• What about near the edge?the filter window falls off the edge of the image
need to extrapolate
methods:• clip filter (black)• wrap around• copy edge• reflect across edge• vary filter near edge
46
[Phi
lip G
reen
spun
]
Hacettepe BCO511 Spring 2012 • Week10
Yucky details
• What about near the edge?the filter window falls off the edge of the image
need to extrapolate
methods:• clip filter (black)• wrap around• copy edge• reflect across edge• vary filter near edge
46
[Phi
lip G
reen
spun
]
Hacettepe BCO511 Spring 2012 • Week10
Yucky details
• What about near the edge?the filter window falls off the edge of the image
need to extrapolate
methods:• clip filter (black)• wrap around• copy edge• reflect across edge• vary filter near edge
46
[Phi
lip G
reen
spun
]
Hacettepe BCO511 Spring 2012 • Week10
Yucky details
• What about near the edge?the filter window falls off the edge of the image
need to extrapolate
methods:• clip filter (black)• wrap around• copy edge• reflect across edge• vary filter near edge
46
[Phi
lip G
reen
spun
]
Hacettepe BCO511 Spring 2012 • Week10
Yucky details
• What about near the edge?the filter window falls off the edge of the image
need to extrapolate
methods:• clip filter (black)• wrap around• copy edge• reflect across edge• vary filter near edge
46
[Phi
lip G
reen
spun
]
Hacettepe BCO511 Spring 2012 • Week10
Reducing and enlarging
• Very common operationdevices have differing resolutions
applications have different memory/quality tradeoffs
• Also very commonly done poorly
• Simple approach: drop/replicate pixels
• Correct approach: use resampling
47
Hacettepe BCO511 Spring 2012 • Week10
Resampling example
48
Hacettepe BCO511 Spring 2012 • Week10
Resampling example
48
Hacettepe BCO511 Spring 2012 • Week10
Resampling example
48
Hacettepe BCO511 Spring 2012 • Week10
Resampling example
48
Hacettepe BCO511 Spring 2012 • Week10
Resampling example
48
Hacettepe BCO511 Spring 2012 • Week10
Resampling example
48
Hacettepe BCO511 Spring 2012 • Week10
Resampling example
48
Hacettepe BCO511 Spring 2012 • Week10
Resampling example
48
Hacettepe BCO511 Spring 2012 • Week10
Resampling example
48
Hacettepe BCO511 Spring 2012 • Week10
Resampling example
48
Hacettepe BCO511 Spring 2012 • Week10
Reducing and enlarging
• Very common operationdevices have differing resolutions
applications have different memory/quality tradeoffs
• Also very commonly done poorly
• Simple approach: drop/replicate pixels
• Correct approach: use resampling
49
Hacettepe BCO511 Spring 2012 • Week10
1000 pixel width [Philip Greenspun]
50
Hacettepe BCO511 Spring 2012 • Week10
250 pixel width
by dropping pixels gaussian !lter
[Philip Greenspun]
51
Hacettepe BCO511 Spring 2012 • Week10
4000 pixel width
box reconstruction !lter bicubic reconstruction !lter
[Phi
lip G
reen
spun
]
52
Hacettepe BCO511 Spring 2012 • Week10
Types of artifacts
• Garden varietywhat we saw in this natural image
fine features become jagged or sparkle
• Moiré patterns
53
Hacettepe BCO511 Spring 2012 • Week10
600ppi scan of a color halftone image
[Hea
rn &
Bak
er c
over
]
54
Hacettepe BCO511 Spring 2012 • Week10
downsampling a high resolution scan
by dropping pixels gaussian !lter
[Hea
rn &
Bak
er c
over
]
55
Hacettepe BCO511 Spring 2012 • Week10
Types of artifacts
• Garden varietywhat we saw in this natural image
fine features become jagged or sparkle
• Moiré patternscaused by repetitive patterns in input
produce large-scale artifacts; highly visible
• These artifacts are aliasing just like in the audio example earlier
• How do I know what filter is best at preventing aliasing?practical answer: experience
theoretical answer: there is another layer of cool math behind all this• based on Fourier transforms• provides much insight into aliasing, filtering, sampling, and reconstruction
56