Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction...
-
Upload
bennett-strickland -
Category
Documents
-
view
230 -
download
4
Transcript of Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction...
![Page 1: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/1.jpg)
Super-Resolution
Barak Zackay
Yaron Kassner
![Page 2: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/2.jpg)
Outline
• Introduction to Super-Resolution• Reconstruction Based Super Resolution
– An Algorithm– Limits on Reconstruction Based Super Resolution
• Example Based Super Resolution– Halucination– Example Based– Single Image Super Resolution
• Summary
![Page 3: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/3.jpg)
Introduction to Super Resolution
![Page 4: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/4.jpg)
Definition of the Problem
• Super-resolution is the process of combining multiple low resolution images to form a higher resolution one.
• No cheating! – Resulting image should represent reality better than
all the input images.
![Page 5: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/5.jpg)
The Imaging Process
![Page 6: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/6.jpg)
Physical Properties
• Each camera suffers from some inherent optical issues:– Finite size of the aperture - generates blur, modeled
by the Point-Spread-Function (PSF).
– Noise
![Page 7: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/7.jpg)
Mathematical Model
• Each pixel in the resulting image is given by:
• Loi(m) – the i-th LR image in pixel m.
• Ei (x) – total photon count from the direction x
• PSFi – Point Spread Function
![Page 8: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/8.jpg)
Deresolution
• Given HR image
• Project to LR image
• Each LR pixel is a linear combination of HR pixels
HR HR HR
LR
![Page 9: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/9.jpg)
Reconstruction-based Super Resolution
• Reconstruct hidden HR pixels out of known linear combinations.
HR HR HR HRHR HR HR HR HR HR HR
LRLR
LRLR
LR
LR
LR
LR
LRLR
LRLR
LR
LR
LR
LR
![Page 10: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/10.jpg)
Example-BasedSuper Resolution
• Use prior knowledge to reconstruct a HR image.
Prior Knowledge of faces
![Page 11: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/11.jpg)
Reconstruction Based Super Resolution
fromImproving Resolution by Image Registration
Michal Irani and Shmuel Peleg
![Page 12: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/12.jpg)
Basic Idea
• The HR image should create the LR images when deresoluted.
![Page 13: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/13.jpg)
Notation
• : The kth observed LR image.• : The approximation to the HR image after n
iterations.• : The LR image obtained by applying the
simulated imaging process to .• : The point spread function of the imaging blur.• : a HR pixel• : a LR pixel influenced by x• : The center of the receptive field of y.
nkg
nf
kg
PSFh
x
y
nf
yz
![Page 14: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/14.jpg)
Problem Formulation
• Find a HR image , that gives .
–
nf gg n
![Page 15: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/15.jpg)
Algorithm Overview
• Register the LR images.
• Guess the HR image .
• Iteration n:– Simulate the imaging process to create
from .– Compare and . – Correct in the direction of the error.
• output
nf0
nkg
nf
nkg kg
nf
nf
![Page 16: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/16.jpg)
Registration
HR HR HR HRHR HR HR HR HR HR HR
LRLR
LRLR
LRLR
LRLR
LRLR
LRLR
LRLR
LRLR
LRLR
LRLR
LRLR
LRLR
![Page 17: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/17.jpg)
Iteration• Take the current guess.• Decrease its resolution to get • Update each HR pixel x according to the error in all LR pixels (y) it
influences.c is a constant normalizing factor.
– c is a constant normalizing factor.– Yk,x is the group of all pixels y that are influenced by x.
– is a back-projection kernel applied on that represents the way the HR pixel x should be updated from y.
BPxyh yzx
nkg
![Page 18: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/18.jpg)
Wasach
One of three input images
Initial guess (average of input images)
Output
![Page 19: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/19.jpg)
Debluring
Original Image Blurred Image Restored Image
![Page 20: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/20.jpg)
Wasach
Initial GuessBlurred Image Restored Image
![Page 21: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/21.jpg)
Limits on Reconstruction Based Methods
fromLimits on Super-Resolution and How to
Break ThemSimon Baker and Takeo Kanade
![Page 22: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/22.jpg)
Large Magnification Factor is Problematic
• Large magnification factor causes:– Overly smooth HR image– Fine details are not recovered
• An explanation is needed.
![Page 23: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/23.jpg)
Evil Example
• Suppose we want to increase the resolution by exactly M=2.
• Lets look on a checkboard like scene, where each pixel is either white or black.
HR HR HR
LR
![Page 24: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/24.jpg)
Information is Inherently Missing
• The resulting image would be grey independently from the offset of the LR image!
• Conclusion: some information is inherently missing on our LR images!
![Page 25: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/25.jpg)
When M is not an Integer
HR HR HR
LR
![Page 26: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/26.jpg)
Limits of Super-Resolution
• Size of LR images: N pixels.• Size of HR image: NM 2 pixels.• Each HR pixel can be added noise of amplitude smaller
than M 2 which wont change the LR image!• Volume of possible HR solutions: O(M 2N) 1
• It can be shown that under practical considerations the effective magnification factor (M) is bounded by 1.6, no matter how many LR images are taken2.
1 Limits on Super-Resolution and How to Break Them, Simon Baker and Takeo Kanade
2 Fundamental Limits of Reconstruction-Based Superresolution Algorithms under Local Translation, Zhouchen Lin, and Heung-Yeung Shum
![Page 27: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/27.jpg)
Break
• Introduction to Super-Resolution• Reconstruction Based Super Resolution
– An Algorithm– Limits on Reconstruction Based Super Resolution
• Example Based Super Resolution– Halucination– Example Based– Single Image Super Resolution
• Summary
![Page 28: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/28.jpg)
Example Based Super Resolution
![Page 29: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/29.jpg)
Introduction to Example-Based Super Resolution
• Reconstruction constraints are not enough.
• One has to use prior knowledge of the image to break the reconstruction limits.
• The following algorithms will use priors learned from databases of example images.
![Page 30: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/30.jpg)
Recogstruction or Hallucination
fromLimits on Super-Resolution and How to Break Them
Simon Baker and Takeo Kanade
![Page 31: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/31.jpg)
General Idea
• Find a HR image Su that satisfies two kinds of constraints:– Reconstruction constraints: When projected to
the LR dimensions, the image is similar to the observed input images.
– Recognition constraints: The pixels of Su should resemble pixels from images in the DB that where found to have similar features to the observed LR images’ features.
![Page 32: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/32.jpg)
MAP formulation
• To solve the problem, given the LR images, we need to find the HR image that maximizes
- Su: the HR image
- Lo: the LR images
Reconstruction Constraints
Recognition Constraints
![Page 33: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/33.jpg)
Reconstruction Constraints
• The probability of the LR images given the HR image can be computed from the distance between the deresoluted HR image and the LR images.
– : the noise variance– PSF: Point Spread Function– : The pixel in Lo that corresponds to pixel z in Su.– m: a LR pixel index zri
![Page 34: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/34.jpg)
![Page 35: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/35.jpg)
Recognition: LR features
• We use “Parent Structures” to describe LR features.
![Page 36: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/36.jpg)
Recognition: Choosing the Pixels from the DB
PS = Parent StructureF = Features – like First deriviative, or Laplacian
![Page 37: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/37.jpg)
Formulation of Recognition Constraints
• Instead of estimating the probability of the HR image, Su, we estimate its probability given each pixel’s “recognition”.
H0 – Horizontal derivativeV0 – Vertical derivative. - Variance of the recognition errors.T - the training images.BI – best images for the pixels of the LR images.BP – best pixel indices in the best images for the pixels of the LR images.Ci,BP,BI – Class of all images that would have the Best corresponding Images BI, and the Best corresponding Pixels BP in the db. - The function that fits a LR pixel index to the corresponding HR pixel index.2k – the ratio between the HR image scale and the LR image scale.
![Page 38: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/38.jpg)
Maximization
• Note that the function we need to maximize is quadratic with the HR image’s pixels.
• Do gradient descent.
![Page 39: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/39.jpg)
Algorithm Summary
• Preliminary work:– Take a training set of images.– Build a DB that matches parent structures to HR
pixels.
• Compute the reconstruction constraints.• For each LR image:
– For each HR pixel index:• Search for the corresponding parent structure in the DB.
• Find the HR image that fits best both the reconstruction constraints and the HR pixels extracted from the database.
![Page 40: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/40.jpg)
Comparison
![Page 41: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/41.jpg)
Comparison
![Page 42: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/42.jpg)
Best and Worst Image
![Page 43: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/43.jpg)
Noise Effect
![Page 44: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/44.jpg)
Image Size
![Page 45: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/45.jpg)
Hallucination
![Page 46: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/46.jpg)
Hallucination
![Page 47: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/47.jpg)
Results on Text
![Page 48: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/48.jpg)
Example Based Super Resolution
William T. Freeman, Thouis R. Jones and Egon C. Pasztor
![Page 49: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/49.jpg)
Algorithm Overview
• Construct a DB of matching LR-HR patches
• Algorithmically find the most coherent patch assignment to generate a good image
![Page 50: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/50.jpg)
Constructing the DB
• Given a DB of images• Make a table from LR patches to HR patches. • Each image in the DB is treated as follows:
– Take each 7x7 patch from the image and deresolute into a 5x5 patch
– Normalize the 5x5 patches to have the same mean and relative contrast.
– Arrange the DB by the low frequencies of the LR patches
![Page 51: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/51.jpg)
Local Patch Matching
• Match a LR patch to a HR patch from the DB, using low frequencies.
• Get an estimation to the unknown (high) frequencies, based on the match.
• Remaining problem: match between neighboring overlapping patches.
![Page 52: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/52.jpg)
Global Patch Matching
• Run over patches from left to right and from top to bottom
• Match each patch its nearest neighbor in the DB using the predetermined patches as additional constraints.
![Page 53: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/53.jpg)
Algorithm
![Page 54: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/54.jpg)
Wasach
![Page 55: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/55.jpg)
Wasach
![Page 56: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/56.jpg)
Wasach
Cubic-spline Super-resolution True high-resolution image
![Page 57: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/57.jpg)
Wasach
![Page 58: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/58.jpg)
Complete Failure
![Page 59: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/59.jpg)
Priors are definitely used!
![Page 60: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/60.jpg)
Super Resolution From a Single Image
Daniel Glaser, Shai Bagon and Michal Irani
![Page 61: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/61.jpg)
Patch Redundancy in a Single Image
![Page 62: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/62.jpg)
Employing in-scale Patch Redundancy
![Page 63: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/63.jpg)
Employing Cross-scale Patch Redundancy
• Build a cascade of decreasing resolution images from the LR image.
• For each patch in the LR image, search for its Nearest Neighbour in the even lower resolution image.
• Take the found neighbour’s parent in the original LR image and copy it to be the HR image.
![Page 64: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/64.jpg)
Combining the Methods
![Page 65: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/65.jpg)
Wasach
Bicubic interpolation Unified single-image SR (x3) Ground truth image
![Page 66: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/66.jpg)
Wasach
Unified single-image SR (x3)Bicubic interpolation
![Page 67: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/67.jpg)
Wasach
![Page 68: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/68.jpg)
Wasach
![Page 69: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/69.jpg)
Wasach
![Page 70: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/70.jpg)
Wasach
![Page 71: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/71.jpg)
Wasach
![Page 72: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/72.jpg)
Wasach
![Page 73: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/73.jpg)
Wasach
![Page 74: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/74.jpg)
Wasach
![Page 75: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/75.jpg)
Wasach
![Page 76: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/76.jpg)
Summary
• We have presented two basic approaches for super resolution:– Reconstruction-based – which simply tries to reverse the
imaging process– Example-based – which uses example images to reconstruct the
original image.• We have shown that there are limits to reconstruction
based methods, which are independent of the number of LR images we use.
• We have presented an algorithm that combines both approaches to achieve SR from a single image.
![Page 77: Super-Resolution Barak Zackay Yaron Kassner. Outline Introduction to Super-Resolution Reconstruction Based Super Resolution –An Algorithm –Limits on Reconstruction.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649dbd5503460f94ab0b22/html5/thumbnails/77.jpg)
Questions?