Streaming Multigrid for Gradient-Domain Operations on Large Images
description
Transcript of Streaming Multigrid for Gradient-Domain Operations on Large Images
![Page 1: Streaming Multigrid for Gradient-Domain Operations on Large Images](https://reader036.fdocuments.us/reader036/viewer/2022081515/56816647550346895dd9bc68/html5/thumbnails/1.jpg)
Streaming Multigridfor Gradient-Domain Operations
on Large Images
Johns Hopkins University Misha Kazhdan
Microsoft ResearchHugues Hoppeand
![Page 2: Streaming Multigrid for Gradient-Domain Operations on Large Images](https://reader036.fdocuments.us/reader036/viewer/2022081515/56816647550346895dd9bc68/html5/thumbnails/2.jpg)
Stitching Image PanoramasDifferent exposures Seams in the panorama
Courtesy of Uyttendaele
Gradient Domain Fix:Set seam-crossing gradients to zero.
![Page 3: Streaming Multigrid for Gradient-Domain Operations on Large Images](https://reader036.fdocuments.us/reader036/viewer/2022081515/56816647550346895dd9bc68/html5/thumbnails/3.jpg)
High Dynamic Range CompressionGradient Domain Fix:Amplify small gradients and dampen large ones.
Medium ExposureShort ExposureCourtesy of Industrial Light & Magic
Long Exposure
No single image exposes all the detail
Tone-Mapped Image
![Page 4: Streaming Multigrid for Gradient-Domain Operations on Large Images](https://reader036.fdocuments.us/reader036/viewer/2022081515/56816647550346895dd9bc68/html5/thumbnails/4.jpg)
Solver
Image0
Image1
Imagen
ImageGradientDomain
Operator
Grad0
Gradn
Grad1 Grad
Gradient-Domain Image ProcessingMany image processing operations are easier to implement in the gradient-domain.
Solver
Image0
Image1
Imagen
ImageGradientDomain
Operator
Grad0
Gradn
Grad1 Grad
![Page 5: Streaming Multigrid for Gradient-Domain Operations on Large Images](https://reader036.fdocuments.us/reader036/viewer/2022081515/56816647550346895dd9bc68/html5/thumbnails/5.jpg)
Solver
Image0
Image1
Imagen
ImageGradientDomain
Operator
Grad0
Gradn
Grad1 Grad
Gradient-Domain Image ProcessingMany image processing operations are easier to implement in the gradient-domain.
Removing Lighting EffectsRange CompressionImage CompositingImage StitchingImage Authoring
[Horn ’74, Weiss ’01, Finlayson ’02, Agrawal ’05][Fattal ’02, Weyrich ’07][Perez ’03, Agarwala ’04, Jia ’06][Levin ’04, Agarwala ’07][McCann ’08]
Solver
Image0
Image1
Imagen
ImageGradientDomain
Operator
Grad0
Gradn
Grad1 Grad
![Page 6: Streaming Multigrid for Gradient-Domain Operations on Large Images](https://reader036.fdocuments.us/reader036/viewer/2022081515/56816647550346895dd9bc68/html5/thumbnails/6.jpg)
Solver
Image0
Image1
Imagen
ImageGradientDomain
Operator
Grad0
Gradn
Grad1 Grad
Gradient-Domain Image ProcessingMany image processing operations are easier to implement in the gradient-domain.
Image0
Image1
Imagen
ImageGradientDomain
Operator
Grad0
Gradn
Grad1 SolverGrad
Removing Lighting EffectsRange CompressionImage CompositingImage StitchingImage Authoring
[Horn ’74, Weiss ’01, Finlayson ’02, Agrawal ’05][Fattal ’02, Weyrich ’07][Perez ’03, Agarwala ’04, Jia ’06][Levin ’04, Agarwala ’07][McCann ’08]
![Page 7: Streaming Multigrid for Gradient-Domain Operations on Large Images](https://reader036.fdocuments.us/reader036/viewer/2022081515/56816647550346895dd9bc68/html5/thumbnails/7.jpg)
Outline
• Introduction• Solving for the Image– The Poisson Equation– Solving the Linear System
• Choosing the System• Solving for Big Images• Results and Discussion
![Page 8: Streaming Multigrid for Gradient-Domain Operations on Large Images](https://reader036.fdocuments.us/reader036/viewer/2022081515/56816647550346895dd9bc68/html5/thumbnails/8.jpg)
The Poisson EquationGradient-domain processing requires fitting an image to local-difference constraints:
1
gu
fu
g uf=div(g)
)(div)(div gu
1
averaged^
![Page 9: Streaming Multigrid for Gradient-Domain Operations on Large Images](https://reader036.fdocuments.us/reader036/viewer/2022081515/56816647550346895dd9bc68/html5/thumbnails/9.jpg)
Gauss-Seidel SolversGradient-domain processing requires fitting an image to local-difference constraints.
Iterative Algorithm:
For each pixel (i,j)1. Assume the rest of the pixel values are correct2. Update u[i][j] to satisfy the Laplacian:
u[i][j] (Average of Neighbors) + f[i][j]
gu
fu
averaged^
![Page 10: Streaming Multigrid for Gradient-Domain Operations on Large Images](https://reader036.fdocuments.us/reader036/viewer/2022081515/56816647550346895dd9bc68/html5/thumbnails/10.jpg)
Gauss-Seidel Solvers
This quickly solves for the high frequencies, but the low frequencies converge slowly.
Desired Solution
10 Iterations 100 Iterations10 IterationResidual
100 IterationResidual
2 Iterations2 IterationResidual
![Page 11: Streaming Multigrid for Gradient-Domain Operations on Large Images](https://reader036.fdocuments.us/reader036/viewer/2022081515/56816647550346895dd9bc68/html5/thumbnails/11.jpg)
Multigrid SolversV-Cycle Algorithm:The process is recursed for the low-res problem.
DesiredBest-Guess
Iterated
Residual
1. Perform a few updates on the high-res image2. Compute the residual constraints
Desired
3. Down-sample, solve, and up-sample
SolvedUp-Sampled
Best-Guess
4. Sum the low-res and high-res results
Iterated
5. Refine the high-res image
+
-
![Page 12: Streaming Multigrid for Gradient-Domain Operations on Large Images](https://reader036.fdocuments.us/reader036/viewer/2022081515/56816647550346895dd9bc68/html5/thumbnails/12.jpg)
Outline
• Introduction• Solving for the Image• Choosing the System• Solving for Big Images• Results and Discussion
![Page 13: Streaming Multigrid for Gradient-Domain Operations on Large Images](https://reader036.fdocuments.us/reader036/viewer/2022081515/56816647550346895dd9bc68/html5/thumbnails/13.jpg)
Choosing the SystemThe choice depends on the interpretation of pixels as elements of a continuous function.
Pixel values 0th-order interpretation1st-order interpretation2nd-order interpretation3rd-order interpretation
Elements
![Page 14: Streaming Multigrid for Gradient-Domain Operations on Large Images](https://reader036.fdocuments.us/reader036/viewer/2022081515/56816647550346895dd9bc68/html5/thumbnails/14.jpg)
Choosing the SystemImplementing Up-/Down-Sampling:The nesting of the elements determines the transition between the levels of the solver.
0th-order elements 1st-order elements 2nd-order elements 3rd-order elements1 1
1/2
1
1/21/4
3/4 3/4
1/4 1/81/2
3/41/2
1/8
![Page 15: Streaming Multigrid for Gradient-Domain Operations on Large Images](https://reader036.fdocuments.us/reader036/viewer/2022081515/56816647550346895dd9bc68/html5/thumbnails/15.jpg)
Choosing the SystemDefining the Laplacian:The neighborhood of the Laplacian is defined by the element overlap.
Undefined
0th-order elements 1st-order elements 2nd-order elements 3rd-order elements
1-Ring Average 2-Ring Average 3-Ring Average
![Page 16: Streaming Multigrid for Gradient-Domain Operations on Large Images](https://reader036.fdocuments.us/reader036/viewer/2022081515/56816647550346895dd9bc68/html5/thumbnails/16.jpg)
Second-Order ElementsWhich order elementconverges fastest to anaccurate solution?
Original Image Image Gradients
1st-order error
2nd-order error
3rd-order error0 5
1st-Order2nd-Order3rd Order
256
1
1/256
Average Error
GSUpdates
![Page 17: Streaming Multigrid for Gradient-Domain Operations on Large Images](https://reader036.fdocuments.us/reader036/viewer/2022081515/56816647550346895dd9bc68/html5/thumbnails/17.jpg)
Second-Order Elements
Spectral Analysis of Convergence:The eigenvalues of the multigrid operator determine how efficiently the residual is dampened.
0 320.0
0.2
0.4
Two-Grid Spectra (k=5)
Eigenvalue Index
Eigenvalue
1st-order2nd-order3rd-order
G.S Update
G.S Update
G.S Update
Two-Grid Operator
Down-Sample
Up-Sample
![Page 18: Streaming Multigrid for Gradient-Domain Operations on Large Images](https://reader036.fdocuments.us/reader036/viewer/2022081515/56816647550346895dd9bc68/html5/thumbnails/18.jpg)
Outline
• Introduction• Solving for the Image• Choosing the System• Solving for Big Images– Naïve Out-of-Core Multigrid– Temporal Blocking– Interleaved Multilevel Streaming
• Results and Discussion
![Page 19: Streaming Multigrid for Gradient-Domain Operations on Large Images](https://reader036.fdocuments.us/reader036/viewer/2022081515/56816647550346895dd9bc68/html5/thumbnails/19.jpg)
Naïve Out-of-Core MultigridComponents of the solver:
1. Gauss-Seidel updates2. Down-sampling3. Up-sampling
The multigrid solvercan be streamed byadvancing a fixed-size window.
Local data access
On Disk
In Memory
![Page 20: Streaming Multigrid for Gradient-Domain Operations on Large Images](https://reader036.fdocuments.us/reader036/viewer/2022081515/56816647550346895dd9bc68/html5/thumbnails/20.jpg)
Naïve Out-of-Core MultigridThe large linear system can be solved by streaming each step of the V-cycle.
Low-Res Solve
Down-Sample
Down-Sample
Up-Sample
Up-Sample
G.S.Update
Solution
G.S. Update
G.S. Update
k k
kk
G.S. Update
Constraints
Disk Streaming
k Gauss-Seidel Updates
2(k+2) Streaming Passes
Residual
Residual
![Page 21: Streaming Multigrid for Gradient-Domain Operations on Large Images](https://reader036.fdocuments.us/reader036/viewer/2022081515/56816647550346895dd9bc68/html5/thumbnails/21.jpg)
Temporal BlockingA pixel can be updated for the kth time only if all other pixels have been updated k-1 times.neighbor
![Page 22: Streaming Multigrid for Gradient-Domain Operations on Large Images](https://reader036.fdocuments.us/reader036/viewer/2022081515/56816647550346895dd9bc68/html5/thumbnails/22.jpg)
Temporal BlockingA pixel can be updated for the kth time only if all other pixels have been updated k-1 times.
000
0 1
0
0 1
0
0 1
1 2
0
0 1
1 2
0
0 1
1 2
2
neighbor
Win
dow
k=2 Gauss-Seidel Updates
2nd-order elements
![Page 23: Streaming Multigrid for Gradient-Domain Operations on Large Images](https://reader036.fdocuments.us/reader036/viewer/2022081515/56816647550346895dd9bc68/html5/thumbnails/23.jpg)
Temporal BlockingA pixel can be updated for the kth time only if all other pixels have been updated k-1 times.
SolutionConstraints
neighbor
Low-Res Solve
Down-Sample
Down-Sample
Up-Sample
Up-Sample
G.S.Update
G.S. Update
G.S. Update
G.S. Update
Residual
Residual
Disk Streaming
![Page 24: Streaming Multigrid for Gradient-Domain Operations on Large Images](https://reader036.fdocuments.us/reader036/viewer/2022081515/56816647550346895dd9bc68/html5/thumbnails/24.jpg)
Interleaved Multilevel StreamingProcessing at one level is buffered for the next.
piped into
![Page 25: Streaming Multigrid for Gradient-Domain Operations on Large Images](https://reader036.fdocuments.us/reader036/viewer/2022081515/56816647550346895dd9bc68/html5/thumbnails/25.jpg)
Processing at one level is buffered for the next.
1/4
3/4 3/4
1/4
¾¾¼
¼
Interleaved Multilevel Streaming
000
0 1
0
0 1
0
0 1
1 2
0
0 1
1 2
0
0 1
1 2
0
2
0 1
1 2
0
0 1
1 2
2
0
0 1
1 2
0
2
0 1
1 2
00
0 11 2
0 1
1 2
20000 1
¾¾¼
¼¾¾¼
¼
Resid
ual
High-Resolution Low-Resolution
2nd-order elements
piped into
![Page 26: Streaming Multigrid for Gradient-Domain Operations on Large Images](https://reader036.fdocuments.us/reader036/viewer/2022081515/56816647550346895dd9bc68/html5/thumbnails/26.jpg)
Interleaved Multilevel StreamingProcessing at one level is buffered for the next.
SolutionConstraints
Low-Res Solve
Down-Sample
Down-Sample
Up-Sample
Up-Sample
G.S.Update
G.S. Update
G.S. Update
G.S. Update
Residual
Residual
Streaming Pass 1 Streaming Pass 2
Globality of the Poisson equation
Two streaming passes is optimal
Disk StreamingMemory Streaming
Disk Streaming
piped into
![Page 27: Streaming Multigrid for Gradient-Domain Operations on Large Images](https://reader036.fdocuments.us/reader036/viewer/2022081515/56816647550346895dd9bc68/html5/thumbnails/27.jpg)
For gradient-domain operators that are local, processing only takes two streaming passes.
+1 streaming pass for each additional V-cycle.
Streaming through the Gradient-Domain
Pass 3uu
Update
Updatef
Solve
Update…
u0
u1
g0
g1 g
un gn
Op…
Update
…
Memory StreamingDisk Streaming
Pass 1 Pass 2
Solve
Update Update
UpdateUpdate
![Page 28: Streaming Multigrid for Gradient-Domain Operations on Large Images](https://reader036.fdocuments.us/reader036/viewer/2022081515/56816647550346895dd9bc68/html5/thumbnails/28.jpg)
Outline
• Introduction• Solving for the Image• Choosing the System• Solving for Big Images• Results and Discussion
![Page 29: Streaming Multigrid for Gradient-Domain Operations on Large Images](https://reader036.fdocuments.us/reader036/viewer/2022081515/56816647550346895dd9bc68/html5/thumbnails/29.jpg)
Stitching Big Images
Courtesy of Agarwala
Agarwala ’07:A high-res solution is only required near the seams, so solve over an adapted quadtree.
![Page 30: Streaming Multigrid for Gradient-Domain Operations on Large Images](https://reader036.fdocuments.us/reader036/viewer/2022081515/56816647550346895dd9bc68/html5/thumbnails/30.jpg)
Stitching Results
Courtesy of Agarwala and Curless
Beynac: 6,0461,920 Rainier: 10,9602,096
Edinburgh: 16,9502,956
Redrock: 19,5884,457
Dataset Pixels ImagesError [0,256) Memory (MB) Time (seconds)
Quadtree Streaming Quadtree Streaming Quadtree Streaming
Beynac 12x106 3 0.01 0.01 16 190 8 17
Rainier 23x106 5 0.02 0.01 17 110 14 33
Edinburgh 50x106 25 0.01 123 203 122 79
Redrock 87x106 9 0.01 112 133 118 118
![Page 31: Streaming Multigrid for Gradient-Domain Operations on Large Images](https://reader036.fdocuments.us/reader036/viewer/2022081515/56816647550346895dd9bc68/html5/thumbnails/31.jpg)
Tone-Mapping Results
Bring out subtle details by amplifying small gradients and dampening large ones.
![Page 32: Streaming Multigrid for Gradient-Domain Operations on Large Images](https://reader036.fdocuments.us/reader036/viewer/2022081515/56816647550346895dd9bc68/html5/thumbnails/32.jpg)
Large Image ResultsSt James:• Stitched from 643 photographs• Contains 3.3 billion (88,309 x 37,842) pixels
Tone-MappedMemory: 224 MBTime: 45:10Max Error [0,256): 0.5 Courtesy of Uyttendaele
StitchedMemory: 408 MBTime: 1:27:50Max Error [0,256): 0.15
![Page 33: Streaming Multigrid for Gradient-Domain Operations on Large Images](https://reader036.fdocuments.us/reader036/viewer/2022081515/56816647550346895dd9bc68/html5/thumbnails/33.jpg)
Large Image Results
Courtesy of Uyttendaele
St James:• Peak Memory: 408 MB• Solver Time: 1:27:50
Total disk I/O:• Down-Sampling
– Read in the gradient constraints: 40 GB– Write out solution and Laplacian: 53 GB
• Up-Sampling– Read in solution and Laplacian: 53 GB– Write out the image: 20 GB
53GB in-core1:19:02 I/O Time
![Page 34: Streaming Multigrid for Gradient-Domain Operations on Large Images](https://reader036.fdocuments.us/reader036/viewer/2022081515/56816647550346895dd9bc68/html5/thumbnails/34.jpg)
Summary
Streaming multigrid for big image processing:– 2nd order elements– Temporal blocking– Interleaved streaming
Issues addressed in the paper:– Integration with finite-differences– Setting the average color–Non power-of-two images
Accurate solution in two streaming passes
![Page 35: Streaming Multigrid for Gradient-Domain Operations on Large Images](https://reader036.fdocuments.us/reader036/viewer/2022081515/56816647550346895dd9bc68/html5/thumbnails/35.jpg)
Issues for Future Work
1. Non-trivial boundaries and constraints?e.g. cut-and-paste, matting
2. Adaptive weighting?e.g. local tone adjustment
3. Extensions to higher dimensions?for incompressible fluid simulation
4. Less I/O in the streaming passes?on-the-fly compression
5. Really big images?parallelized and distributed solvers
![Page 36: Streaming Multigrid for Gradient-Domain Operations on Large Images](https://reader036.fdocuments.us/reader036/viewer/2022081515/56816647550346895dd9bc68/html5/thumbnails/36.jpg)
AcknowledgmentsDatasets:
Aseem AgarwalaBrian CurlessMatt Uyttendaele
Discussions and Ideas:Ketan Dalal Rick SzeliskiBill Bolosky Ann WolvertonCAVGRAPH Reviewers SIGGRAPH Reviewers
![Page 37: Streaming Multigrid for Gradient-Domain Operations on Large Images](https://reader036.fdocuments.us/reader036/viewer/2022081515/56816647550346895dd9bc68/html5/thumbnails/37.jpg)
Code & Images @ http://www.cs.jhu.edu/~misha/Code/SMG
Thank You!
![Page 38: Streaming Multigrid for Gradient-Domain Operations on Large Images](https://reader036.fdocuments.us/reader036/viewer/2022081515/56816647550346895dd9bc68/html5/thumbnails/38.jpg)
![Page 39: Streaming Multigrid for Gradient-Domain Operations on Large Images](https://reader036.fdocuments.us/reader036/viewer/2022081515/56816647550346895dd9bc68/html5/thumbnails/39.jpg)
General Temporal BlockingTo perform k updates:
Updating the 3rd row from front,The row two behind that,The row two behind that,
000
0 1
0
0 1
0
0 1
1 2
0
0 1
1 2
0
0 1
1 2
2 33
k=3 Gauss-Seidel Updates
00
0 1
1 2
2 3
0 1
1 2
2 3
k
Win
dow
…
![Page 40: Streaming Multigrid for Gradient-Domain Operations on Large Images](https://reader036.fdocuments.us/reader036/viewer/2022081515/56816647550346895dd9bc68/html5/thumbnails/40.jpg)
General Temporal BlockingTo perform k updates:
Updating the 3rd row from front,The row two behind that,The row two behind that,…
k=3 Gauss-Seidel Updates
k
with d-th order elements^
3
00
0 1
1 2
2 3
0 1
1 2
2 3
Win
dow
(d+1)-stdd
![Page 41: Streaming Multigrid for Gradient-Domain Operations on Large Images](https://reader036.fdocuments.us/reader036/viewer/2022081515/56816647550346895dd9bc68/html5/thumbnails/41.jpg)
Finite Differences to Finite Elements
The derivative of a d-th order element is the difference of elements of order d-1.
Finite Differences
1st-order Element
0th-order Elements
Coefficients of Derivative ElementsDifferentiation
![Page 42: Streaming Multigrid for Gradient-Domain Operations on Large Images](https://reader036.fdocuments.us/reader036/viewer/2022081515/56816647550346895dd9bc68/html5/thumbnails/42.jpg)
Second-Order ElementsWhich order elementconverges fastest to anaccurate solution?
Original Image Image Gradients
1st-order error
2nd-order error
3rd-order error0 5 10 15 20
1st-Order2nd-Order3rd-Order
GS Updates
256
1
1/256
Average Error256
Average Error
0 1000
1st-Order2nd-Order3rd-Order
Operations
256Average Error256
1
1/256
Average Error
1/2561/256