High resolution holographic image synthesis for future ... · Future of Personal Computing 2. 3...
Transcript of High resolution holographic image synthesis for future ... · Future of Personal Computing 2. 3...
![Page 1: High resolution holographic image synthesis for future ... · Future of Personal Computing 2. 3 Wide field of view High resolution Accommodation support Eyeglasses-Style Near-Eye](https://reader033.fdocuments.us/reader033/viewer/2022050516/5fa06b0613a11e265c1bdb1b/html5/thumbnails/1.jpg)
High resolution holographic image synthesis for future display eyeglasses
Praneeth Chakravarthula
UNC Chapel Hill
![Page 2: High resolution holographic image synthesis for future ... · Future of Personal Computing 2. 3 Wide field of view High resolution Accommodation support Eyeglasses-Style Near-Eye](https://reader033.fdocuments.us/reader033/viewer/2022050516/5fa06b0613a11e265c1bdb1b/html5/thumbnails/2.jpg)
Future of Personal Computing
2
![Page 3: High resolution holographic image synthesis for future ... · Future of Personal Computing 2. 3 Wide field of view High resolution Accommodation support Eyeglasses-Style Near-Eye](https://reader033.fdocuments.us/reader033/viewer/2022050516/5fa06b0613a11e265c1bdb1b/html5/thumbnails/3.jpg)
3
Wide field of view
High resolution
Accommodation support
Eyeglasses-Style Near-Eye Display Optics
![Page 4: High resolution holographic image synthesis for future ... · Future of Personal Computing 2. 3 Wide field of view High resolution Accommodation support Eyeglasses-Style Near-Eye](https://reader033.fdocuments.us/reader033/viewer/2022050516/5fa06b0613a11e265c1bdb1b/html5/thumbnails/4.jpg)
Eyeglasses-Style Near-Eye Display Optics
4
Wide field of view
Moderate resolution
Accommodation support
Holography is the only demonstrated technology for getting everything
Maimone et al. 2017
![Page 5: High resolution holographic image synthesis for future ... · Future of Personal Computing 2. 3 Wide field of view High resolution Accommodation support Eyeglasses-Style Near-Eye](https://reader033.fdocuments.us/reader033/viewer/2022050516/5fa06b0613a11e265c1bdb1b/html5/thumbnails/5.jpg)
Basics of Digital Holography
5
![Page 6: High resolution holographic image synthesis for future ... · Future of Personal Computing 2. 3 Wide field of view High resolution Accommodation support Eyeglasses-Style Near-Eye](https://reader033.fdocuments.us/reader033/viewer/2022050516/5fa06b0613a11e265c1bdb1b/html5/thumbnails/6.jpg)
Principle of Holography
6
![Page 7: High resolution holographic image synthesis for future ... · Future of Personal Computing 2. 3 Wide field of view High resolution Accommodation support Eyeglasses-Style Near-Eye](https://reader033.fdocuments.us/reader033/viewer/2022050516/5fa06b0613a11e265c1bdb1b/html5/thumbnails/7.jpg)
Principle of Holography
7
![Page 8: High resolution holographic image synthesis for future ... · Future of Personal Computing 2. 3 Wide field of view High resolution Accommodation support Eyeglasses-Style Near-Eye](https://reader033.fdocuments.us/reader033/viewer/2022050516/5fa06b0613a11e265c1bdb1b/html5/thumbnails/8.jpg)
Principle of Holography
8
Step 1: Recording
![Page 9: High resolution holographic image synthesis for future ... · Future of Personal Computing 2. 3 Wide field of view High resolution Accommodation support Eyeglasses-Style Near-Eye](https://reader033.fdocuments.us/reader033/viewer/2022050516/5fa06b0613a11e265c1bdb1b/html5/thumbnails/9.jpg)
Principle of Holography
9
Step 1: Recording
![Page 10: High resolution holographic image synthesis for future ... · Future of Personal Computing 2. 3 Wide field of view High resolution Accommodation support Eyeglasses-Style Near-Eye](https://reader033.fdocuments.us/reader033/viewer/2022050516/5fa06b0613a11e265c1bdb1b/html5/thumbnails/10.jpg)
Principle of Holography
10
Step 1: Recording Step 2: Playback
![Page 11: High resolution holographic image synthesis for future ... · Future of Personal Computing 2. 3 Wide field of view High resolution Accommodation support Eyeglasses-Style Near-Eye](https://reader033.fdocuments.us/reader033/viewer/2022050516/5fa06b0613a11e265c1bdb1b/html5/thumbnails/11.jpg)
Principle of Holography
11
Step 1: Recording
![Page 12: High resolution holographic image synthesis for future ... · Future of Personal Computing 2. 3 Wide field of view High resolution Accommodation support Eyeglasses-Style Near-Eye](https://reader033.fdocuments.us/reader033/viewer/2022050516/5fa06b0613a11e265c1bdb1b/html5/thumbnails/12.jpg)
Principle of Holography
12Step 2: Playback
Step 1: Recording
![Page 13: High resolution holographic image synthesis for future ... · Future of Personal Computing 2. 3 Wide field of view High resolution Accommodation support Eyeglasses-Style Near-Eye](https://reader033.fdocuments.us/reader033/viewer/2022050516/5fa06b0613a11e265c1bdb1b/html5/thumbnails/13.jpg)
13
Incident light modulated by Hologram H
Holographic Image Formation
![Page 14: High resolution holographic image synthesis for future ... · Future of Personal Computing 2. 3 Wide field of view High resolution Accommodation support Eyeglasses-Style Near-Eye](https://reader033.fdocuments.us/reader033/viewer/2022050516/5fa06b0613a11e265c1bdb1b/html5/thumbnails/14.jpg)
14
Propagates to result in the final field z
and final image |z|2
Holographic Image Formation
![Page 15: High resolution holographic image synthesis for future ... · Future of Personal Computing 2. 3 Wide field of view High resolution Accommodation support Eyeglasses-Style Near-Eye](https://reader033.fdocuments.us/reader033/viewer/2022050516/5fa06b0613a11e265c1bdb1b/html5/thumbnails/15.jpg)
Phase Hologram Reconstructed Image
Reference
Heuristic Hologram Phase Retrieval
Reconstruction
![Page 16: High resolution holographic image synthesis for future ... · Future of Personal Computing 2. 3 Wide field of view High resolution Accommodation support Eyeglasses-Style Near-Eye](https://reader033.fdocuments.us/reader033/viewer/2022050516/5fa06b0613a11e265c1bdb1b/html5/thumbnails/16.jpg)
Double Phase Encoding Hologram
Phase Hologram Reconstructed Image
Reference Reconstruction
![Page 17: High resolution holographic image synthesis for future ... · Future of Personal Computing 2. 3 Wide field of view High resolution Accommodation support Eyeglasses-Style Near-Eye](https://reader033.fdocuments.us/reader033/viewer/2022050516/5fa06b0613a11e265c1bdb1b/html5/thumbnails/17.jpg)
Wirtinger Holography
Phase Hologram Reconstructed Image
Reference Reconstruction
![Page 18: High resolution holographic image synthesis for future ... · Future of Personal Computing 2. 3 Wide field of view High resolution Accommodation support Eyeglasses-Style Near-Eye](https://reader033.fdocuments.us/reader033/viewer/2022050516/5fa06b0613a11e265c1bdb1b/html5/thumbnails/18.jpg)
Wirtinger Holography
18
![Page 19: High resolution holographic image synthesis for future ... · Future of Personal Computing 2. 3 Wide field of view High resolution Accommodation support Eyeglasses-Style Near-Eye](https://reader033.fdocuments.us/reader033/viewer/2022050516/5fa06b0613a11e265c1bdb1b/html5/thumbnails/19.jpg)
19
Differentiable forward model
Compute complex Wirtinger gradients
Optimize for phase holograms
Wirtinger Holography Overview
![Page 20: High resolution holographic image synthesis for future ... · Future of Personal Computing 2. 3 Wide field of view High resolution Accommodation support Eyeglasses-Style Near-Eye](https://reader033.fdocuments.us/reader033/viewer/2022050516/5fa06b0613a11e265c1bdb1b/html5/thumbnails/20.jpg)
20
Differentiable forward model
Step 1: For a penalty function f, compute the error between the holographic reconstruction and the target image
Compute complex Wirtingergradients
Optimize for phase holograms
![Page 21: High resolution holographic image synthesis for future ... · Future of Personal Computing 2. 3 Wide field of view High resolution Accommodation support Eyeglasses-Style Near-Eye](https://reader033.fdocuments.us/reader033/viewer/2022050516/5fa06b0613a11e265c1bdb1b/html5/thumbnails/21.jpg)
21
Differentiable forward model
Compute complex Wirtingergradients
Optimize for phase holograms
Step 1: For a penalty function f, compute the error between the holographic reconstruction and he target image
![Page 22: High resolution holographic image synthesis for future ... · Future of Personal Computing 2. 3 Wide field of view High resolution Accommodation support Eyeglasses-Style Near-Eye](https://reader033.fdocuments.us/reader033/viewer/2022050516/5fa06b0613a11e265c1bdb1b/html5/thumbnails/22.jpg)
22
Differentiable forward model
Step 1: For a penalty function f, compute the error between the holographic reconstruction and the target image
Compute complex Wirtingergradients
Optimize for phase holograms
![Page 23: High resolution holographic image synthesis for future ... · Future of Personal Computing 2. 3 Wide field of view High resolution Accommodation support Eyeglasses-Style Near-Eye](https://reader033.fdocuments.us/reader033/viewer/2022050516/5fa06b0613a11e265c1bdb1b/html5/thumbnails/23.jpg)
23
Differentiable forward model
Step 1: For a penalty function f, compute the error between the holographic reconstruction and the target image
Compute complex Wirtingergradients
Optimize for phase holograms
![Page 24: High resolution holographic image synthesis for future ... · Future of Personal Computing 2. 3 Wide field of view High resolution Accommodation support Eyeglasses-Style Near-Eye](https://reader033.fdocuments.us/reader033/viewer/2022050516/5fa06b0613a11e265c1bdb1b/html5/thumbnails/24.jpg)
24
Differentiable forward model
Step 1: For a penalty function f, compute the error between the holographic reconstruction and the target image
Compute complex Wirtingergradients
Optimize for phase holograms
![Page 25: High resolution holographic image synthesis for future ... · Future of Personal Computing 2. 3 Wide field of view High resolution Accommodation support Eyeglasses-Style Near-Eye](https://reader033.fdocuments.us/reader033/viewer/2022050516/5fa06b0613a11e265c1bdb1b/html5/thumbnails/25.jpg)
25
Differentiable forward model
Step 1: For a penalty function f, compute the error between the holographic reconstruction and the target image
Compute complex Wirtingergradients
Optimize for phase holograms
![Page 26: High resolution holographic image synthesis for future ... · Future of Personal Computing 2. 3 Wide field of view High resolution Accommodation support Eyeglasses-Style Near-Eye](https://reader033.fdocuments.us/reader033/viewer/2022050516/5fa06b0613a11e265c1bdb1b/html5/thumbnails/26.jpg)
26
Differentiable forward model
Compute complex Wirtingergradients
Optimize for phase holograms
Step 2: Construct the cost function to minimize the error with optional regularizer to obtain the optimal phase
![Page 27: High resolution holographic image synthesis for future ... · Future of Personal Computing 2. 3 Wide field of view High resolution Accommodation support Eyeglasses-Style Near-Eye](https://reader033.fdocuments.us/reader033/viewer/2022050516/5fa06b0613a11e265c1bdb1b/html5/thumbnails/27.jpg)
27
Differentiable forward model
Compute complex Wirtingergradients
Optimize for phase holograms
Step 2: Construct the cost function to minimize the error with optional regularizer to obtain the optimal phase
![Page 28: High resolution holographic image synthesis for future ... · Future of Personal Computing 2. 3 Wide field of view High resolution Accommodation support Eyeglasses-Style Near-Eye](https://reader033.fdocuments.us/reader033/viewer/2022050516/5fa06b0613a11e265c1bdb1b/html5/thumbnails/28.jpg)
28
Differentiable forward model
Compute complex Wirtingergradients
Optimize for phase holograms
Step 2: Construct the cost function to minimize the error with optional regularizer to obtain the optimal phase
![Page 29: High resolution holographic image synthesis for future ... · Future of Personal Computing 2. 3 Wide field of view High resolution Accommodation support Eyeglasses-Style Near-Eye](https://reader033.fdocuments.us/reader033/viewer/2022050516/5fa06b0613a11e265c1bdb1b/html5/thumbnails/29.jpg)
29
Differentiable forward model
Compute complex Wirtingergradients
Optimize for phase holograms
Step 2: Construct the cost function to minimize the error with optional regularizer to obtain the optimal phase
![Page 30: High resolution holographic image synthesis for future ... · Future of Personal Computing 2. 3 Wide field of view High resolution Accommodation support Eyeglasses-Style Near-Eye](https://reader033.fdocuments.us/reader033/viewer/2022050516/5fa06b0613a11e265c1bdb1b/html5/thumbnails/30.jpg)
30
Differentiable forward model
Compute complex Wirtingergradients
Optimize for phase holograms
Step 2: Construct the cost function to minimize the error with optional regularizer to obtain the optimal phase
We can use standard optimizers if there is a gradient
![Page 31: High resolution holographic image synthesis for future ... · Future of Personal Computing 2. 3 Wide field of view High resolution Accommodation support Eyeglasses-Style Near-Eye](https://reader033.fdocuments.us/reader033/viewer/2022050516/5fa06b0613a11e265c1bdb1b/html5/thumbnails/31.jpg)
31
Differentiable forward model
Compute complex Wirtingergradients
Optimize for phase holograms
Holomorphic function: Complex function that is complex differentiable
Derivative of holomorphic real-valued function is always ZERO
Derivatives of any order are NOT DEFINED for our objective function
Real-valued function
Complex-valued argument
![Page 32: High resolution holographic image synthesis for future ... · Future of Personal Computing 2. 3 Wide field of view High resolution Accommodation support Eyeglasses-Style Near-Eye](https://reader033.fdocuments.us/reader033/viewer/2022050516/5fa06b0613a11e265c1bdb1b/html5/thumbnails/32.jpg)
Optimize for phase holograms
Differentiable forward model
Compute complex Wirtingergradients
32
Two important properties of gradient:
1) Direction of maximal rate of change
2) Is zero at stationary points
Step 3: Define approximate gradient and compute Wirtinger derivatives for each propagation model
REFER TO MY SIGGRAPH Asia 2019PAPER AND SUPPLEMENT
![Page 33: High resolution holographic image synthesis for future ... · Future of Personal Computing 2. 3 Wide field of view High resolution Accommodation support Eyeglasses-Style Near-Eye](https://reader033.fdocuments.us/reader033/viewer/2022050516/5fa06b0613a11e265c1bdb1b/html5/thumbnails/33.jpg)
33
Differentiable forward model
Compute complex Wirtingergradients
Optimize for phase holograms
Standard off-the-shelf optimization methods
1) Quasi-Newton
2) Stochastic gradient descent
Step 4: Optimize for holograms using off-the-shelf methods
![Page 34: High resolution holographic image synthesis for future ... · Future of Personal Computing 2. 3 Wide field of view High resolution Accommodation support Eyeglasses-Style Near-Eye](https://reader033.fdocuments.us/reader033/viewer/2022050516/5fa06b0613a11e265c1bdb1b/html5/thumbnails/34.jpg)
Simulation Results
34
![Page 35: High resolution holographic image synthesis for future ... · Future of Personal Computing 2. 3 Wide field of view High resolution Accommodation support Eyeglasses-Style Near-Eye](https://reader033.fdocuments.us/reader033/viewer/2022050516/5fa06b0613a11e265c1bdb1b/html5/thumbnails/35.jpg)
Reference
Reconstruction
![Page 36: High resolution holographic image synthesis for future ... · Future of Personal Computing 2. 3 Wide field of view High resolution Accommodation support Eyeglasses-Style Near-Eye](https://reader033.fdocuments.us/reader033/viewer/2022050516/5fa06b0613a11e265c1bdb1b/html5/thumbnails/36.jpg)
Reference Reconstruction
![Page 37: High resolution holographic image synthesis for future ... · Future of Personal Computing 2. 3 Wide field of view High resolution Accommodation support Eyeglasses-Style Near-Eye](https://reader033.fdocuments.us/reader033/viewer/2022050516/5fa06b0613a11e265c1bdb1b/html5/thumbnails/37.jpg)
37
Simulation Results
TargetModified GS
(Peng et al. 2017)Double phase
(Maimone et al. 2017) Our Method
![Page 38: High resolution holographic image synthesis for future ... · Future of Personal Computing 2. 3 Wide field of view High resolution Accommodation support Eyeglasses-Style Near-Eye](https://reader033.fdocuments.us/reader033/viewer/2022050516/5fa06b0613a11e265c1bdb1b/html5/thumbnails/38.jpg)
38
Simulation Results
TargetModified GS
(Peng et al. 2017)Double phase
(Maimone et al. 2017) Our Method
![Page 39: High resolution holographic image synthesis for future ... · Future of Personal Computing 2. 3 Wide field of view High resolution Accommodation support Eyeglasses-Style Near-Eye](https://reader033.fdocuments.us/reader033/viewer/2022050516/5fa06b0613a11e265c1bdb1b/html5/thumbnails/39.jpg)
![Page 40: High resolution holographic image synthesis for future ... · Future of Personal Computing 2. 3 Wide field of view High resolution Accommodation support Eyeglasses-Style Near-Eye](https://reader033.fdocuments.us/reader033/viewer/2022050516/5fa06b0613a11e265c1bdb1b/html5/thumbnails/40.jpg)
Prototype Hardware Results
![Page 41: High resolution holographic image synthesis for future ... · Future of Personal Computing 2. 3 Wide field of view High resolution Accommodation support Eyeglasses-Style Near-Eye](https://reader033.fdocuments.us/reader033/viewer/2022050516/5fa06b0613a11e265c1bdb1b/html5/thumbnails/41.jpg)
41
Prototype Hardware Results
TargetModified GS
(Peng et al. 2017)Double phase
(Maimone et al. 2017) Our Method
![Page 42: High resolution holographic image synthesis for future ... · Future of Personal Computing 2. 3 Wide field of view High resolution Accommodation support Eyeglasses-Style Near-Eye](https://reader033.fdocuments.us/reader033/viewer/2022050516/5fa06b0613a11e265c1bdb1b/html5/thumbnails/42.jpg)
42
TargetModified GS
(Peng et al. 2017)Double phase
(Maimone et al. 2017) Our Method
Prototype Hardware Results
![Page 43: High resolution holographic image synthesis for future ... · Future of Personal Computing 2. 3 Wide field of view High resolution Accommodation support Eyeglasses-Style Near-Eye](https://reader033.fdocuments.us/reader033/viewer/2022050516/5fa06b0613a11e265c1bdb1b/html5/thumbnails/43.jpg)
43
TargetModified GS
(Peng et al. 2017)Double phase
(Maimone et al. 2017) Our Method
Prototype Hardware Results
![Page 44: High resolution holographic image synthesis for future ... · Future of Personal Computing 2. 3 Wide field of view High resolution Accommodation support Eyeglasses-Style Near-Eye](https://reader033.fdocuments.us/reader033/viewer/2022050516/5fa06b0613a11e265c1bdb1b/html5/thumbnails/44.jpg)
Results Reference
Experiment
![Page 45: High resolution holographic image synthesis for future ... · Future of Personal Computing 2. 3 Wide field of view High resolution Accommodation support Eyeglasses-Style Near-Eye](https://reader033.fdocuments.us/reader033/viewer/2022050516/5fa06b0613a11e265c1bdb1b/html5/thumbnails/45.jpg)
Reference Experiment
![Page 46: High resolution holographic image synthesis for future ... · Future of Personal Computing 2. 3 Wide field of view High resolution Accommodation support Eyeglasses-Style Near-Eye](https://reader033.fdocuments.us/reader033/viewer/2022050516/5fa06b0613a11e265c1bdb1b/html5/thumbnails/46.jpg)
46
Perceptually Optimized Holography
Learned Perceptual Image Patch Similarity (LPIPS) ( Zhang et al. 2018 )
Optimize for deep learning based perceptual losses
![Page 47: High resolution holographic image synthesis for future ... · Future of Personal Computing 2. 3 Wide field of view High resolution Accommodation support Eyeglasses-Style Near-Eye](https://reader033.fdocuments.us/reader033/viewer/2022050516/5fa06b0613a11e265c1bdb1b/html5/thumbnails/47.jpg)
47
Perceptually Optimized Holography
Target LPIPS optimizedMS-SSIM optimizedL2 optimized
![Page 48: High resolution holographic image synthesis for future ... · Future of Personal Computing 2. 3 Wide field of view High resolution Accommodation support Eyeglasses-Style Near-Eye](https://reader033.fdocuments.us/reader033/viewer/2022050516/5fa06b0613a11e265c1bdb1b/html5/thumbnails/48.jpg)
48
Cascaded Superresolution Holography
![Page 49: High resolution holographic image synthesis for future ... · Future of Personal Computing 2. 3 Wide field of view High resolution Accommodation support Eyeglasses-Style Near-Eye](https://reader033.fdocuments.us/reader033/viewer/2022050516/5fa06b0613a11e265c1bdb1b/html5/thumbnails/49.jpg)
49
TargetModified GS
(Peng et al. 2017)Double phase
(Maimone et al. 2017) Our Method
Prototype Hardware Results
![Page 50: High resolution holographic image synthesis for future ... · Future of Personal Computing 2. 3 Wide field of view High resolution Accommodation support Eyeglasses-Style Near-Eye](https://reader033.fdocuments.us/reader033/viewer/2022050516/5fa06b0613a11e265c1bdb1b/html5/thumbnails/50.jpg)
Real World Deviations
![Page 51: High resolution holographic image synthesis for future ... · Future of Personal Computing 2. 3 Wide field of view High resolution Accommodation support Eyeglasses-Style Near-Eye](https://reader033.fdocuments.us/reader033/viewer/2022050516/5fa06b0613a11e265c1bdb1b/html5/thumbnails/51.jpg)
51
Ideal Wave Propagation
![Page 52: High resolution holographic image synthesis for future ... · Future of Personal Computing 2. 3 Wide field of view High resolution Accommodation support Eyeglasses-Style Near-Eye](https://reader033.fdocuments.us/reader033/viewer/2022050516/5fa06b0613a11e265c1bdb1b/html5/thumbnails/52.jpg)
52
Real World Wave Propagation
![Page 53: High resolution holographic image synthesis for future ... · Future of Personal Computing 2. 3 Wide field of view High resolution Accommodation support Eyeglasses-Style Near-Eye](https://reader033.fdocuments.us/reader033/viewer/2022050516/5fa06b0613a11e265c1bdb1b/html5/thumbnails/53.jpg)
53
Compensating real world deviations viaHardware-in-the-loop phase retrieval
Target (Chakravarthula et al. 2019)Wirtinger HolographyDouble Phase Encoding
(Maimone et al. 2017) Our Method
Upcoming at SIGGRAPH Asia 2020
![Page 54: High resolution holographic image synthesis for future ... · Future of Personal Computing 2. 3 Wide field of view High resolution Accommodation support Eyeglasses-Style Near-Eye](https://reader033.fdocuments.us/reader033/viewer/2022050516/5fa06b0613a11e265c1bdb1b/html5/thumbnails/54.jpg)
54
Target (Chakravarthula et al. 2019)Wirtinger HolographyDouble Phase Encoding
(Maimone et al. 2017) Our Method
Target (Chakravarthula et al. 2019)Wirtinger HolographyDouble Phase Encoding
(Maimone et al. 2017) Our Method
Compensating real world deviations viaHardware-in-the-loop phase retrieval
Upcoming at SIGGRAPH Asia 2020
![Page 55: High resolution holographic image synthesis for future ... · Future of Personal Computing 2. 3 Wide field of view High resolution Accommodation support Eyeglasses-Style Near-Eye](https://reader033.fdocuments.us/reader033/viewer/2022050516/5fa06b0613a11e265c1bdb1b/html5/thumbnails/55.jpg)
Differentiable forward model
Compute complex Wirtinger gradients
Optimize for phase holograms
WirtingerHolography
www.cs.unc.edu/~cpkPraneeth Chakravarthula