Efficient Single Image Super-Resolution via Hybrid ... · Single image super-resolution (SISR)...
Transcript of Efficient Single Image Super-Resolution via Hybrid ... · Single image super-resolution (SISR)...
![Page 1: Efficient Single Image Super-Resolution via Hybrid ... · Single image super-resolution (SISR) Low-resolution(LR) and lack of details . High-resolution(HR) and clear . Restore. Related](https://reader033.fdocuments.us/reader033/viewer/2022050513/5f9d9a02c179a476d32d84a2/html5/thumbnails/1.jpg)
Efficient Single Image Super-Resolution via Hybrid Residual Feature Learning
with Compact Back-Projection NetworkFeiyang Zhu, Qijun Zhao
College of Computer Science, Sichuan University, Chengdu, Sichuan, P. R. China
ICCV 2019, Seoul, Republic of Korea
![Page 2: Efficient Single Image Super-Resolution via Hybrid ... · Single image super-resolution (SISR) Low-resolution(LR) and lack of details . High-resolution(HR) and clear . Restore. Related](https://reader033.fdocuments.us/reader033/viewer/2022050513/5f9d9a02c179a476d32d84a2/html5/thumbnails/2.jpg)
Outline
• Background• Related Work• Method• Experiment• Conclusion
2
![Page 3: Efficient Single Image Super-Resolution via Hybrid ... · Single image super-resolution (SISR) Low-resolution(LR) and lack of details . High-resolution(HR) and clear . Restore. Related](https://reader033.fdocuments.us/reader033/viewer/2022050513/5f9d9a02c179a476d32d84a2/html5/thumbnails/3.jpg)
Background
3
Single image super-resolution (SISR)
Low-resolution(LR) and lack of details High-resolution(HR) and clear
Restore
![Page 4: Efficient Single Image Super-Resolution via Hybrid ... · Single image super-resolution (SISR) Low-resolution(LR) and lack of details . High-resolution(HR) and clear . Restore. Related](https://reader033.fdocuments.us/reader033/viewer/2022050513/5f9d9a02c179a476d32d84a2/html5/thumbnails/4.jpg)
Related Work• Deep Back-Projection Networks (DBPN[1])
• Iteration of up-projection unit and down-projection unit.• Concat the output of the up-projection units to reconstruct the
HR image. Mult-Adds: 5,112GTime: 6.25s
Not practical
Parameters: 10,426KModel size: 39.8MB
[1] M. Haris, G. Shakhnarovich, and N. Ukita. Deep back-projection networks for super-resolution. In CVPR, pages 1664–1673, 2018. 4
![Page 5: Efficient Single Image Super-Resolution via Hybrid ... · Single image super-resolution (SISR) Low-resolution(LR) and lack of details . High-resolution(HR) and clear . Restore. Related](https://reader033.fdocuments.us/reader033/viewer/2022050513/5f9d9a02c179a476d32d84a2/html5/thumbnails/5.jpg)
Related Work• Projection unit of DBPN
Iteration of LR to HR features
Iteration of HR to LR features
5
![Page 6: Efficient Single Image Super-Resolution via Hybrid ... · Single image super-resolution (SISR) Low-resolution(LR) and lack of details . High-resolution(HR) and clear . Restore. Related](https://reader033.fdocuments.us/reader033/viewer/2022050513/5f9d9a02c179a476d32d84a2/html5/thumbnails/6.jpg)
Method
6
• Motivation for improving DBPN– The feature information in DBPN network is not fully utilized.
– The large number of parameters and operations of DBPN.
– The learning pressure of the network is too great.
![Page 7: Efficient Single Image Super-Resolution via Hybrid ... · Single image super-resolution (SISR) Low-resolution(LR) and lack of details . High-resolution(HR) and clear . Restore. Related](https://reader033.fdocuments.us/reader033/viewer/2022050513/5f9d9a02c179a476d32d84a2/html5/thumbnails/7.jpg)
Method
7
• Network structure on ×4 scale
Three parts:1) low-level feature extraction2) projection3) reconstruction
Mult-Adds: 97.9GTime: 0.40s
Parameters: 1,197KModel size: 4.8MB
![Page 8: Efficient Single Image Super-Resolution via Hybrid ... · Single image super-resolution (SISR) Low-resolution(LR) and lack of details . High-resolution(HR) and clear . Restore. Related](https://reader033.fdocuments.us/reader033/viewer/2022050513/5f9d9a02c179a476d32d84a2/html5/thumbnails/8.jpg)
Method• Network structure on ×4 scale
Global residual connection 𝐼𝐼𝐻𝐻𝐻𝐻 = 𝐼𝐼𝐻𝐻0 + 𝛥𝛥𝐼𝐼𝐻𝐻
![Page 9: Efficient Single Image Super-Resolution via Hybrid ... · Single image super-resolution (SISR) Low-resolution(LR) and lack of details . High-resolution(HR) and clear . Restore. Related](https://reader033.fdocuments.us/reader033/viewer/2022050513/5f9d9a02c179a476d32d84a2/html5/thumbnails/9.jpg)
Method
9
• Network structure: UD Block
• Deconvolution layer• 64 filters • Use HR features
• Sub-pixel convolution layer• 32 filters• Use hybrid residual features
DBPN Ours
![Page 10: Efficient Single Image Super-Resolution via Hybrid ... · Single image super-resolution (SISR) Low-resolution(LR) and lack of details . High-resolution(HR) and clear . Restore. Related](https://reader033.fdocuments.us/reader033/viewer/2022050513/5f9d9a02c179a476d32d84a2/html5/thumbnails/10.jpg)
Method
10
• More lightweight network
Con
cat
Con
cat
Parameters Mult-Adds
No compression layer 1,664K 122.0G
64 filters 4,746K 388.0G
Add compression layers 1,197K 97.9G
32 filters 1,197K 97.9G
![Page 11: Efficient Single Image Super-Resolution via Hybrid ... · Single image super-resolution (SISR) Low-resolution(LR) and lack of details . High-resolution(HR) and clear . Restore. Related](https://reader033.fdocuments.us/reader033/viewer/2022050513/5f9d9a02c179a476d32d84a2/html5/thumbnails/11.jpg)
Experiments
11
• Dataset– DIV2K(800 images for training, 100 images for validating)– Testing dataset: Set5/Set14/BSDS100/Urban100
• Data expansion– Randomly flipping LR images horizontally or vertically– Randomly rotating LR images by 90°
![Page 12: Efficient Single Image Super-Resolution via Hybrid ... · Single image super-resolution (SISR) Low-resolution(LR) and lack of details . High-resolution(HR) and clear . Restore. Related](https://reader033.fdocuments.us/reader033/viewer/2022050513/5f9d9a02c179a476d32d84a2/html5/thumbnails/12.jpg)
Experiments
12
What features are better in the reconstruction?LR or HR residual features ? or hybrid residual features?
HR residual features
LR residual features
SR accuracy in terms of PSNR (dB) of our CBPNwith or without using LR/HR features on three benchmark datasets for ×2 SR.
![Page 13: Efficient Single Image Super-Resolution via Hybrid ... · Single image super-resolution (SISR) Low-resolution(LR) and lack of details . High-resolution(HR) and clear . Restore. Related](https://reader033.fdocuments.us/reader033/viewer/2022050513/5f9d9a02c179a476d32d84a2/html5/thumbnails/13.jpg)
Experiments
13
Reconstruction quality
The number of HR residual features
The model gets best SR results when T=3. SR accuracy in terms of PSNR of our CBPN for ×2 SR on B100 and Urban100 datasets w.r.t. the number of used intermediate HR residual features generated by the UD blocks
• Ablation study
![Page 14: Efficient Single Image Super-Resolution via Hybrid ... · Single image super-resolution (SISR) Low-resolution(LR) and lack of details . High-resolution(HR) and clear . Restore. Related](https://reader033.fdocuments.us/reader033/viewer/2022050513/5f9d9a02c179a476d32d84a2/html5/thumbnails/14.jpg)
Experiments
14
• Trade-off between SR accuracy (PSNR) and the number of operations
Quantitative comparison results between our CBPNand D-DBPN-L for × 4 SR.
![Page 15: Efficient Single Image Super-Resolution via Hybrid ... · Single image super-resolution (SISR) Low-resolution(LR) and lack of details . High-resolution(HR) and clear . Restore. Related](https://reader033.fdocuments.us/reader033/viewer/2022050513/5f9d9a02c179a476d32d84a2/html5/thumbnails/15.jpg)
Experiments
15
• Quantitative results on X2 scale
![Page 16: Efficient Single Image Super-Resolution via Hybrid ... · Single image super-resolution (SISR) Low-resolution(LR) and lack of details . High-resolution(HR) and clear . Restore. Related](https://reader033.fdocuments.us/reader033/viewer/2022050513/5f9d9a02c179a476d32d84a2/html5/thumbnails/16.jpg)
Experiments
16
• Quantitative results on X4 scale
![Page 17: Efficient Single Image Super-Resolution via Hybrid ... · Single image super-resolution (SISR) Low-resolution(LR) and lack of details . High-resolution(HR) and clear . Restore. Related](https://reader033.fdocuments.us/reader033/viewer/2022050513/5f9d9a02c179a476d32d84a2/html5/thumbnails/17.jpg)
Experiments• Visual comparison on ×2 scale
17
HR Bicubic SRCNN FALSR-A CARN CARN-M IDN CBPN CBPN-S
![Page 18: Efficient Single Image Super-Resolution via Hybrid ... · Single image super-resolution (SISR) Low-resolution(LR) and lack of details . High-resolution(HR) and clear . Restore. Related](https://reader033.fdocuments.us/reader033/viewer/2022050513/5f9d9a02c179a476d32d84a2/html5/thumbnails/18.jpg)
Experiments• Visual comparison on × 4 scale
18HR Bicubic SRCNN CARN CBPN
![Page 19: Efficient Single Image Super-Resolution via Hybrid ... · Single image super-resolution (SISR) Low-resolution(LR) and lack of details . High-resolution(HR) and clear . Restore. Related](https://reader033.fdocuments.us/reader033/viewer/2022050513/5f9d9a02c179a476d32d84a2/html5/thumbnails/19.jpg)
Conclusion• We propose compact back-projection networks (CBPN) for efficient
single image super-resolution.• UD-block ensures the efficiency of our model and hybrid residual features
information are used in the final reconstruction.• We use global and local residual connection to promote our network
learning residual images between HR images and interpolated images.• Compression layers are employed to fusion features and reduce the
number of parameters and operations of our network.
19
![Page 20: Efficient Single Image Super-Resolution via Hybrid ... · Single image super-resolution (SISR) Low-resolution(LR) and lack of details . High-resolution(HR) and clear . Restore. Related](https://reader033.fdocuments.us/reader033/viewer/2022050513/5f9d9a02c179a476d32d84a2/html5/thumbnails/20.jpg)
Q & A
Thanks!
20