Objective evaluation and state-of-the-art video ... · video compression standard based solutions...
Transcript of Objective evaluation and state-of-the-art video ... · video compression standard based solutions...
![Page 1: Objective evaluation and state-of-the-art video ... · video compression standard based solutions for plenoptic image content Roger Olsson, PhD Mid Sweden University, Sundsvall, Sweden](https://reader030.fdocuments.us/reader030/viewer/2022011823/5ed5bf0da3a704371133d271/html5/thumbnails/1.jpg)
Objective evaluation and state-of-the-art video compression standard based
solutions for plenoptic image contentRoger Olsson, PhD
Mid Sweden University, Sundsvall, Sweden
JPEG PLENO Workshop, Warzaw, Poland, June 23, 20151
![Page 2: Objective evaluation and state-of-the-art video ... · video compression standard based solutions for plenoptic image content Roger Olsson, PhD Mid Sweden University, Sundsvall, Sweden](https://reader030.fdocuments.us/reader030/viewer/2022011823/5ed5bf0da3a704371133d271/html5/thumbnails/2.jpg)
Outline• Introduction
• Plenoptic imaging
• An extended signal pipeline
• Video compression applied to plenoptic images
• Objective evaluation methodology
• Summary
2
![Page 3: Objective evaluation and state-of-the-art video ... · video compression standard based solutions for plenoptic image content Roger Olsson, PhD Mid Sweden University, Sundsvall, Sweden](https://reader030.fdocuments.us/reader030/viewer/2022011823/5ed5bf0da3a704371133d271/html5/thumbnails/3.jpg)
Plenoptic research at MIUN
• Active in the field of plenoptics since 2005
• 4 senior researchers, 2-3 PhD students
• Research interests• Modeling of, and synthesis from, plenoptic systems• Compression of plenoptic image and video• Quality evaluation
• Plenoptic related projects• Plenocap – Plenoptic Capture and Computational Photography
(2013-2015)• LIFE – Light Field Evaluation System
(2015-2018)• Marie Sklodowska-Curie International Training Network Full Parallax
Imaging (2015-2019)
www.miun.se/stc/Realistic3D
2 PhD student positions open
3
![Page 4: Objective evaluation and state-of-the-art video ... · video compression standard based solutions for plenoptic image content Roger Olsson, PhD Mid Sweden University, Sundsvall, Sweden](https://reader030.fdocuments.us/reader030/viewer/2022011823/5ed5bf0da3a704371133d271/html5/thumbnails/4.jpg)
Plenoptic imaging
Digitized film
Based on infographics from Optics Central Australia and Gregor Halenda, Popular Mechanics 4
![Page 5: Objective evaluation and state-of-the-art video ... · video compression standard based solutions for plenoptic image content Roger Olsson, PhD Mid Sweden University, Sundsvall, Sweden](https://reader030.fdocuments.us/reader030/viewer/2022011823/5ed5bf0da3a704371133d271/html5/thumbnails/5.jpg)
Plenoptic imaging
Digitized film
Digitized optics
Based on infographics from Optics Central Australia and Gregor Halenda, Popular Mechanics 4
![Page 6: Objective evaluation and state-of-the-art video ... · video compression standard based solutions for plenoptic image content Roger Olsson, PhD Mid Sweden University, Sundsvall, Sweden](https://reader030.fdocuments.us/reader030/viewer/2022011823/5ed5bf0da3a704371133d271/html5/thumbnails/6.jpg)
Plenoptic imaging
Digitized film
Digitized optics
Based on infographics from Optics Central Australia and Gregor Halenda, Popular Mechanics
”Upgrade now!Best Lens Award 2025”
4
![Page 7: Objective evaluation and state-of-the-art video ... · video compression standard based solutions for plenoptic image content Roger Olsson, PhD Mid Sweden University, Sundsvall, Sweden](https://reader030.fdocuments.us/reader030/viewer/2022011823/5ed5bf0da3a704371133d271/html5/thumbnails/7.jpg)
Plenoptic imaging• 4D light field
spatially multiplexed onto 2D sensor
• The spatial multiplexing properties are yet not fixed
• The statistical properties of the 2D sensor ”image” varies
5
![Page 8: Objective evaluation and state-of-the-art video ... · video compression standard based solutions for plenoptic image content Roger Olsson, PhD Mid Sweden University, Sundsvall, Sweden](https://reader030.fdocuments.us/reader030/viewer/2022011823/5ed5bf0da3a704371133d271/html5/thumbnails/8.jpg)
Plenoptic imaging compressionThe signal target has become targets and they are all moving in terms of properties as new technologies emerge
6
![Page 9: Objective evaluation and state-of-the-art video ... · video compression standard based solutions for plenoptic image content Roger Olsson, PhD Mid Sweden University, Sundsvall, Sweden](https://reader030.fdocuments.us/reader030/viewer/2022011823/5ed5bf0da3a704371133d271/html5/thumbnails/9.jpg)
Signal processing chainconventional imaging
Source Pre-processing Compression Distribution
DecompressionPost-processingSink
2D-image
2D-image
7
![Page 10: Objective evaluation and state-of-the-art video ... · video compression standard based solutions for plenoptic image content Roger Olsson, PhD Mid Sweden University, Sundsvall, Sweden](https://reader030.fdocuments.us/reader030/viewer/2022011823/5ed5bf0da3a704371133d271/html5/thumbnails/10.jpg)
Signal processing chainconventional imaging
Source Pre-processing Compression Distribution
DecompressionPost-processingSink
2D-image
2D-image
Evaluate
HVS properties highly relevant
7
![Page 11: Objective evaluation and state-of-the-art video ... · video compression standard based solutions for plenoptic image content Roger Olsson, PhD Mid Sweden University, Sundsvall, Sweden](https://reader030.fdocuments.us/reader030/viewer/2022011823/5ed5bf0da3a704371133d271/html5/thumbnails/11.jpg)
Signal processing chainconventional imaging
Source Pre-processing Compression Distribution
DecompressionPost-processingSink
2D-image
2D-image
Evaluate
HVS properties highly relevant
7
![Page 12: Objective evaluation and state-of-the-art video ... · video compression standard based solutions for plenoptic image content Roger Olsson, PhD Mid Sweden University, Sundsvall, Sweden](https://reader030.fdocuments.us/reader030/viewer/2022011823/5ed5bf0da3a704371133d271/html5/thumbnails/12.jpg)
Signal processing chainconventional imaging
Source Pre-processing Compression Distribution
DecompressionPost-processingSink
2D-image
2D-image
Evaluate
HVS properties highly relevant
7
![Page 13: Objective evaluation and state-of-the-art video ... · video compression standard based solutions for plenoptic image content Roger Olsson, PhD Mid Sweden University, Sundsvall, Sweden](https://reader030.fdocuments.us/reader030/viewer/2022011823/5ed5bf0da3a704371133d271/html5/thumbnails/13.jpg)
Signal processing chain plenoptic imaging
Source Pre-processing Compression Distribution
Post-processingSinc
4D datalight rays
Multiview Post-processing
Post-processing
Sinc
Sinc
Decompression
2D image
ND (N ≤ 4)
8
![Page 14: Objective evaluation and state-of-the-art video ... · video compression standard based solutions for plenoptic image content Roger Olsson, PhD Mid Sweden University, Sundsvall, Sweden](https://reader030.fdocuments.us/reader030/viewer/2022011823/5ed5bf0da3a704371133d271/html5/thumbnails/14.jpg)
Signal processing chain plenoptic imaging
Source Pre-processing Compression Distribution
Post-processingSinc
Post-processing
Post-processing
Sinc
Sinc
Decompression
• Depth• Reflectance• Transparency• …• Full light field
Evaluate
HVS properties less relevant
8
![Page 15: Objective evaluation and state-of-the-art video ... · video compression standard based solutions for plenoptic image content Roger Olsson, PhD Mid Sweden University, Sundsvall, Sweden](https://reader030.fdocuments.us/reader030/viewer/2022011823/5ed5bf0da3a704371133d271/html5/thumbnails/15.jpg)
Signal processing chain plenoptic imaging
Source Pre-processing Compression Distribution
Post-processingSinc
Post-processing
Post-processing
Sinc
Sinc
Decompression
• Depth• Reflectance• Transparency• …• Full light field
Evaluate
HVS properties highly relevant
8
![Page 16: Objective evaluation and state-of-the-art video ... · video compression standard based solutions for plenoptic image content Roger Olsson, PhD Mid Sweden University, Sundsvall, Sweden](https://reader030.fdocuments.us/reader030/viewer/2022011823/5ed5bf0da3a704371133d271/html5/thumbnails/16.jpg)
Rendering to 3D
Uncompressed 24 bpp Uncompressed 24 bpp
JPEG 2000 0.22 bpp JPEG 2000 0.22 bpp
MPEG-4 AVC 0.22 bpp MPEG-4 AVC 0.22 bpp
Right view Left view 10
![Page 17: Objective evaluation and state-of-the-art video ... · video compression standard based solutions for plenoptic image content Roger Olsson, PhD Mid Sweden University, Sundsvall, Sweden](https://reader030.fdocuments.us/reader030/viewer/2022011823/5ed5bf0da3a704371133d271/html5/thumbnails/17.jpg)
Objective evaluation
Metric dimensionality
0D
1D
2D
3D
4D
Perceptualrelevance
Low HighMedium
11
![Page 18: Objective evaluation and state-of-the-art video ... · video compression standard based solutions for plenoptic image content Roger Olsson, PhD Mid Sweden University, Sundsvall, Sweden](https://reader030.fdocuments.us/reader030/viewer/2022011823/5ed5bf0da3a704371133d271/html5/thumbnails/18.jpg)
Objective evaluation
Metric dimensionality
0D
1D
2D
3D
4D
Perceptualrelevance
Low HighMedium
HVS
Metrology
11
![Page 19: Objective evaluation and state-of-the-art video ... · video compression standard based solutions for plenoptic image content Roger Olsson, PhD Mid Sweden University, Sundsvall, Sweden](https://reader030.fdocuments.us/reader030/viewer/2022011823/5ed5bf0da3a704371133d271/html5/thumbnails/19.jpg)
Objective evaluationExtract signal components and evaluate those separately
Olsson, R. & Sjöström, M. (2007). A depth dependent quality metric for evaluation of coded integral imaging based 3d-images. In Proceedings of 3DTV Conference. New York : . S. 403--406.
Quality as a function of viewing angle Quality as a function of scene depth
12
![Page 20: Objective evaluation and state-of-the-art video ... · video compression standard based solutions for plenoptic image content Roger Olsson, PhD Mid Sweden University, Sundsvall, Sweden](https://reader030.fdocuments.us/reader030/viewer/2022011823/5ed5bf0da3a704371133d271/html5/thumbnails/20.jpg)
Video compression for plenoptic image content• Extend HEVC to make
temporal compression tools available for intra compression
• Multi-hypothesis prediction(Bi-directional prediction)
L0L1L0
Prediction Bi-directional prediction13
![Page 21: Objective evaluation and state-of-the-art video ... · video compression standard based solutions for plenoptic image content Roger Olsson, PhD Mid Sweden University, Sundsvall, Sweden](https://reader030.fdocuments.us/reader030/viewer/2022011823/5ed5bf0da3a704371133d271/html5/thumbnails/21.jpg)
Rendering to all-in-focus
Images rendered using light field from Todor Georgiev, QualcommLi, Y. , Sjöström, M. , Olsson, R. & Jennehag, U. (2014). Efficient Intra Prediction Scheme For Light Field Image Compression. I ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings.. S. Art. no. 6853654
14
![Page 22: Objective evaluation and state-of-the-art video ... · video compression standard based solutions for plenoptic image content Roger Olsson, PhD Mid Sweden University, Sundsvall, Sweden](https://reader030.fdocuments.us/reader030/viewer/2022011823/5ed5bf0da3a704371133d271/html5/thumbnails/22.jpg)
Summary• Plenoptic imaging with digitized optics is a paradigm shift
• Applications relying on plenoptic imaging will• produce and process a number of intermediate signals• generate outputs with different dimensionality
• Evaluating the performance of each processing step is necessary
• Perceptual relevant metrics, and signal property metrics, are both important
15
![Page 23: Objective evaluation and state-of-the-art video ... · video compression standard based solutions for plenoptic image content Roger Olsson, PhD Mid Sweden University, Sundsvall, Sweden](https://reader030.fdocuments.us/reader030/viewer/2022011823/5ed5bf0da3a704371133d271/html5/thumbnails/23.jpg)
Thank you
16