Perception-motivated High Dynamic Range Video Encoding
description
Transcript of Perception-motivated High Dynamic Range Video Encoding
Perception-motivated Perception-motivated High Dynamic Range High Dynamic Range
Video Video EncodingEncoding
Perception-motivated Perception-motivated High Dynamic Range High Dynamic Range
Video Video EncodingEncoding
Rafal Mantiuk, Grzegorz Krawczyk,Rafal Mantiuk, Grzegorz Krawczyk,Karol Myszkowski, Hans-Peter SeidelKarol Myszkowski, Hans-Peter SeidelRafal Mantiuk, Grzegorz Krawczyk,Rafal Mantiuk, Grzegorz Krawczyk,
Karol Myszkowski, Hans-Peter SeidelKarol Myszkowski, Hans-Peter Seidel
INFORMATIKINFORMATIK
[log cd/m^2]Luminance
-6 -4 -2 0 2 4 6 8
Bright projector
LCD Monitor
CRT Monitor
Human eye
High Dynamic RangeHigh Dynamic RangeHigh Dynamic RangeHigh Dynamic Range
High vs Low High vs Low Dynamic Range VideoDynamic Range VideoHigh vs Low High vs Low Dynamic Range VideoDynamic Range Video
LDR Video
Intended for
existing displays
Relative pixel brightness
HDR Video
Intended for
the human eye
Photometric or radiometric units [cd/m2, Watt/m2sr]
High Dynamic Range VideoHigh Dynamic Range VideoHigh Dynamic Range VideoHigh Dynamic Range Video
Goal: Efficient encoding of full dynamic range of luminance perceived by the human observer
Luminance-6 -4 -2 0 2 4 6 8
[log cd/m^2]
• 1st demo
OverviewOverviewOverviewOverview
1. HDR Pipeline
2. HDR Video Encoding• Luminance Quantization
• Edge Coding
3. Results • vs. MPEG-4
• vs. OpenEXR
4. Demo & Applications
RelatedRelated Work WorkRelatedRelated Work Work
HDR Pipeline
Acquisition Storage Display
RelatedRelated Work WorkRelatedRelated Work Work
HDR Pipeline
Acquisition Storage Display
• Global Illumination
• HDR Cameras
•HDRC (IMS Chips)
•Lars III (Silicon Vision)
•Autobrite (SMal Camera Technologies)
•LM9628 (National)
•Digital Pixel System (Pixim)
•Technology overview [Nayar2003]
HDRC – IMS Chips
RelatedRelated Work WorkRelatedRelated Work Work
HDR Pipeline
Acquisition Storage Display
• Still images
•Radiance – RGBE [Ward91]
•OpenEXR [Bogart2003]
•logLuv TIFF [Ward98]
•HDR JPEG [Ward2004]
• Video
•No video format
RelatedRelated Work WorkRelatedRelated Work Work
HDR Pipeline
Acquisition Storage Display
• LDR Displays
•But Tone Mapping necessary
• HDR displays start to appear
•University of British Columbia [Seetzen2004]
HDR Encoding FrameworkHDR Encoding FrameworkHDR Encoding FrameworkHDR Encoding Framework
Detail level 1: Input & Output
White: MPEG
Orange: HDR Encoder
HDR
LDR bitstreamVideo encoder
HDR Encoding FrameworkHDR Encoding FrameworkHDR Encoding FrameworkHDR Encoding Framework
Detail level 2: Color Transform
White: MPEG
Orange: HDR Encoder
bitstreamColorTransform Encoder
Video
HDR
LDR YCrCb
L u'v'p
HDR Encoding FrameworkHDR Encoding FrameworkHDR Encoding FrameworkHDR Encoding Framework
Detail level 3: Edge Coding
White: MPEG
Orange: HDR Encoder
ColorTran. Comp.
MotionCodingDCT
Run-length
EdgeCoding
LDR
HDR
bitstream
Variablelength
ColorTran. Comp.
MotionCodingDCT
Run-length
EdgeCoding
LDR RGB
HDR XYZ
bitstream
Variablelength
Encoding of ColorEncoding of ColorEncoding of ColorEncoding of Color
Encoding of ColorEncoding of ColorEncoding of ColorEncoding of Color
How to represent color data?• Floating Points – ineffective compression• Integers – ok, but require quantization
How to quantize color data?• Quantization errors < threshold of perception• Use uniform color space (L*u*v*, L*a*b*) [Ward98]• Find minimum number of bits
Color (u*v*) – 8 bits are enough
Encoding of Encoding of LuminanceLuminanceEncoding of Encoding of LuminanceLuminance
How to quantize luminance?• Gamma correction?• Logarithm?
Integer representation
log
Lu
min
anc
e Y
-4
-2
0
2
4
6
8
log(Y)?
Threshold Versus IntensityThreshold Versus IntensityThreshold Versus IntensityThreshold Versus Intensity
Psychophysical measurements• The smallest perceivable difference Y for a
certain adaptation level YA
• tvi [Ferwerda96, CIE 12/2.1]
YA - Adaptation
Luminance
Ytvi
-2-4 0 2 4 6 8-2
0
2
4
6
log Adaptation Luminance YA
log
Thr
esh
old
Y
Maximum quantization error
Luminance QuantizationLuminance QuantizationLuminance QuantizationLuminance Quantization
-4
-2
0
2
4
6
8
0 2048Integers Lp
log
Lu
min
anc
e Y
f
Ytvi )(
maxe
Just below threshold of perception
Maximum quantization error
Luminance QuantizationLuminance QuantizationLuminance QuantizationLuminance Quantization
-4
-2
0
2
4
6
8
0 2048Integers Lp
log
Lu
min
anc
e Y
f
Ytvi )( ))((2)( 1 ltvif
dl
ld
maxe
decrease threshold
mapping to)(
space in
f
YLl
LLl
P
P
Just below threshold of perception
Capacity function [Ashkihmin02]
Grayscale Standard Display Function [DICOM03]
10 – 11 bits are enough
Luminance QuantizationsLuminance QuantizationsComparisonComparisonLuminance QuantizationsLuminance QuantizationsComparisonComparison
RGBE
-2 0 2 4-4 6 8log
Con
tras
t Thr
esho
ld
log Adapting Luminance
-4
-2
0
2
32-bit LogLuv
cvi11-bit percep. quant.
ColorTran. Comp.
MotionCodingDCT
Run-length
EdgeCoding
LDR RGB
HDR XYZ
bitstream
Variablelength
Edge CodingEdge CodingEdge CodingEdge Coding
Edge CodingEdge Coding: : MotivationMotivationEdge CodingEdge Coding: : MotivationMotivation
HDR video can contain sharp contrast edges• Light sources, shadows
DCT coding of sharp contrast may cause high frequency artifacts
DCT coding Edge coding
Edge CodingEdge Coding: : SolutionSolutionEdge CodingEdge Coding: : SolutionSolution
Solution: Encode sharp edges in spatial domain, the rest in frequency domain
Original Signal
Sharp edge signal
Smoothed signal
Run-length
encoding
DCT
encoding
Edge CodingEdge Coding: : AlgorithmAlgorithmEdge CodingEdge Coding: : AlgorithmAlgorithm
original
horizontal decomposition
edge block
horiz. edges
edge block
vert. edges
horizontal DCT
vertical decomposition
vertical DCT
I
II
III
IV
Edge CodingEdge Coding: : AlgorithmAlgorithmEdge CodingEdge Coding: : AlgorithmAlgorithm
original
horizontal decomposition
edge block
horiz. edges
edge block
vert. edges
horizontal DCT
vertical decomposition
vertical DCT
I
II
III
IV
Edge CodingEdge Coding: : AlgorithmAlgorithmEdge CodingEdge Coding: : AlgorithmAlgorithm
original
horizontal decomposition
edge block
horiz. edges
edge block
vert. edges
horizontal DCT
vertical decomposition
vertical DCT
I
II
III
IV
Edge CodingEdge Coding: : AlgorithmAlgorithmEdge CodingEdge Coding: : AlgorithmAlgorithm
original
horizontal decomposition
edge block
horiz. edges
edge block
vert. edges
horizontal DCT
vertical decomposition
vertical DCT
I
II
III
IV
Edge CodingEdge Coding: : AlgorithmAlgorithmEdge CodingEdge Coding: : AlgorithmAlgorithm
original
horizontal decomposition
edge block
horiz. edges
edge block
vert. edges
horizontal DCT
vertical decomposition
vertical DCT
I
II
III
IV
ResultsResultsResultsResults
2x size of tone-mapped MPEG-4 video
20-30x saving compared to intra-frame compression (OpenEXR)
27
1
0.55
OpenEXR
HDR Enc.
MPEG-4
Bit-stream Size
Demo & ApplicationsDemo & ApplicationsDemo & ApplicationsDemo & Applications
Display dependent rendering
Choice of tone-mapping
Extended postprocessing
ConclusionsConclusionsConclusionsConclusions
HDR video compression• Modest changes to MPEG-4
• Lpu’v’ color space– Luminance quantization (10-11 bits)
• Edge coding
Applications• On-the-fly tone mapping
• Blooming, motion blur, night vision
• Tuned for display– LDR / HDR Display
AcknowledgmentsAcknowledgmentsAcknowledgmentsAcknowledgments
HDR Images and Sequences• Paul Debevec• SpheronVR• Jozef Zajac• Christian Fuchs• Patrick Reuter
HDR Camera• HDRC(R) VGAx
courtesy of IMS CHIPSwww.hdrc.com
Comments and help• Volker Blanz
• Scott Daly
• Michael Goesele
• Jeffrey Schoner
Thank youThank youThank youThank you
http://www.mpi-sb.mpg.de/resources/hdrvideo/http://www.mpi-sb.mpg.de/resources/hdrvideo/http://www.mpi-sb.mpg.de/resources/hdrvideo/http://www.mpi-sb.mpg.de/resources/hdrvideo/