Perception-Based Histogram Equalization for Tone Mapping ...
INFORMATIK Design of a Tone Mapping Operator for High Dynamic Range Images based upon Psychophysical...
-
Upload
frederica-may -
Category
Documents
-
view
219 -
download
0
Transcript of INFORMATIK Design of a Tone Mapping Operator for High Dynamic Range Images based upon Psychophysical...
INFORMATIKINFORMATIK
Design of a Tone Mapping Operator Design of a Tone Mapping Operator for High Dynamic Range Images based for High Dynamic Range Images based upon Psychophysical Evaluation and upon Psychophysical Evaluation and
Preference MappingPreference Mapping
F. DragoF. Drago11, W. Martens, W. Martens22, K. Myszkowski, K. Myszkowski33, , and N. Chibaand N. Chiba11
11Iwate University and Iwate University and 22Aizu University, JapanAizu University, Japan33Max-Planck-Institut fMax-Planck-Institut füür Informatik, Germanyr Informatik, Germany
F. DragoF. Drago11, W. Martens, W. Martens22, K. Myszkowski, K. Myszkowski33, , and N. Chibaand N. Chiba11
11Iwate University and Iwate University and 22Aizu University, JapanAizu University, Japan33Max-Planck-Institut fMax-Planck-Institut füür Informatik, Germanyr Informatik, Germany
INFORMATIKINFORMATIKOverviewOverview
• MotivationMotivation• Previous workPrevious work• Psychophysical experimentPsychophysical experiment• Enhancements of Retinex for HDR imagesEnhancements of Retinex for HDR images• ConclusionsConclusions
• MotivationMotivation• Previous workPrevious work• Psychophysical experimentPsychophysical experiment• Enhancements of Retinex for HDR imagesEnhancements of Retinex for HDR images• ConclusionsConclusions
INFORMATIKINFORMATIKMotivationMotivation
Many applicationsMany applications• Lighting simulation and realistic renderingLighting simulation and realistic rendering• High Dynamic Range photographyHigh Dynamic Range photography• Multimedia: distributing HDR video streamsMultimedia: distributing HDR video streams
Many applicationsMany applications• Lighting simulation and realistic renderingLighting simulation and realistic rendering• High Dynamic Range photographyHigh Dynamic Range photography• Multimedia: distributing HDR video streamsMultimedia: distributing HDR video streams
INFORMATIKINFORMATIK
HDR Photographs + Rendering: HDR Photographs + Rendering: Real World LightingReal World Lighting
1) Photographs of mirror sphere at varying exposure times
2) High-dynamicrange environment map
3) Use as light source in Monte Carlo radiosity algorithm
Philippe Bekaert
INFORMATIKINFORMATIKGoalsGoals• Technical requirement Technical requirement
– Match the dynamic range of image to the range Match the dynamic range of image to the range available on a given display deviceavailable on a given display device
• Various objectivesVarious objectives– Get good perceptual match between the real-world Get good perceptual match between the real-world
and corresponding imagesand corresponding images– Reproducing detailsReproducing details– Maximize reproducible contrastMaximize reproducible contrast– Just to get “nice-looking” imagesJust to get “nice-looking” images
• Technical requirement Technical requirement – Match the dynamic range of image to the range Match the dynamic range of image to the range
available on a given display deviceavailable on a given display device
• Various objectivesVarious objectives– Get good perceptual match between the real-world Get good perceptual match between the real-world
and corresponding imagesand corresponding images– Reproducing detailsReproducing details– Maximize reproducible contrastMaximize reproducible contrast– Just to get “nice-looking” imagesJust to get “nice-looking” images
INFORMATIKINFORMATIKVarious ClassificationsVarious Classifications
• Theoretical foundationsTheoretical foundations– Perception-based Perception-based – Pure image processing techniquesPure image processing techniques
• Mapping functionMapping function– Global – the same for all pixelsGlobal – the same for all pixels– Local – depends on local image contentsLocal – depends on local image contents
• Temporal processingTemporal processing– StaticStatic– DynamicDynamic
• Theoretical foundationsTheoretical foundations– Perception-based Perception-based – Pure image processing techniquesPure image processing techniques
• Mapping functionMapping function– Global – the same for all pixelsGlobal – the same for all pixels– Local – depends on local image contentsLocal – depends on local image contents
• Temporal processingTemporal processing– StaticStatic– DynamicDynamic
INFORMATIKINFORMATIK
Previous Work:Previous Work:Global MethodsGlobal Methods
Perception-basedPerception-based• Tumblin and Rushmeier (1993,1999)Tumblin and Rushmeier (1993,1999)
– Brightness matchingBrightness matching• Ward (1994), Ferwerda et al. (1996)Ward (1994), Ferwerda et al. (1996)
– Contrast matching (a linear function is used)Contrast matching (a linear function is used)• Ward et al. (1997)Ward et al. (1997)
– Adjusting image histogram to avoid exceeding Adjusting image histogram to avoid exceeding display contrast in respect to the real-world scenedisplay contrast in respect to the real-world scene
Efficiency-drivenEfficiency-driven• Schlick (1994)Schlick (1994)
– Rational functionsRational functions
Perception-basedPerception-based• Tumblin and Rushmeier (1993,1999)Tumblin and Rushmeier (1993,1999)
– Brightness matchingBrightness matching• Ward (1994), Ferwerda et al. (1996)Ward (1994), Ferwerda et al. (1996)
– Contrast matching (a linear function is used)Contrast matching (a linear function is used)• Ward et al. (1997)Ward et al. (1997)
– Adjusting image histogram to avoid exceeding Adjusting image histogram to avoid exceeding display contrast in respect to the real-world scenedisplay contrast in respect to the real-world scene
Efficiency-drivenEfficiency-driven• Schlick (1994)Schlick (1994)
– Rational functionsRational functions
INFORMATIKINFORMATIKExamplesExamples
Ferwerda et al. Tumblin (1999) Ward et al. SchlickFerwerda et al. Tumblin (1999) Ward et al. Schlick
INFORMATIKINFORMATIK
Previous Work:Previous Work:Local MethodsLocal Methods
• Early methods – prone to halo artifactsEarly methods – prone to halo artifacts– Chiu et al. (1993), Schlick (1994), Chiu et al. (1993), Schlick (1994), – Land (1971), Jobson et al. (1997): RetinexLand (1971), Jobson et al. (1997): Retinex– Pattanaik et al. (1998): The most comprehensive Pattanaik et al. (1998): The most comprehensive
model of HVS used in CGmodel of HVS used in CG• LCIS: Tumblin and Turk (1999)LCIS: Tumblin and Turk (1999)
– Based on an anisotropic diffusion procedureBased on an anisotropic diffusion procedure– Emphasize on details but compress excessively Emphasize on details but compress excessively
contrastcontrast• New wave: Fattal et al., Reinhard et al., Durand and New wave: Fattal et al., Reinhard et al., Durand and
Dorsey, Ashikhmin (2002)Dorsey, Ashikhmin (2002)
• Early methods – prone to halo artifactsEarly methods – prone to halo artifacts– Chiu et al. (1993), Schlick (1994), Chiu et al. (1993), Schlick (1994), – Land (1971), Jobson et al. (1997): RetinexLand (1971), Jobson et al. (1997): Retinex– Pattanaik et al. (1998): The most comprehensive Pattanaik et al. (1998): The most comprehensive
model of HVS used in CGmodel of HVS used in CG• LCIS: Tumblin and Turk (1999)LCIS: Tumblin and Turk (1999)
– Based on an anisotropic diffusion procedureBased on an anisotropic diffusion procedure– Emphasize on details but compress excessively Emphasize on details but compress excessively
contrastcontrast• New wave: Fattal et al., Reinhard et al., Durand and New wave: Fattal et al., Reinhard et al., Durand and
Dorsey, Ashikhmin (2002)Dorsey, Ashikhmin (2002)
INFORMATIKINFORMATIK
Tumblin and Turk Retinex Ashikhmin
ExamplesExamples
INFORMATIKINFORMATIK
Durand and Dorsey Reinhard et al. Fattal et al.
ExamplesExamples
INFORMATIKINFORMATIK
Durand and Dorsey
Reinhard et al. Fattal et al.
Ashikhmin
INFORMATIKINFORMATIKPsychophysical ExperimentPsychophysical Experiment• Perceptual evaluation of subject preference by pairwise Perceptual evaluation of subject preference by pairwise
comparison of tone mapped imagescomparison of tone mapped images
• Seven tone mapping algorithms examined: Seven tone mapping algorithms examined:
– Tumblin and Rushmeier (1993), Tumblin and Rushmeier (1993),
– Ferwerda et al. (1996), Ferwerda et al. (1996),
– Ward et al. (1997), Ward et al. (1997),
– Schlick (1994), Schlick (1994),
– Retinex - based on Funt and Ciurea (2001) implementation Retinex - based on Funt and Ciurea (2001) implementation but with our extensions toward suppressing halobut with our extensions toward suppressing halo
– Reinhard et al. (2002) – photographic methodReinhard et al. (2002) – photographic method
– Tumblin and Turk (1999) - LCISTumblin and Turk (1999) - LCIS
• Four scenes consideredFour scenes considered
• Perceptual evaluation of subject preference by pairwise Perceptual evaluation of subject preference by pairwise comparison of tone mapped imagescomparison of tone mapped images
• Seven tone mapping algorithms examined: Seven tone mapping algorithms examined:
– Tumblin and Rushmeier (1993), Tumblin and Rushmeier (1993),
– Ferwerda et al. (1996), Ferwerda et al. (1996),
– Ward et al. (1997), Ward et al. (1997),
– Schlick (1994), Schlick (1994),
– Retinex - based on Funt and Ciurea (2001) implementation Retinex - based on Funt and Ciurea (2001) implementation but with our extensions toward suppressing halobut with our extensions toward suppressing halo
– Reinhard et al. (2002) – photographic methodReinhard et al. (2002) – photographic method
– Tumblin and Turk (1999) - LCISTumblin and Turk (1999) - LCIS
• Four scenes consideredFour scenes considered
INFORMATIKINFORMATIK
INFORMATIKINFORMATIKStatistical Data ProcessingStatistical Data Processing
• 11 subjects participated11 subjects participated• Dissimilarity ratings for pairwise comparisons of Dissimilarity ratings for pairwise comparisons of
images submitted to Individual Differences Scaling images submitted to Individual Differences Scaling (INDSCAL) analysis(INDSCAL) analysis
• Stimulus Space configures the stimuli such that Stimulus Space configures the stimuli such that Euclidian distances between the stimuli match the Euclidian distances between the stimuli match the obtained dissimilarity judgmentsobtained dissimilarity judgments
• Axes labeled based upon correlation of the Axes labeled based upon correlation of the dimensional coordinates with independently dimensional coordinates with independently generated attribute ratings (naturalness, detail and generated attribute ratings (naturalness, detail and contrast reproduction)contrast reproduction)
• ““Ideal” preference point obtained through PREFMAP Ideal” preference point obtained through PREFMAP analysisanalysis
• 11 subjects participated11 subjects participated• Dissimilarity ratings for pairwise comparisons of Dissimilarity ratings for pairwise comparisons of
images submitted to Individual Differences Scaling images submitted to Individual Differences Scaling (INDSCAL) analysis(INDSCAL) analysis
• Stimulus Space configures the stimuli such that Stimulus Space configures the stimuli such that Euclidian distances between the stimuli match the Euclidian distances between the stimuli match the obtained dissimilarity judgmentsobtained dissimilarity judgments
• Axes labeled based upon correlation of the Axes labeled based upon correlation of the dimensional coordinates with independently dimensional coordinates with independently generated attribute ratings (naturalness, detail and generated attribute ratings (naturalness, detail and contrast reproduction)contrast reproduction)
• ““Ideal” preference point obtained through PREFMAP Ideal” preference point obtained through PREFMAP analysisanalysis
INFORMATIKINFORMATIKSubject PreferencesSubject Preferences– TT: Tumblin & R.: Tumblin & R.
– VV: Ferwerda et al. : Ferwerda et al.
– HH: Ward et al. : Ward et al.
– QQ: Schlick : Schlick
– XX: Retinex: Retinex
– PP: Reinhard et al. : Reinhard et al.
– TT: Tumblin & R.: Tumblin & R.
– VV: Ferwerda et al. : Ferwerda et al.
– HH: Ward et al. : Ward et al.
– QQ: Schlick : Schlick
– XX: Retinex: Retinex
– PP: Reinhard et al. : Reinhard et al.
INFORMATIKINFORMATIKRetinexRetinexWe use the “Frankle-McCann Retinex” algorithmWe use the “Frankle-McCann Retinex” algorithm• ratio-product-reset-averageratio-product-reset-average
– NP(x,y) new pixel value is obtained from the original image R() and previous iteration image OP() as follows: NP(x,y) new pixel value is obtained from the original image R() and previous iteration image OP() as follows:
– Reset testReset test
• In each iteration (the number of iterations predefined by the user)In each iteration (the number of iterations predefined by the user)
– the distance D between pixels (x,y) and (xs,ys) is halved the distance D between pixels (x,y) and (xs,ys) is halved
– the direction for pixel comparison is rotated 90the direction for pixel comparison is rotated 90oo clockwise clockwise
• Main problem: Suppressing halo effectsMain problem: Suppressing halo effects
We use the “Frankle-McCann Retinex” algorithmWe use the “Frankle-McCann Retinex” algorithm• ratio-product-reset-averageratio-product-reset-average
– NP(x,y) new pixel value is obtained from the original image R() and previous iteration image OP() as follows: NP(x,y) new pixel value is obtained from the original image R() and previous iteration image OP() as follows:
– Reset testReset test
• In each iteration (the number of iterations predefined by the user)In each iteration (the number of iterations predefined by the user)
– the distance D between pixels (x,y) and (xs,ys) is halved the distance D between pixels (x,y) and (xs,ys) is halved
– the direction for pixel comparison is rotated 90the direction for pixel comparison is rotated 90oo clockwise clockwise
• Main problem: Suppressing halo effectsMain problem: Suppressing halo effects
2
),(log)),(log(),(log),((log),(log
yxOPysxsRyxRysxsOPyxNP
sceneLysxsRyxRysxsOP maxlog)),(log(),(log),((log
INFORMATIKINFORMATIKRetinex Extensions: for HDRRetinex Extensions: for HDR
• Main problem: Suppressing halo effectsMain problem: Suppressing halo effects
– Adding counterclockwise rotation of the pathAdding counterclockwise rotation of the path
suggested by Cooperssuggested by Coopers
– Spatially varying levels of pixel interaction based Spatially varying levels of pixel interaction based contrast informationcontrast information
Suggested by Sobol, but we use a smooth Suggested by Sobol, but we use a smooth function for clippingfunction for clipping
– Adjusting a reset ratio to the maximum luminance Adjusting a reset ratio to the maximum luminance of the display device instead of the maximum of the display device instead of the maximum luminance of the scene luminance of the scene
• Main problem: Suppressing halo effectsMain problem: Suppressing halo effects
– Adding counterclockwise rotation of the pathAdding counterclockwise rotation of the path
suggested by Cooperssuggested by Coopers
– Spatially varying levels of pixel interaction based Spatially varying levels of pixel interaction based contrast informationcontrast information
Suggested by Sobol, but we use a smooth Suggested by Sobol, but we use a smooth function for clippingfunction for clipping
– Adjusting a reset ratio to the maximum luminance Adjusting a reset ratio to the maximum luminance of the display device instead of the maximum of the display device instead of the maximum luminance of the scene luminance of the scene
INFORMATIKINFORMATIK
Halo Reduction: Halo Reduction: Retinex RotationRetinex Rotation
CounterClockwiseClockwise Both Ways
All images for 40 iterations
INFORMATIKINFORMATIK
Halo Reduction:Halo Reduction:Retinex Contrast Crop with BiasRetinex Contrast Crop with Bias
5.0log
log
)(functionBiasa
xxf
7.05.0log
log
a
DipContrastCl
;
)(
;
)(
),(log),(log
ipContrastClContrast
ipContrastClContrastifelse
ipContrastClContrast
ipContrastClContrastif
ysxsRyxRContrast
2
),(log)),(log(),(log),((log),(log
yxOPysxsRyxRysxsOPyxNP
INFORMATIKINFORMATIK
Halo Reduction:Halo Reduction:Retinex Contrast Crop with BiasRetinex Contrast Crop with Bias
Standard Retinex Standard Retinex 33 iterations cw and ccw33 iterations cw and ccw
The same settingsThe same settingsbut crop with bias addedbut crop with bias added
INFORMATIKINFORMATIK
Halo Reduction:Halo Reduction:Retinex Contrast Crop with BiasRetinex Contrast Crop with Bias
95.08.0
7.05.0log
log
aa
DipContrastCla
33 Retinex iterations33 Retinex iterations 33 Retinex iterations33 Retinex iterations
INFORMATIKINFORMATIK
Halo Reduction:Halo Reduction:Retinex Contrast Crop with BiasRetinex Contrast Crop with Bias
95.0
7.05.0log
log
a
DipContrastCla
4 Retinex iterations4 Retinex iterations 30 Retinex iterations30 Retinex iterations
INFORMATIKINFORMATIKRetinex Maximum Reset Retinex Maximum Reset
Maximum = 226.5 cd/m^2 Maximum = 100 cd/m^2
displayLysxsRyxRysxsOP maxlog)),(log(),(log),((log
display
scene
L
L
max
max
to
fromchange
INFORMATIKINFORMATIK
LinearLinearmappingmapping
RetinexRetinex4 iterations4 iterations
ExtendedExtendedRetinexRetinex
4 iterations4 iterations
ExtendedExtendedRetinexRetinex
4 iterations4 iterations
sceneLmaxdisplayLmax
INFORMATIKINFORMATIKRetinex + Tone Mapping Op.Retinex + Tone Mapping Op.
Ferwerda et al. (1996)Ferwerda et al. (1996) Logmap - newLogmap - new
INFORMATIKINFORMATIKLogmap EquationLogmap Equation
INFORMATIKINFORMATIK
Performance:Performance:• SoftwareSoftware
– 30 fps on 30 fps on PentiumIV, PentiumIV, 2.2GHz2.2GHz
• HardwareHardware– ??
Performance:Performance:• SoftwareSoftware
– 30 fps on 30 fps on PentiumIV, PentiumIV, 2.2GHz2.2GHz
• HardwareHardware– ??
Adaptive Adaptive Logarithmic Logarithmic Mapping Mapping
INFORMATIKINFORMATIKConclusionsConclusions
• We performed psychophysical of seven existing tone We performed psychophysical of seven existing tone mapping operators. More details in our TechRep:mapping operators. More details in our TechRep:
http://data.mpi-sb.mpg.de/internet/reports.nsf/AG4NumberView?OpenView
• Good performance of Retinex in the experiment Good performance of Retinex in the experiment encouraged us extend it toward reducing hallo encouraged us extend it toward reducing hallo artifactsartifacts
• Addind a regular tone mapping processing atop of Addind a regular tone mapping processing atop of Retinex results make the resulting images more Retinex results make the resulting images more independent on the number of Retinex iterations and independent on the number of Retinex iterations and improve the image naturalnessimprove the image naturalness
• Future work: repeating psychophysical with all recent Future work: repeating psychophysical with all recent local tone mapping operators and our extended local tone mapping operators and our extended Retinex Retinex
• We performed psychophysical of seven existing tone We performed psychophysical of seven existing tone mapping operators. More details in our TechRep:mapping operators. More details in our TechRep:
http://data.mpi-sb.mpg.de/internet/reports.nsf/AG4NumberView?OpenView
• Good performance of Retinex in the experiment Good performance of Retinex in the experiment encouraged us extend it toward reducing hallo encouraged us extend it toward reducing hallo artifactsartifacts
• Addind a regular tone mapping processing atop of Addind a regular tone mapping processing atop of Retinex results make the resulting images more Retinex results make the resulting images more independent on the number of Retinex iterations and independent on the number of Retinex iterations and improve the image naturalnessimprove the image naturalness
• Future work: repeating psychophysical with all recent Future work: repeating psychophysical with all recent local tone mapping operators and our extended local tone mapping operators and our extended Retinex Retinex
INFORMATIKINFORMATIKColor Balance CorrectionColor Balance Correction
Retinex Applied to All Channels Retinex Applied to All Channels in LMS Color Spacein LMS Color Space
INFORMATIKINFORMATIK
Stanford Memorial Church Stanford Memorial Church PhotographPhotograph
INFORMATIKINFORMATIK
Stanford Memorial Church Stanford Memorial Church PhotographPhotograph
INFORMATIKINFORMATIKAcknowledgments Acknowledgments
We would like to thank Michael Ashikhmin, We would like to thank Michael Ashikhmin, Paul Debevec, Fredo Durand, Dani Paul Debevec, Fredo Durand, Dani Lischinski, Eric Reinhard, and Greg Ward for Lischinski, Eric Reinhard, and Greg Ward for providing us with some images used in this providing us with some images used in this presentation.presentation.
We would like also to thank Greg Ward for We would like also to thank Greg Ward for his precious comments concerning our his precious comments concerning our work.work.
We would like to thank Michael Ashikhmin, We would like to thank Michael Ashikhmin, Paul Debevec, Fredo Durand, Dani Paul Debevec, Fredo Durand, Dani Lischinski, Eric Reinhard, and Greg Ward for Lischinski, Eric Reinhard, and Greg Ward for providing us with some images used in this providing us with some images used in this presentation.presentation.
We would like also to thank Greg Ward for We would like also to thank Greg Ward for his precious comments concerning our his precious comments concerning our work.work.