CS B553: A LGORITHMS FOR O PTIMIZATION AND L EARNING Global optimization 1.
P ERCEPTUAL E VALUATION OF M ULTI -E XPOSURE I MAGE F USION A LGORITHMS Kai Zeng, Kede Ma, Rania...
-
Upload
audrey-bruce -
Category
Documents
-
view
214 -
download
1
Transcript of P ERCEPTUAL E VALUATION OF M ULTI -E XPOSURE I MAGE F USION A LGORITHMS Kai Zeng, Kede Ma, Rania...
PERCEPTUAL EVALUATION OF MULTI-EXPOSUREIMAGE FUSION ALGORITHMS
Kai Zeng, Kede Ma, Rania Hassen and Zhou Wang
Department of Electrical and Computer Engineering University of Waterloo
September 2014
Introduction Subjective Experiment
Results Conclusion
MEF vs Tone Mapping
HDRreconstruction Tone mapping
MEF
Alternative Methods High dynamic range (HDR) image reconstruction followed by tone mapping MEF – bypassing the intermediate HDR stage
3 / 18
Introduction Subjective Experiment
Results Conclusion
Existing MEF algorithms
Cues used in existing MEF methods
Local energy and correlation (in Laplacian pyramid)
[Burt1993] Entropy [Goshtasby2005]
Contrast, color saturation and luminance exposure
[Mertens2007] Edge information (using bilateral filter)
[Raman2009]
Visual contrast and scene gradient [Song2012]
Visibility and consistency (from gradient information)
[Zhang2012] Local contrast, brightness and color dissimilarity
[Li2012]
Pixel saliency and spatial consistency [Li2013]
......
Question:Which method produces the best quality image?
4 / 18
IntroductionSubjective Experiment
Results Conclusions
Existing objective image quality assessment (IQA) models for image fusion
Existing IQA models for image fusion
Information theoretical models [Hossny, 2008, Cvejic, 2006, Wang, 2008]
Image feature based models [Xydeas, 2000, WangPW, 2008, Zheng, 2007]
Image structural similarity based models [Piella, 2003]
Human perception based models [Chen, 2007, Chen, 2009]
Question:
Which model better predicts perceptual quality?
5 / 20
Image Database
17 natural scenes of images with multi-exposure levels
8 image fusion algorithms
Simple methods: local and global energy weighted linear combination Advanced models: [Raman09, Gu12, ShutaoLi12, Shutaoli13, Li12, Mertens07]
Subjective Test
Absolute category rating from 1 to 10
No reference images are shown
25 naïve observers (15 male and 10 female, between 22 and 30)
Each session limited to within 30 mins
6 / 18
Introduction Subjective Experiment
Results Conclusion
Subjective Experiment Mapping
Introduction Subjective Experiment
Results Conclusion
Images Examples
(a1) Under Exposure (a2) Normal Exposure (a3) Over Exposure
(b) Global Energy Weighted (c) Gu12 (d) Local Energy Weighted
8 / 18
Introduction Subjective Experiment
Results Conclusion
Images Examples
(e) Li12 (f) Li13 (g) Mertens07
(h) Raman09 (i) ShutaoLi12
9 / 18
Introduction Subjective Experiment
Results Conclusion
Subjective Data Analysis
Table: Correlation between individual and average subject scores
10 / 18
Subject PLCC SRCC Subject PLCC SRCC1 0.8743 0.8631 13 0.8411 0.79892 0.8245 0.7984 14 0.8781 0.87433 0.7102 0.6735 15 0.8988 0.89244 0.8093 0.8182 16 0.7413 0.73135 0.6785 0.6649 17 0.7347 0.64886 0.6544 0.6567 18 0.7797 0.74867 0.8198 0.8030 19 0.6732 0.68148 0.8951 0.8849 20 0.7854 0.76439 0.7961 0.7835 21 0.6045 0.5638
10 0.6924 0.6826 22 0.6213 0.612111 0.8298 0.8275 23 0.7976 0.755812 0.6145 0.5795 Average 0.763
30.7438
Introduction Subjective Experiment
Results Conclusion
Subjective Data Analysis: Pearson Linear CC
Figure: Mean and standard deviation of PLCC between individual subject and average subject for each image set. The rightmost column gives the average performance of all subjects.
11 / 18
Introduction Subjective Experiment
Results Conclusion
Subjective Data Analysis: Spearman ROCC
Figure: Mean and standard deviation of SRCC between individual subject and average subject for each image set. The rightmost column gives the average performance of all subjects.
12 / 18
Considerable consistency between subjects
Behavior of individual subjects may vary
“Average Subject”: a useful concept
Baseline to observe behaviors of individual subject
Baseline to evaluate behaviors of objective models
Introduction Subjective Experiment
Results Conclusion
Subjective Data Analysis: Observations
Introduction Subjective Experiment
Results Conclusion
Performance of MEF Algorithms
Figure: Mean and standard deviation of subjective rankings of individual fuser acrossall image sets. 13 /
18
• Considerable subject agreement on individual MEF but performance difference between MEFs could be small
• No single method performs best on all image sets
• Best performance [Mertens07] (on average)
Second best [Li12]: a detail-enhanced version of [Mertens07]
• Global energy weighting significantly better than local energy weighting
• Some “advanced” MEFs show no advantage over simple methods
Introduction Subjective Experiment
Results Conclusion
Performance of MEF Algorithms: Observations
Table: Performance evaluation of objective IQA models
16 / 18
IQA model PLCC SRCCµ σ µ σ
Hossny08 -0.2939 0.2054 -0.2784 0.2803Cvejic06 0.0604 0.4311 0.0590 0.4968
Q. Wang08 -0.2992 0.2008 -0.2524 0.2976Xydeas00 0.6949 0.1655 0.6198 0.2452
P.W. Wang08 0.6356 0.1634 0.5771 0.1761Zheng07 0.4332 0.2317 0.4614 0.1820Piella03 0.3798 0.2409 0.4131 0.1725
H. Chen07 -0.5544 0.4089 -0.5611 0.4640Y. Chen09 0.2667 0.4830 0.3274 0.4628
Introduction Subjective Experiment
Results Conclusion
Performance of Objective IQA Models
• State-of-the-art IQA models do not provide adequate predictions
• Entropy-based models: very poor performance
entropy too limited at capturing image characteristics
• Models based on structure preservation: most promising
but they are limited in capturing spatial consistency
Introduction Subjective Experiment
Results Conclusion
Performance of Objective IQA Models: Observations
Introduction Subjective Experiment
Results Conclusion
Conclusions
• A new subject-rated MEF image database
• Evaluation of MEF algorithms
• Evaluation of objective IQA model for image fusion
Summary
19 / 20
Future Work: Design of MEF and objective IQA algorithms
• No existing IQA model works adequately
• Need balance between structure-preservation and spatial consistency