PERCEPTUAL EVALUATION OF MULTI-EXPOSUREIMAGE FUSION ALGORITHMS

Kai Zeng, Kede Ma, Rania Hassen and Zhou Wang

Department of Electrical and Computer Engineering University of Waterloo

September 2014

Introduction Subjective Experiment

Results Conclusion

Multi-Exposure Fusion (MEF)

... ...

2 / 18

Introduction Subjective Experiment

Results Conclusion

MEF vs Tone Mapping

HDRreconstruction Tone mapping

MEF

Alternative Methods High dynamic range (HDR) image reconstruction followed by tone mapping MEF – bypassing the intermediate HDR stage

3 / 18

Introduction Subjective Experiment

Results Conclusion

Existing MEF algorithms

Cues used in existing MEF methods

Local energy and correlation (in Laplacian pyramid)

[Burt1993] Entropy [Goshtasby2005]

Contrast, color saturation and luminance exposure

[Mertens2007] Edge information (using bilateral filter)

[Raman2009]

Visual contrast and scene gradient [Song2012]

Visibility and consistency (from gradient information)

[Zhang2012] Local contrast, brightness and color dissimilarity

[Li2012]

Pixel saliency and spatial consistency [Li2013]

......

Question:Which method produces the best quality image?

4 / 18

IntroductionSubjective Experiment

Results Conclusions

Existing objective image quality assessment (IQA) models for image fusion

Existing IQA models for image fusion

Information theoretical models [Hossny, 2008, Cvejic, 2006, Wang, 2008]

Image feature based models [Xydeas, 2000, WangPW, 2008, Zheng, 2007]

Image structural similarity based models [Piella, 2003]

Human perception based models [Chen, 2007, Chen, 2009]

Question:

Which model better predicts perceptual quality?

5 / 20

Image Database

17 natural scenes of images with multi-exposure levels

8 image fusion algorithms

Simple methods: local and global energy weighted linear combination Advanced models: [Raman09, Gu12, ShutaoLi12, Shutaoli13, Li12, Mertens07]

Subjective Test

Absolute category rating from 1 to 10

No reference images are shown

25 naïve observers (15 male and 10 female, between 22 and 30)

Each session limited to within 30 mins

6 / 18

Introduction Subjective Experiment

Results Conclusion

Subjective Experiment Mapping

Introduction Subjective Experiment

Results Conclusion

Reference Images

7 / 18

Introduction Subjective Experiment

Results Conclusion

Images Examples

(a1) Under Exposure (a2) Normal Exposure (a3) Over Exposure

(b) Global Energy Weighted (c) Gu12 (d) Local Energy Weighted

8 / 18

Introduction Subjective Experiment

Results Conclusion

Images Examples

(e) Li12 (f) Li13 (g) Mertens07

(h) Raman09 (i) ShutaoLi12

9 / 18

Introduction Subjective Experiment

Results Conclusion

Subjective Data Analysis

Table: Correlation between individual and average subject scores

10 / 18

Subject PLCC SRCC Subject PLCC SRCC1 0.8743 0.8631 13 0.8411 0.79892 0.8245 0.7984 14 0.8781 0.87433 0.7102 0.6735 15 0.8988 0.89244 0.8093 0.8182 16 0.7413 0.73135 0.6785 0.6649 17 0.7347 0.64886 0.6544 0.6567 18 0.7797 0.74867 0.8198 0.8030 19 0.6732 0.68148 0.8951 0.8849 20 0.7854 0.76439 0.7961 0.7835 21 0.6045 0.5638

10 0.6924 0.6826 22 0.6213 0.612111 0.8298 0.8275 23 0.7976 0.755812 0.6145 0.5795 Average 0.763

30.7438

Introduction Subjective Experiment

Results Conclusion

Subjective Data Analysis: Pearson Linear CC

Figure: Mean and standard deviation of PLCC between individual subject and average subject for each image set. The rightmost column gives the average performance of all subjects.

11 / 18

Introduction Subjective Experiment

Results Conclusion

Subjective Data Analysis: Spearman ROCC

Figure: Mean and standard deviation of SRCC between individual subject and average subject for each image set. The rightmost column gives the average performance of all subjects.

12 / 18

Considerable consistency between subjects

Behavior of individual subjects may vary

“Average Subject”: a useful concept

Baseline to observe behaviors of individual subject

Baseline to evaluate behaviors of objective models

Introduction Subjective Experiment

Results Conclusion

Subjective Data Analysis: Observations

Introduction Subjective Experiment

Results Conclusion

Performance of MEF Algorithms

Figure: Mean and standard deviation of subjective rankings of individual fuser acrossall image sets. 13 /

18

• Considerable subject agreement on individual MEF but performance difference between MEFs could be small

• No single method performs best on all image sets

• Best performance [Mertens07] (on average)

Second best [Li12]: a detail-enhanced version of [Mertens07]

• Global energy weighting significantly better than local energy weighting

• Some “advanced” MEFs show no advantage over simple methods

Introduction Subjective Experiment

Results Conclusion

Performance of MEF Algorithms: Observations

Table: Performance evaluation of objective IQA models

16 / 18

IQA model PLCC SRCCµ σ µ σ

Hossny08 -0.2939 0.2054 -0.2784 0.2803Cvejic06 0.0604 0.4311 0.0590 0.4968

Q. Wang08 -0.2992 0.2008 -0.2524 0.2976Xydeas00 0.6949 0.1655 0.6198 0.2452

P.W. Wang08 0.6356 0.1634 0.5771 0.1761Zheng07 0.4332 0.2317 0.4614 0.1820Piella03 0.3798 0.2409 0.4131 0.1725

H. Chen07 -0.5544 0.4089 -0.5611 0.4640Y. Chen09 0.2667 0.4830 0.3274 0.4628

Introduction Subjective Experiment

Results Conclusion

Performance of Objective IQA Models

Introduction Subjective Experiment

Results Conclusion

Performance of Objective IQA Models

17 / 18

• State-of-the-art IQA models do not provide adequate predictions

• Entropy-based models: very poor performance

entropy too limited at capturing image characteristics

• Models based on structure preservation: most promising

but they are limited in capturing spatial consistency

Introduction Subjective Experiment

Results Conclusion

Performance of Objective IQA Models: Observations

Introduction Subjective Experiment

Results Conclusion

Conclusions

• A new subject-rated MEF image database

• Evaluation of MEF algorithms

• Evaluation of objective IQA model for image fusion

Summary

19 / 20

Future Work: Design of MEF and objective IQA algorithms

• No existing IQA model works adequately

• Need balance between structure-preservation and spatial consistency

Introduction Subjective Experiment

Results Conclusion

Thank you

20 / 18