Most Apparent DistortionA dual strategy for full reference image quality assessment

Most Apparent DistortionA dual strategy for full reference image quality assessment

Eric Larson Cuong Vu, Damon Chandler Image Coding and Analysis Lab (ICAN)

Motivating example…Motivating example…

Motivating example…Motivating example…

Motivating example…Motivating example…

Most Apparent Distortion

MADMost Apparent Distortion

MAD

OutlineOutline

Introduction/Motivation Current Methods An example of two strategies

Methodology of MAD Detection modeling Appearance modeling Adaptation

Results Conclusion

MotivationMethods

Results

Three Types of IndicesThree Types of Indices

Mathematical efficiency Peak Signal-to-Noise Ratio (PSNR), Mean-Squared

Error (MSE)

Low level properties of Human Visual System (HVS) Visual Difference Predictor (VDP)[Daly,1992], Perceptual

Structure[Carnec, et al., 2003] ,Visual Signal-to-Noise Ratio (VSNR)[Chandler,Hemami,2007], Wavelet-based quality assessment (WQA)[Ninnassi, et al. 2008] ,

MotivationMethods

Results

Three Types of IndicesThree Types of Indices

Mathematical efficiency Peak Signal-to-Noise Ratio (PSNR), Mean-Squared

Error (MSE)

Low level properties of Human Visual System (HVS) Visual Difference Predictor (VDP)[Daly,1992], Perceptual

Structure[Carnec, et al., 2003] ,Visual Signal-to-Noise Ratio (VSNR)[Chandler,Hemami,2007], Wavelet-based quality assessment (WQA)[Ninnassi, et al. 2008] ,

Overarching principles of human vision Structural SIMilarity (SSIM)[Wang,2004], Visual Information

Fidelity (VIF)[Sheikh,2006], VSNR, Perceptual Structure

MotivationMethods

Results

Three Types of IndicesThree Types of Indices

Mathematical efficiency Peak Signal-to-Noise Ratio (PSNR), Mean-Squared

Error (MSE)

Low level properties of Human Visual System (HVS) Visual Difference Predictor (VDP)[Daly,1992], Perceptual

Structure[Carnec, et al., 2003] ,Visual Signal-to-Noise Ratio (VSNR)[Chandler,Hemami,2007], Wavelet-based quality assessment (WQA)[Ninnassi, et al. 2008] ,

Overarching principles of human vision Structural SIMilarity (SSIM)[Wang,2004], Visual Information

Fidelity (VIF)[Sheikh,2006], VSNR, Perceptual Structure

Single most relevant strategy is modeled

MotivationMethods

Results

A Task for High QualityA Task for High Quality

Can we see the distortion? How intense is the distortion?Motivatio

nMethods

Results

A Task for High QualityA Task for High Quality

Can we see the distortion? How intense is the distortion?Motivatio

nMethods

Results

Original JPEG Noise

A Task for High QualityA Task for High Quality

Can we see the distortion? How intense is the distortion?Motivatio

nMethods

Results

17.7 23.50.0

Original JPEG Noise

A Task for Low QualityA Task for Low Quality

How much does the image look like the original, given that there are so many visible distortions?

MotivationMethods

Results

A Task for Low QualityA Task for Low Quality

How much does the image look like the original, given that there are so many visible distortions?

MotivationMethods

Results

[9]

A Task for Low QualityA Task for Low Quality

How much does the image look like the original, given that there are so many visible distortions?

MotivationMethods

Results

74.64

59.0

67.09

82.74

[9]

A Task for Low QualityA Task for Low Quality

How much does the image look like the original, given that there are so many visible distortions?

MotivationMethods

Results

Motivation SummaryMotivation Summary

Approximate high quality task: Visibility Intensity

Approximate low quality task: Preserve content (appearance)

Adaptively change strategies

MotivationMethods

Results

MethodologyHigh Quality

A Strategy for High QualityA Strategy for High Quality Conversion to perceived

brightness Pixel to luminance luminance to L*

MotivationMethods

Results

A Strategy for High QualityA Strategy for High Quality Conversion to perceived

brightness Pixel to luminance luminance to L*

Contrast sensitivity

MotivationMethods

Results

)*,(' CSFLfilterI refref

)*,(' CSFLfilterI errerr

A Strategy for High QualityA Strategy for High Quality Contrast and luminance

maskingMotivationMethods

Results

)(

)()(

p

ppC

ref

refref

A Strategy for High QualityA Strategy for High Quality Spatial frequency and

luminance maskingMotivationMethods

Results

)(

)()(

p

ppC

ref

refref

)(),(),(),(min)( 22211211 ppppp refrefrefrefref

p11

p22

p12

p21

A Strategy for High QualityA Strategy for High Quality Contrast and luminance

maskingMotivationMethods

Results

)(

)()(

p

ppC

ref

refref

otherwise ,0

9.0)( if , )(

)()( ref p

p

ppC ref

err

diff

A Strategy for High QualityA Strategy for High Quality Contrast and luminance

maskingMotivationMethods

Results

)(

)()(

p

ppC

ref

refref

otherwise ,0

9.0)( if , )(

)()( ref p

p

ppC ref

err

diff

otherwise ,0

)()( if 1, )( 4

3 pCpCpVisib referr

A Strategy for High QualityA Strategy for High Quality Contrast and luminance

maskingMotivationMethods

Results

)(

)()(

p

ppC

ref

refref

otherwise ,0

9.0)( if , )(

)()( ref p

p

ppC ref

err

diff

otherwise ,0

)()( if 1, )( 4

3 pCpCpVisib referr

pNji

err jiIpLMSE,

2

2),('

16

1)(

A Strategy for High QualityA Strategy for High Quality Contrast and luminance

maskingMotivationMethods

Results

)(

)()(

p

ppC

ref

refref

NPp

high pVisibpLMSEN

PD 2)()(1

otherwise ,0

9.0)( if , )(

)()( ref p

p

ppC ref

err

diff

otherwise ,0

)()( if 1, )( 4

3 pCpCpVisib referr

pNji

err jiIpLMSE,

2

2),('

16

1)(

A Strategy for High QualityA Strategy for High Quality

MotivationMethods

Results

Visibility Map

Visibility Map

LMSEMap

LMSEMap

CombineMaps

CombineMaps

Collapse with two-

norm

Collapse with two-

norm

Read Images

Read Images

A Strategy for High QualityA Strategy for High Quality

MotivationMethods

Results

Visibility Map

Visibility Map

LMSEMap

LMSEMap

CombineMaps

CombineMaps

Collapse with two-

norm

Collapse with two-

norm

Read Images

Read Images

A Strategy for High QualityA Strategy for High Quality

MotivationMethods

Results

Visibility Map

Visibility Map

LMSEMap

LMSEMap

CombineMaps

CombineMaps

Collapse with two-

norm

Collapse with two-

norm

Read Images

Read Images

A Strategy for High QualityA Strategy for High Quality

MotivationMethods

Results

Visibility Map

Visibility Map

LMSEMap

LMSEMap

CombineMaps

CombineMaps

Collapse with two-

norm

Collapse with two-

norm

Read Images

Read Images

A Strategy for High QualityA Strategy for High Quality

MotivationMethods

Results

Visibility Map

Visibility Map

LMSEMap

LMSEMap

CombineMaps

CombineMaps

Collapse with two-

norm

Collapse with two-

norm

Read Images

Read Images

PDhigh

MethodologyLow Quality

A Strategy for Low QualityA Strategy for Low Quality Defining appearance:

Biological motivation: the log-Gabor filter bank[Field 1987, Kovesi]Motivatio

nMethods

Results

A Strategy for Low QualityA Strategy for Low Quality Defining appearance:

Biological motivation: the log-Gabor filter bank[Field 1987, Kovesi]

Five Scales Four Orientations

MotivationMethods

Results

2

2

2

2

2

)(exp

)/log(2

)/log(exp),(

os

o

f

fffG

A Strategy for Low QualityA Strategy for Low Quality Gather appearance based upon

statistics of Gso

Statistics have been used to model Animal Camouflage [Larson, Chandler 2007]

Texture Appearance[Kingdom, et al. 2003]

Variance, Skewness, and Kurtosis

MotivationMethods

Results

A Strategy for Low QualityA Strategy for Low Quality Gather appearance based upon

statistics of Gso

where ws = [0.5, 0.75, 1, 5, 6]

MotivationMethods

Results

pp OoSs dst

sorefso

dstso

refso

dstso

refso

swp,

||

||2

||

)(

A Strategy for Low QualityA Strategy for Low Quality Gather appearance based upon

statistics of Gso

where ws = [0.5, 0.75, 1, 5, 6]

MotivationMethods

Results

NPp

low pN

PD 2)(1

pp OoSs

dstso

refso

dstso

refso

dstso

refsoswp

,

|]|||2|[|)(

A Strategy for Low QualityA Strategy for Low Quality Gather appearance based upon

statistics of Gmag,so

where ws = [1, 2, 6, 10, 12]

MotivationMethods

Results

pp OoSs

dstso

dstso

dstso

refso

refso

refsoswp

,

)]()[()( 22

NPp

low pN

Q 21)(

Gabor FilteringGabor

Filtering

PatchStatistics

PatchStatistics

Collapse with two-

norm

Collapse with two-

norm

Read Images

Read Images

A Strategy for Low QualityA Strategy for Low Quality Gather appearance based upon

statistics of Gmag,so

where ws = [1, 2, 6, 10, 12]

MotivationMethods

Results

pp OoSs

dstso

dstso

dstso

refso

refso

refsoswp

,

)]()[()( 22

NPp

low pN

Q 21)(

Gabor FilteringGabor

Filtering

PatchStatistics

PatchStatistics

Collapse with two-

norm

Collapse with two-

norm

Read Images

Read Images

A Strategy for Low QualityA Strategy for Low Quality Gather appearance based upon

statistics of Gmag,so

where ws = [1, 2, 6, 10, 12]

MotivationMethods

Results

pp OoSs

dstso

dstso

dstso

refso

refso

refsoswp

,

)]()[()( 22

NPp

low pN

Q 21)(

Gabor FilteringGabor

Filtering

PatchStatistics

PatchStatistics

Collapse with two-

norm

Collapse with two-

norm

Read Images

Read Images

A Strategy for Low QualityA Strategy for Low Quality Gather appearance based upon

statistics of Gmag,so

where ws = [1, 2, 6, 10, 12]

MotivationMethods

Results

pp OoSs

dstso

dstso

dstso

refso

refso

refsoswp

,

)]()[()( 22

NPp

low pN

Q 21)(

Gabor FilteringGabor

Filtering

PatchStatistics

PatchStatistics

Collapse with two-

norm

Collapse with two-

norm

Read Images

Read Images

A Strategy for Low QualityA Strategy for Low Quality Gather appearance based upon

statistics of Gmag,so

where ws = [1, 2, 6, 10, 12]

MotivationMethods

Results

pp OoSs

dstso

dstso

dstso

refso

refso

refsoswp

,

)]()[()( 22

NPp

low pN

Q 21)(

Gabor FilteringGabor

Filtering

PatchStatistics

PatchStatistics

Collapse with two-

norm

Collapse with two-

norm

Read Images

Read Images

PDlow

AdaptationAdaptation

We can adaptively model their interaction based upon PDhigh

The final index is a weighted geometric mean

MotivationMethods

Results

2)(1

1

1

highPD

)1()()( lowhigh PDPDMAD

Results

ResultsResults

LIVE[9] Quality Database: 779 Distorted Images, 29 Original 29 Observers JPEG, JPEG2000, Blurring, AWGN, and

simulated packet loss

CSIQ (Categorical Subjective Image Quality) Database: 10 original images, 300 distorted

versions 10 observers Blur, contrast, AWGN, JPEG, JPEG2000,

APGN (1/f noise)

MotivationMethods

Results

ResultsResults

LIVE Performance, All images

MotivationMethods

Results

ALL PSNR SSIM VSNR VIF MAD

CC 0.8707 0.9378 0.9233 0.9595 0.9695

SROCC 0.8763 0.9473 0.9278 0.9633 0.9703

Rout 68.16% 59.18% 58.79% 54.56% 42.40%

ResultsResults

LIVE Performance, All images

MotivationMethods

Results

ALL PSNR SSIM VSNR VIF MAD

CC 0.8707 0.9378 0.9233 0.9595 0.9695

SROCC 0.8763 0.9473 0.9278 0.9633 0.9703

Rout 68.16% 59.18% 58.79% 54.56% 42.40%

Statistical SignificanceStatistical Significance

LIVE Database, 99% confidence

1 = better, 0 = same, -1 = worse

MotivationMethods

Results

PSNR SSIM VSNR VIF MADPSNR 0 - - - -SSIM 1 0 - - -VSNR 1 -1 0 - -

VIF 1 1 1 0 -

MAD 1 1 1 1 0

ResultsResults

CSIQ Overall Performance

MotivationMethods

Results

ALL PSNR SSIM VSNR VIF MAD

CC 0.8455 0.8893 0.8472 0.9079 0.9487

SROCC

0.8428 0.9019 0.8577 0.9063 0.9469

Rout 35.6% 30.5% 28.2% 33.9% 23.5%

ResultsResults

CSIQ Overall Performance

MotivationMethods

Results

Logistic MAD

CC = 0.9487

DM

OS

Statistical SignificanceStatistical Significance

CSIQ Database, 99% Confidence

1 = better, 0 = same, -1 = worse

MotivationMethods

Results

PSNR SSIM VSNR VIF MADPSNR 0 - - - -SSIM 1 0 - - -VSNR 0 -1 0 - -

VIF 1 0 1 0 -

MAD 1 1 1 1 0

ConclusionConclusion

Quality prediction algorithms can enhance performance by adaptively changing strategy

MAD performs significantly better than any other existing index on two databases

MAD shows promise in generalizing to a range of distortions

Multiple strategies

MotivationMethods

Results

Thank YouThank You

Questions?

MotivationMethods

Results

Thank YouThank You

MotivationMethods

Results

ReferencesReferences1. B. Girod, What’s worng with the mean squared error?, pp207-240. MIT Press, 2nd ed., 19932. T. Chen, Invited Lecture, Carnegie Mellon University, 2008 IEEE Southwest Symposium on Image Analysis and

Interpretation. 3. S. Daly, “Visible differences predictor: an algorithm for the assessment of image fidelity,” in Proc. SPIE Vol. 1666, p. 2-

15, Human Vision, Visual Processing, and Digital Display III, Bernice E. Rogowitz; Ed. (B. E. Rogowitz, ed.), vol. 1666 of Presented at the Society of Photo-Optical Instrumentation Engineers (SPIE) Conference, pp. 2–15, Aug. 1992.

4. D. Chandler and S. Hemami, “Vsnr: A wavelet based visual signal to noise ratio for natural images,” IEEE Transactions on Image Processing, vol. 16, pp. 2284 -2298, Sept. 2007.

5. Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” IEEE Transactions on Image Processing, vol. 13, pp. 600–612, April 2004.

6. H. R. Sheikh and A. C. Bovik, “Image information and visual quality,” IEEE Transactions on Image Processing, vol. 15, pp. 430–444, Feb. 2006.

7. E. Peli, L. E. Arend, G. M. Young, and R. B. Goldstein, “Contrast sensitivity to patch stimuli: Effects of spatial bandwidth and temporal presentation,” Spatial Vision, vol. 7, pp. 1–14, 1993.

8. G. E. Legge and J. M. Foley, “Contrast masking in human vision,” J. of Opt. Soc. Am., vol. 70, pp. 1458–1470, 1980.9. Z. W. H. R. Sheikh, A. C. Bovik, and L. K. Cormack. Image and Video Quality Assessment Research at LIVE [Online].

Available: http://live.ece.utexas.edu/research/quality/.10.B. A. Olshausen and D. J. Field, “Sparse coding with an overcomplete basis set: A strategy employed by v1?,” Vision

Research, vol. 37, pp. 3311–3325, Dec. 1997.11.P. D. Kovesi.   MATLAB and Octave Functions for Computer Vision and Image Processing. School of Computer Science &

Software Engineering, The University of Western Australia.   Available from: http://www.csse.uwa.edu.au/~pk/research/matlabfns/

12.Kingdom, F. A. A., Hayes, A. & Field, D. J. (2001) Sensitivity to contrast histogram differences in synthetic wavelet-textures. Vision Research, 41, 585-598.

13.N. P. S. D. I. A. [Online].14.“Vqeg, final report from the video quality experts group on the validation of objective models of video quality

assessment, phase ii,” August 2003 [Online]. Available: http://www.vqeg.org.15.H. Sheikh, M. Sabir, and A. Bovik, “A statistical evaluation of recent full reference image quality assessment

algorithms,” IEEE Transactions on Image Processing, vol. 15, pp. 1349–1364, Nov. 2006.16.Correlation, Wikipedia, http://en.wikipedia.org/wiki/Correlation17. Regression Analysis, Wikipedia, http://en.wikipedia.org/wiki/Regression_analysis

ResultsResults

CSIQ (Categorical Subjective Image Quality) Database

Preliminary Numbers: 4 observers 10 original images 300 distorted versions Six distortion types:

Blur, contrast, AWGN, JPEG, JPEG2000, APGN (1/f noise)

MotivationMethods

Results

Camp Two: StructureCamp Two: Structure

SSIM captures Gaussian windowed spatial statistics

Collapse quality by taking mean of map

MotivationMethods

Results

))((

))((),(

222

122

21 22

CC

CCyxSSIM

yxyx

xyyx

Camp Two: InformationCamp Two: Information VIF models mutual information by

Analyzing in wavelet domain[6]

Applying reference and distorted images to HVS model

Where CU is the principle image covariance of sub-band i,

σv is the distortion noise, σn is the Visual noise variance,

s and g are sub-band scaling constants

MotivationMethods

Results IIC 2

21

2222

1n

N

jnUji sREFI loglog)(

N

jnvnvUjji sgDSTI

1

222

222222

1IIC )(log)(log)(

subbandsi

isubbandsi

i )I(REF)I(DSTVIF

A Task for High QualityA Task for High Quality

Can we see the distortion? How intense is the distortion?Motivatio

nMethods

Results

[9]

A Task for High QualityA Task for High Quality

Can we see the distortion? How intense is the distortion?Motivatio

nMethods

Results

20.52

8.06

13.03

20.30

[9]

A Strategy for Low QualityA Strategy for Low Quality Gather appearance based upon

statistics of Gmag,so

where ws = [1, 2, 6, 10, 12]

MotivationMethods

Results

pp OoSs

dstso

dstso

dstso

refso

refso

refsoswp

,

)]||()||[()( 22

Performance MeasuresPerformance Measures

LIVE[9] Quality Database: 779 Distorted Images, 29 Original 29 Observers Over 20,000 ratings of image

fidelity, in terms of Differential Mean opinion Score (DMOS)

Five categories of distortion: JPEG, JPEG2000, Blurring, AWGN, and

simulated packet loss

MotivationMethods

Results

ResultsResults

LIVE Performance figures

MotivationMethods

Results

ResultsResults

LIVE Performance figures

MotivationMethods

Results

Statistical SignificanceStatistical Significance

In terms of Regression…

MotivationMethods

Results

[16]

Statistical SignificanceStatistical Significance

Gaussian Residuals

MotivationMethods

Results

[16]

Statistical SignificanceStatistical Significance

LIVE Database, 99% confidence

1 = better, 0 = same, -1 = worse

MotivationMethods

Results

ALL PSNR SSIM VSNR VIF MADPSNR 0 -1 -1 -1 -1SSIM 1 0 1 -1 -1VSNR 1 -1 0 -1 -1

VIF 1 1 1 0 -1MAD 1 1 1 1 0

Gaussian 1 0 1 0 1Conf. 0.007 0.257 0.001 0.081 0.001

JB Stat 11.768 2.583 20.011 4.843 246.610Skew 0.292 -0.139 0.091 0.170 -0.518Kurt 3.143 2.957 3.764 2.818 5.554

Statistical SignificanceStatistical Significance

LIVE Database, Gaussianity

MotivationMethods

Results

Statistical SignificanceStatistical Significance

LIVE Database, Gaussianity

MotivationMethods

Results

Statistical SignificanceStatistical Significance

LIVE Database, Gaussianity

MotivationMethods

Results

J.B. Statistic = 1.5J.B. Statistic = 1.5Is Gaussian with greater than Is Gaussian with greater than 95% confidence95% confidence

ResultsResults

CSIQ (Categorical Subjective Image Quality) Database

Table top randomization approach Reference image always available Distorted images viewable at one

time Placement denotes linear quality Electronic table (four monitor

array)

MotivationMethods

Results

ResultsResults

CSIQ (Categorical Subjective Image Quality) Database

Table top randomization approach Reference image always available Distorted images viewable at one

time Placement denotes quality Electronic table (four monitor

array)

MotivationMethods

Results

Statistical SignificanceStatistical Significance

CSIQ Database, 99% Confidence

1 = better, 0 = same, -1 = worse

MotivationMethods

Results

ALL PSNR SSIM VSNR VIF MADPSNR 0 -1 0 -1 -1SSIM 1 0 1 0 -1VSNR 0 -1 0 -1 -1

VIF 1 0 1 0 -1MAD 1 1 1 1 0

Gaussian 0 1 1 0 0Conf. 0.500> 0.003 0.001 0.500> 0.425

JB Stat 0.414 17.619 24.255 0.652 1.560Skew -0.053 -0.303 -0.569 0.084 0.108Kurt 3.149 4.026 3.812 3.156 3.281

ResultsResults

CSIQ Overall Performance, no contrast

MotivationMethods

Results

PSNR SSIM VSNR VIF MAD

CC 0.9178 0.9313 0.9499 0.9263 0.9637

SROCC 0.9185 0.9364 0.9478 0.9294 0.9594

RMSE 166.55 152.80 131.10 158.11 111.94

Rout 0.344 0.316 0.240 0.312 0.236

RSOD 447.9 383.3 256.6 387.0 237.1