Image Quality Measurements:Necessity, Numbers

and '....nesses'.Ralph Jacobson

Emeritus Professor of Imaging ScienceUniversity of Westminster

Digital Futures 2009: Image Physics & PsychophysicsNovember 3rd, The Institute of Physics, London

Overview• Need for measurement• Quality defined• Historical perspective and transference• Imaging with numbers but without too many

numbers!• The ‘Nesses’• Physical & Psychophysical measures

psychophysics/psychometricsrelationships with observations (HVS),

• Conclusions

Why Measure?To provide a quantitative basis for the useand comparison of products and devices

To provide a means of improving quality

To provide a means for modelling systemsand deciding what aspects need improvingTo understand the systems

What is Quality?Degree of excellence, relative nature or, kind or,character. (OED)

All those features of product (or service) which arerequired by the customer. (ISO 9000)

The integrated set of perceptions of the overalldegree of excellence of an image.

(Engeldrum, 2000)Others to be defined later

The ‘Nesses’Proposed by Engledrum as a characteristics ofimages that we sense (see).

Perceptual attributes :Brightness HuenessChromaness LightnessColorfulness NaturalnessContrastness SharpnessFineness (detail) TexturenessGraininess (noisiness) Usefulness

Prototyping

PsychologicalEffects

Brain

Brain

Memory

PhysicalEffects

PhysiologicalEffects

Colour ofGrass

InPicture

MIND

UsualColour of

Grass

COLOUR PICTURE

ORIGINAL OBJECTS

Hunt, 1967

PrototypingSubjective judgement made by a mentalcomparison of an external image withimage impressions stored andremembered more or less distinctly by theobserver, who allows for loss of detail inareas too small to be resolved by the eye.

Schade, 1975

Acceptability/Perceptibility

Fidelity (Perceptibility):Visibility of a factor, orDiscrimination between images

Klein, 1993, Farrell 1996

Quality (Acceptability):Preference of one image over an other, ordegree to which a factor is bothersome

Acceptability/Perceptibility

0 4 8

16 6432

20R

-20R

30R

-30R

0

10R

-10R

Acceptability/Perceptibility

Naturalness

The degree of apparent match betweenthe reproduced image and the internalreferences, e.g. memory prototypes

Endrikhovski, 2002

Naturalness

OriginalColoursshifted toyellow -Sunset

Coloursshifted toblue -Moonscape

Coloursshifted topurple -Unnatural

Endrikhovski, 2002

Visuo-Cognitive Processing

Janssen, 2001

Image perception

Internalrepresentation interpretation

Interpretedscene

semanticprocessingtask response

memoryrepresentation

UsefulnessRequires modification according toapplication – usefulness or fitness forpurpose:The degree of apparent suitability of thereproduced image to satisfy thecorresponding task.

Usefulness

Examples of usefulness criteria might bethe ability to:

• resolve a defined detail• discriminate one area from another• successfully diagnose or interpret

UsefulnessDental X-ray

Powell, May 2004

Usefulness

Image Quality Approaches

virtual reality

Usefulness

Fidelity

Naturalness

holiday pictures

fine art

Mars imagesmedical images

advertisement

Endrikhovski, 2002

Problems

Image quality has no single unique definition, yetas observers, we are able to decide almostinstantly whether a particular image is of good orpoor quality but for us to quantify how good animage is, and the scale of quality is far moredifficult.

Multidimensional in character

Additional ‘nesses’ for digital imaging

Digital ‘Nesses’ ArtefactsEffect Possible CausesContouring poor bit-depthBlocking (Gibbs effects) compressionJaggies inappropriate pixel sizeRinging sharpening, compressionAliasing samplingStreaking pixel-to-pixel non uniformityPatterning poor spatial resolution,

ditheringColour misregistration images from different

channels not geometricallyidentical

Measurement

• Physical measures• Psychophysical aspects• Inter – relationships

MeasurementsAttribute Physical MeasureColour Spectral data, Chromaticities,

Colour spaces etc.Tone (contrast) Gamma, Density, PV, Characteristic

Curve, Tone Reproduction Curve,Density Histogram, OECF

Resolution (detail) Resolving Power, l/mm, dpi, ppiSharpness (edges) Acutance, PSF, OTF, LSF, MTF, SFRNoise Granularity, Standard Deviation

(graininess, electronic noise)Noise-Power (Wiener) Spectrum

Information Entropy, Information Capacity

Physical MeasurementsBeginnings

Fundamental PhysicalMeasures

(1975)

(1974)

Opto-ElectronicConversion Function (OECF)

050

100150200250

0 0.5 1.0 1.5 2.0 2.5 3.0

Log Relative Exposure

Mean

Outp

ut Le

vel

Methods for MeasuringOpto-electronic ConversionFunctions (OECF): ISO 14524The standard describes methods formeasuring and reporting the relationshipbetween the input scene log luminancevalues and the digital output levels for adigital camera.

Determination of ISO SpeedISO 12232

• ISO speed of a digitalcamera attempt to matchISO rating of film camerasystems

• provides a method formeasuring and reportingISO speed metrics thatcorrelate with imagequality

(Saturation and noise based)

108 109 1010 1011 1012 1013

106

105

104

103

102

101

100

SaturationLevel

EXPOSURE (Photons/sec/cm2 )

SIGN

AL (e

lectro

ns)

Signal

Photon Sho

t Noise

Read Noise

Dark Noise

108 109 1010 1011 1012 1013

106

105

104

103

102

101

100108 109 1010 1011 1012 1013

106

105

104

103

102

101

100

2 )

Signal

Photon Sho

t Noise

Read Noise

Dark Noise

Bestexposure

Muammar, 2008

ISO Test Chart ISO 15739

18% reflectance for calculating noise

Patches used for calculatingincremental gain

Black referencefor calculatingDSC dynamic

range

OECF

Measurements of VisualResolution & SFR: ISO 12233

0.00

0.20

0.40

0.60

0.80

1.00

0 1000 2000 3000 4000 5000Frequency [LW/PH]

Spati

al Fr

eque

ncy R

espo

nse

SFRhorizontaSFRverticalAverage

Aliasing Region

SFR

Visual

Universal Test Target

Image Engineering Dietmar Wueller , 1 Sep 2009

Psychophysics

Science of investigations of the quantitativerelationships between physical events andthe corresponding psychological events.

i.e., quantitative relationships betweenstimuli and responses

PsychometricsProvide quantification of qualitative attributes

2000 2002

Scaling Methods• Pair Comparisons• Categorical Methods• Rank Order• Interval Judgment• Ratio Judgment• Magnitude Estimation• Pass/fail

Part 1: Overview of psychophysical elementsDescribes how the standard could be extendedto include other psychometric techniques.

Part 2: Triplet comparison method Method forsubjective image quality assessment

Part 3: Quality Ruler Method Describes amethod for generating quality rulers varying insharpness.

Psychophysical ExperimentalMethod to Estimate Image

Quality: ISO 20462

Just Noticeable Difference (JND)Attribute JNDmeasure of the detectability of appearancevariations, corresponding to a stimulus differencethat leads to a 75:25 proportion of responses in apaired comparisonQuality JNDmeasure of the significance or importance ofquality variations, corresponding to a stimulusdifference that leads to a 75:25 proportion ofresponses in a paired comparison task in whichmultivariate stimuli pairs are assessed in terms ofoverall image quality

Just Noticeable Difference (JND)

JND

Engledrum, 2000

Metrics

Visual Image Quality Metric (VIQM):Single numbers (figures of merit) derivedfrom physical measurements for the systemand the eye which relate to perceptions ofimage quality

Image Quality Metric (IQM):Single numbers (figures of merit) derivedfrom physical measurements for thesystem which relate to perceptions ofimage quality

Image Quality Metrics

HVS +IQC

HVSOutputdevice

Fidelity

QualityDistortion

ImageData

Distortion(IQMs)

(VIQMs)

Multivariate Metric (Minkowski)

IQM given by:

[0.413(sharpness)-3.14 + 0.422 (10-graininess)-3.4]-1/3.4 – 0.532

Bartleson, 1982

Minkowski Metric

Engledrum, 2002

Image Quality Metric (IQM)Colour Difference:

CIE 1976 CIELAB Colour Difference:

∆∆∆∆ *)(*)(*)(* 222 baLEab

Later colour difference formulae (CIE94,CIEDE2000) do include viewing conditions

• Does not take viewing conditions into account

Variable Exponent MinkowskiMetric

∆Q = ( Σ ∆Q ε )1/ε

In JNDs of quality change

Keelan, 2000

Visual Image Quality Metrics(VIQMs)

VIQM

Contrast(Gamma, Tone) ColourVisualsystem

MTF(SharpnessResolution)

Noise

Generally are based on some form of signal to noise ratio

VIQM Approach

VIQM

020406080

100

0 20 40 60 80 100

JND

PsychophysicalObservations

VisualsystemNoise

Contrast(Gamma, Tone)

MTF(SharpnessResolution)

Sharpness and NoiseMultivariate IQ

012345678

0 1 2 3 4 5 6 7 8

r 2 = 0.944

Perceived Quality

Calcu

lated

IQ

Stone, Jacobson & Attridge 1994

du](u)N+(u)MN(u)(u)M(u)MS(u)+[1=SNR

eye2eye

2eye

2sysln∫

Higgins, 1977

Visual Image Quality MetricBarten’s Square Root Integral (SQRI):

)d(ln)()(

)2ln(1

tu

maxu

minuumuMJ ∫=

Barten, 1990

MTFdisplaysystem

modulationthreshold function

of the eye

Visual Image Quality Metric

udu

)()()()()(

2lnSQRI 225.0

0 22

21 max kuNuMuNk

uMuSk ueyeeye

eyen +

+= ∫

)()........()()({ 2210 uMuMuSuS n= }

(Barten’s SQRIn reformulated by Töpfer & Jacobson, 1993)

Square Root Integral with Noise (SQRIn)

VIQM (SQRIn ) and Sharpness

Töpfer & Jacobson,1993

r 2 = 0.9950

20

40

60

80

100

0 20 40 60 80 100Measured SQRIn (JND)

AMTA

(JND

)

Measured

Predicted

udu

)()()()()(

2lnSQRI 225.0

0 22

21 max kuNuMuNk

uMuSk ueyeeye

eyen +

+= ∫ udu)()()( )()(2lnSQRI 2

25.0

0 22

21 max kuNuMuNk

uMuSk ueyeeye

eyen +

+= ∫

Examples of VIQMsMTFs, Signals, Sharpness, Noise etc: Schade, 1950+System Modulation Transfer Acutance (SMTA) : Crane, 1964Signal-to-Noise Ratio (SNR) : Nelson, 1973Modulation Transfer Acutance (AMTA): Crane, 1983(Quality: graininess and sharpness (Qg/s) : Bartleson, 1982Square-root Integral (SQRI) : Barten, 1990+Perceived Information Capacity (PIC) : Töpfer & Jacobson, 1993Visible Differences Predictor (VDP) : Daly, 1992Visual information and processing: Janssen & Blommaert, 1997+Effective Pictorial Information Capacity (EPIC) : Jenkin et al., 2005Colour Reproduction Index (CRI) : Pointer, 1986+Colour Difference: CIE,1976, CIECAM97s :CIE, 1997, CIEDE2000R-LAB: Fairchild and Berns, 1993S-CIELAB:Zhang et al., 1996Cognition: Usefulness, Naturalness: Endrikhovski et al, 1999+CSF/CIEDE2000, Colour Image Difference Metric:

Johnson and Fairchild , 2002

Metrics Applied to DigitalImages

Mean-Square-Error (MSE, RMSE)Mean-Square-Error after Non-Linearity(MSENL)Mannos-Sakrison (MANNOS)Logarithmic Image Processing (LIP)Distortion Contrast (DCON)Bit Rate (BITS)Mean Intensity (MI)Spectrum Slope (SS)Spectrum Slope over Mean Intensity (SS/MI)Local Contrast (LCON)

Root Mean Square Error (RMSE)

• Cognitive aspects not considered

RMSE = −=

−∑

=

−∑1

0

1 2

0

1

XYf x y f x y

y

Y

x

X( , ) '( , )

( f, f’ are original and changed images at spatial locations x,y, X and Y are total number of horizontal and vertical pixels)

• No account of visual significance

• Output device not considered

Effective PictorialInformation Capacity (EPIC)

1.Determine total MTF of system components and the eye2. Inverse Fourier transform to find effective pixel size3. Determine total noise using effective pixel size as

aperture, including scene dependent noise for distorted(e.g. blurred) images

4.Find Information capacity ( I = n log2m ) in bits5. Convert to bits/steradian from image size and viewing

distanceJenkin, Triantaphillidou and Richardson, 2007

Effective PictorialInformation Capacity (EPIC)

R2 = 0.94

050

100150200250300350400450

-1.5 -1 -0.5 0 0.5 1 1.5 2 2.5Subjective Quality Rating

EPIC

(kBy

tes/

ster

adia

n)

Jenkin, Triantaphillidou and Richardson, 2007

Specific approaches

Standards : e.g. NCITS W1.1Image Quality for Printer Systems –

Considerations of:Text and Line quality, Micro uniformity, Macrouniformity, Gloss/Gloss uniformity,Color rendition, Effective tone levels,Effective resolution in pictorials, Adjacency

VIQM Approaches

VIQM

Psychophysicalobservations

Variable ExponentMinkowski Metrics

(C)VDP basedapproaches

Cognitiveapproaches

Visio-cognitive

Information

CIE + spatial∆E approaches

Physicalmeasure,

MTF basedapproaches

Industry ledStandards

approaches

InformationCapacity

approaches

Colour Principles andMeasurement

Colour Reproduction Index(CRI)

• For determinations of absolute colourappearance

• Based on Hunt model for colour vision• Includes viewing conditions• Excludes spatial aspects

Pointer, 1986

Image Quality Index

For prediction of colourreproduction perceived asnatural, unnatural, pleasant orunpleasant for an averageobserver.

Endrikhovski, 1999

Original Processed

Enhancing Perceived Quality

Endrikhovski, 2002

S-CIELAB

ColourImage

Lum

R/G

B/Y

XYZ S-CIELAB

Spatial filter

Zhang & Wandell, 1997

CIECAM02

Pointer 2009

i-CAMTwo input images are given:

an original and a reproduction

The input images aretransformed into an opponentcolour space

The opponent channels arefiltered using contrast sensitivity

functions which are adaptedbased on the spatial

information in the image. Thefiltering decreases information

that is not visible and increasesinformation that is most visible

Models of local attention andlocal contrast are applied to thefiltered images

The filtered images are thenconverted into CIELAB

coordinates and a Pixel-byPixel colour difference

calculated

Johnson and Fairchild, 2002

iCAM

Orfanidoua, Triantaphillidou and Allen, 2008

The ‘…..ishes’More than 50 VIQMs have been proposedSignal to noise will always give goodcorrelations!Use of charts in VIQM measurementsObserversQuestion asked of ObserversStandard observerScenes, Scene Dependency, ROI(salience)Spatial and colour approachesTraditional to digital transfers

Contrast Sensitivity of theHuman Eye

Barten, 1992

00.20.40.60.81.0

0 5 10 15 20 25 30 35 40Spatial frequency (cycles/degree)

Sens

itivity

16.4o, 34 cd/m2

measured values

The Standard ObserverTC1-60 of Division 1 of the CIE terms ofreference:1) To specify a baseline achromatic CSF withits reference conditions and referenceobserver.2) To specify CSF extensions based ondiscrimination thresholds, as well aschromatic CSFs for both detection anddiscrimination.

Chairman: Eugenio Martinez-Urigas

Scene Dependency andCompression

Non-compressed JPEG 60:1 JPEG2000 60:1

S. Triantaphillidou, E. Allen, R.E. Jacobson and G.G. Attridge, 2002

Standard Images

Lena

ISO 12640-3: 20078 natural scenes

Importance of Lightness inImage Quality

Original

Lightnesssharp

Huesharp

Saturationsharp

Allunsharp

Importance of Lightness inImage Quality

Original ScrambledforHUE

Scrambledfor

CHROMAScrambled

forLIGHTNESS

Endrikhovski, 2002

Digital Transfer Difficulties

• Artefacts• Anisotropy• Non-linearity• Non-stationary

Digital systems are difficult to deal withusing conventional mathematical processes

• Some success in relating complexphysical measures to perceptions ofimage quality

• Provide modelling approach• Require extensive validation• Research & development tool• Move away from VIQM single number

approach to determining metrics by processsteps (e.g.EPIC, S-CIELAB, i-CAM)

Conclusions VIQMs

• All approaches work and are applied!• Provide useful data for modelling system

changes, benchmarking andimprovements

• Lead to determination of fundamentalparameters

• Used by all manufacturers and systemdevelopers

• Are providing new insights and impetus inperception and measurement of imagequality

General Conclusions