1

Color Science, Color Management, and Color Image QualityProfessor Jon Yngve Hardeberg

The Norwegian Color Research Laboratory and Hardeberg Image andGjøvik University College, Gjøvik, Norway Color Technology

jon.hardeberg@hig.no, http://www.colorlab.no, http://color.hardeberg.com

3

IntroductionYou process images

Maybe capturing them Maybe trying to improve their visual appearanceMaybe you want to compress them without sacrificing qualityMaybe you try to recognize objects, something humans do easilyMaybe you want to visualize complex image data using color?

4

IntroductionYou need to

Understand color perceptionMake sure What You See Is What Others SeeBe able to judge image quality

5

OutlineIntroductionColor ScienceColor ManagementColor Image Quality

6

What is Color?

Well, my favorite color is red!

(t-t-t togduan?)

7

What is Color?

Well, my favorite color is red!

“Color consists of the characteristics of lightother than spatial and temporal inhomogeneities;

light being that aspect of radiant energy of which a human observer is aware through the

visual sensations which arise from the stimulation of the retina of the eye.”

[OSA 1940]

2

8

What is Color?

Color is a sensationInteraction between the physical world and our sensesPsychophysics

“Color consists of the characteristics of lightother than spatial and temporal inhomogeneities;

light being that aspect of radiant energy of which a human observer is aware through the

visual sensations which arise from the stimulation of the retina of the eye.”

[OSA 1940]

9When a tree falls in the forest, and no one is around to hear it, is there a sound?

Is the tree green if no one’s there?

"I want you to realize that there exists no color in the natural world, and no sound -nothing of this kind; no textures, no patterns, no beauty, no scent." (Sir John Eccles)

10

Color: Light, surface, eye

Wavelength, λ

Rad

ianc

e, l(

λ)

Light’s Spectral Distribution, l(λ)

Wavelength, λ

Ref

lect

ance

, r(λ

)

Surface’s Spectral Reflectance, r(λ)Eye’s Spectral Sensitivities si(λ), i =1,2,3

Wavelength, λ

Sens

itivi

ty, s

i(λ) S M

L

11

Stare at this flag for 30 seconds...

12 13

What did you see?

#1: A white screen only

#2: The Norwegian flag

#3: The Danish flag

#4: First #3 then #2

Complementary after-images

ConclusionsThe eye is quite different from a CCD sensor Our linear model is limited (but very useful)

3

14

Retinal cone distribution vs. CMOS sensor15

ColorimetryInternational LightingCommission – CIE

1931 Standard Colorimetric ObserverDetermined based on colormatching experiments

Provides a standardizedway of quantifying color

CIEXYZFoundation for consistentcommunication of color

16

Color Spaces3 different photoreceptors, L, M, S

3-output linear modelColor can be specified by three numbersColor space - analogy to geometrical 3D space

C2

C3

C1

C

17

RGB space is most commonNative to cameras, scanners, displaysBased on additive color mixing

CMY(K) for printingSubtractive color mixing

Luminance-chrominancespaces

Important to separate, for instance for ”Chroma Subsampling”HSV, HSL, YIQ, YUV, YCbCR, Lab, Luv, …

Color spaces for images and video

18

YCbCr Color SpaceA luminance-chrominance color space: YCbCr

Standardized for digital video by Recommendation ITU-R BT.601.4 Used in JPEG image compression and MPEG video compression.Closely related to the YUV transform.

Luminance Y’ (a.k.a. “Luma”)Y’ = 0.299R’ + 0.587 G’ + 0.114B’Calculated from gamma-corrected signals R’, G’, B’Note difference from colorimetric luminance Y

Chrominance Cb and CrCb = ((B’−Y’) / 1.772) + 0.5Cr = ((R’−Y’) / 1,402) + 0.5

19

Color Spaces3 different photoreceptors, L, M, S

3-output linear modelColor can be specified by three numbersColor space - analogy to geometrical 3D space

C2

C3

C1

C

Device-independent color spacesLMS, XYZ, CIELUV, CIELAB, sRGB …

Device-dependent color spacesMonitor RGB, Printer CMYK, Scanner RGB ...

4

20

Device independenceDevice-independent color spaces

Defined according to colorimetryLMS, CIEXYZ, CIELUV, CIELAB, sRGB …

Device-dependent color spacesMonitor RGB, Printer CMYK, Scanner RGB ...

21

CIELAB color space (a.k.a. L*a*b*)Device independentPseudo-uniformly related to human perception:

Defined according to a standard observerInternational Standard (CIE) Used in Color Fax and CMS

Lightness L*L* = 100 = white L* = 0 = black

Chromaticity a* and b*a* = green (-) to red (+)b* = blue (-) to yellow (+)

22

sRGB Color SpaceStandardized RGBDevice independentDefinition based on a typical PC monitorAdvantages:

Ease of useInternational Standard (IEC)Endorsed by major business players (HP, MSFT)A common color language - Scan to sRGB - Print from sRGB - - -More and more consumer imaging devices “speak” sRGB

Disadvantages:Yes

http://www.srgb.com

From

http:

//www

.srgb

.com

23

Color DifferenceEuclidean distance in CIELAB space

ΔE*ab “Delta E”A good measure of perceptual color difference Much used as color quality metric:

Original Reproduction

Average ΔE*ab = ?Max ΔE*ab = ?

24

Perceived color depends on...Spectral distribution of lightSpectral reflectance of surfaceSpectral sensitivity of eye

… but also on …

Size of areaViewing directionViewer adaptationSurrounding colorLocal contrast …

Brain

25

5

26

Assimilation27

Simultaneous contrast

28 29

30 31

Local vs. global color changes

2

2

2 Adding a cyan filter over the cushion

3

3

3 Adding a cyan filter over the whole image

Illustrates adaptation to white point -color constancyWhite balance for camerasColor appearance modeling - CAM

1

1

1 Original image

Images courtesy of Dr. R.G.W Hunt, Univ. of Derby, UK

6

32

OutlineIntroductionColor ScienceColor ManagementColor Image Quality

33

The first law of color managementDifferent imaging devices never produce equal color!

34

Scanners ‘speak’ color differently

File B

File AScanner A

Scanner B

OriginalImage

≠

35

Printers ‘speak’ color differently

Printer A

Printer B

Digital file

≠

36

We speak color differently

I wonder if this skirt matches mybeautiful “Barbie Doll Pink” blouse?

Let’s see - they write it’s “Hot Pink,”and in the image it says

(R,G,B) = (245, 174, 203) ??!?

37

Complex imaging systems

7

38

Principle of Color ManagementImage interchange through Device-Independent Color Space

Esperantoof colors

Imaging devices characterized by Profiles

Dictionaries

ScannerProfile

Device-Independent Color Space

PrinterProfile

ImageProfile

MonitorProfile

CameraProfile

39

Principle of Color Management

ScannerProfile

PrinterProfile

Color conversion between devices by “coupling” two profiles - source and destination

40

LUT-based colorspace conversion

G

B

R

One look-up table (3DLUT) per conversion

Pre-calculated conversion for a subset of all possible input colors

Interpolation for in-between valuesContent of LUTs depends on the actual devices, and is determined by device profiling

A.k.a colorimetric characterization

41

Colorspace conversions in an MFP• Local color copy

• Scanner RGB Printer CMYK

• Scan to PC

• Scanner RGB sRGB

• Print from PC

• sRGB Printer CMYK

• ITU Color Fax

• Scanner RGB CIELAB

• CIELAB Printer CMYK

• Internet Color Fax

• Scanner RGB YCbCr

• YCbCr Printer CMYK

Scan Device

ColorspaceConversions

CIELAB

PCApplication

sRGB

CMYK

PSTNColor Fax

YCbCr

Internet Color Fax

Print Device

ScannerRGB

42

International Color Consortium - ICCInternational standard for Color Management

Architecture of Color Management SystemsICC Profile file formatNow also ISO standardSee www.color.org

43

Input Device Profiling (scanner)

Previously known or measured

colorimetric dataScanner Profiling

AlgorithmColor Target

Mean values extraction

ScannedImage

File

ICC Profile

Device Independent

ColorImage

Scanner PC

Device-dependent RGB data

Device-independent color data (CIELAB)

8

44

Output Device Profiling (printer)

PrinterProfiling

CMYK or RGBprinter target

digital file

Device-dependent CMYK or RGB data

Device-independent color data (CIELAB)

Image File from Fax or

Scan

PrintingEngine

PrinterController

ICC Profile

45

Approaches to device profilingA.k.a colorimetric device characterizationTypical algorithms based on

Physical modelsGOG model for CRT displaysNeugebauer model for halftone print++

Data-driven models3DLUT InterpolationPolynomial regressionNeural networksThin-plate splines++

46

Some active research areasImproved algorithms for deviceprofiling / colorimetriccharacterization

LCD and DLP projection displaysHue-preserving cameracharacterization model

Compensation for varyingviewing conditions

Color appearance modelingColor gamut mappingSpectral reproductionWorkflow issues

Ease of useGet people to do the right thingsNew areas such as digital film production

C.F. Andersen and J.Y. Hardeberg, Colorimetric characterization of digital cameras preserving hue planes, 13th ColorImaging Conference, pp. 141-146, 2005

L. Seime and J.Y. Hardeberg, Colorimetric Characterisation of LCD and DLP Projection Displays, Journal of the Society of Information Display, 11(2): 349-358, 2003

J.Y. Hardeberg, Colorimetric ScannerCharacterization, Acta Graphica, 155, 2005

J.Y. Hardeberg, I. Farup, Ø. Kolås, and G. Stjernvang, Color management in digital video: Color Correction in the Editing Phase, Proc. IARIGAI Conference, pp. 166-179, Lucerne, Switzerland, September 2002

E.B. Mikalsen; J.Y. Hardeberg & J.-B. Thomas. Verification and extension of a camera-based end-user calibration method for projection displays. CGIV, 2008

A. Alsam; J. Gerhardt & J.Y. Hardeberg. Inversion of the Spectral NeugebauerPrinter model. AIC Colour 05, 44-62, 2005

47

Color Gamut MappingImaging devices have only a limited number of reproducible colors: Color GamutWhat to do if your image contains “out-of-gamut” colors?

Can’t print anything “more yellow” than the yellow ink!

Need for Gamut Mapping“Do the best you can with the colors you have”Huge unsolved problem in color imaging R&DClipping out of gamut colors is not good enough

48

Gamut Mapping and VisualizationCurrent research topics

Image-dependent and/or interactive new gamut mapping algorithmsImportance of gamut boundary descriptors:

Segment-maxima, convex hull, alpha-shapes, regular structure, …

Spatial gamut mapping With University of Milano

Popular freeware tool developed by us

ICC3D http://www.colorlab.no/icc3d/ I. Farup, J. Y. Hardeberg, A. M. Bakke, S.

Kopperud, and A.Rindal, Visualization and Interactive Manipulation of Color Gamuts, Proc. 10th Color Imaging Conference, pages 250-255, Scottsdale, Arizona, November 2002

Ivar Farup, Jon Y. Hardeberg, and MortenAmsrud, Enhancing the SGCK Colour Gamut Mapping Algorithm, Proc. CGIV 2004, pp. 520-524, Aachen, Germany, April 2004

Ivar Farup and Jon Y. Hardeberg, Interactive Color Gamut Mapping, Proceedings of the 11th International Printing and Graphic Arts Conference, Bordeaux, France, October 2002

Ivar Farup; Carlo Gatta & Alessandro Rizzi. A Multiscale Framework for Spatial Gamut Mapping. IEEE Transactions on Image Processing, 16(10):2423-2435, 2007

Øyvind Kolås & Ivar Farup. Efficient Hue-preserving and Edge-preserving Spatial Color Gamut Mapping. IS&T and SID's 15th Color Imaging Conference, 207-212, 2007.

Arne Magnus Bakke; Jon Yngve Hardeberg & Ivar Farup. Evaluation of Gamut BoundaryDescriptors. IS&T and SID's 14th ColorImaging Conference 50-55, 2006

49

OutlineIntroductionColor ScienceColor ManagementColor Image Quality

9

50

What is color image quality?51

What is color image quality?

52

What is color image quality?

53

What is color image quality?

What is color?What is an image?What is quality?

Dictionary: “Degree of excellence”ISO9000: “The features of a product (or service) which are required by a customer”

54

Quality is what the customer wants!Quality is what the customer wants!Quality is what the customer wants!Quality is what the customer wants!Quality is what the customer wants!Quality is what the customer wants!Quality is what the customer wants!Quality is what the customer wants!

55

Customer-driven quality definitionNorwegian governmental purchase of digital print equipmentE.g. letterhead quality of primordialimportance

P. Nussbaum and J.Y. Hardeberg, Print Quality Evaluation for Governmental Purchase Decisions, Proc. IARIGAI Conference, Copenhagen, September 2004

10

56

How to evaluate color image quality?57

How to evaluate color image quality?From the web site of a major French consumer electronics retailer:

Qualité d'image = taille du point x nuances de couleursImage quality = point size x color depth

NOT

58

How to evaluate color image quality?NOT 59

How to evaluate color image quality?NOT

60

How to evaluate color image quality?

Subjective evaluations”Expert opinions”Customer’s choices

Psychophysical experiments”Panel tests”Statistical analyses

MeasurementsColorimetric and planimetricPixel-by pixel comparisonsPerceptual models etc.

61

Psychophysical experimentsDifferent protocols

Category judgmentRank orderPair comparison

Time consumingThorough statisticalanalysis

But some questionableassumptions?

11

62

Measuring color image qualityBy color measurements

Match of specific colorsE.g. Company logo

Match monitor/printTypical consumer user scenario

Match original-reproductionColor copy, fax

Match to a reference print

Do not use numbers blindly!

Original

Reproduction

Mean ΔE*ab = ?Max ΔE*ab = ?

63

Preferred color reproductionGenerally it is not desired to reproduce the colorimetry of the original scene

(Gio

rgia

nni&

Mad

den,

199

7)

64

Colorimetric match ≠ visual matchBasic colorimetry is inadequate for cross-media color reproduction

65

Inadequacy of RGB color difference

ΔRGB=100 ΔRGB=100

ΔEab=22

CIEDE2000=4,8

ΔEab=63

CIEDE2000=32

66

Reproduction 1 and 2 have the same average ΔEab of 0.6CIE ΔEab colour difference equation is designed to quantify the colour difference for single pair of colour patchesDon’t ever use RMS or PSNR to evaluate yourimage compression algorithm again!

35Eab =Δ

Reproduction 1

Inadequacy of pixel-by-pixel difference

Original Reproduction 2

67

Perceptual image difference metricsExactly how big is the difference betweenthese two images?

12

68

Perceptual image difference metricsMetrics taking into account visual CSFs have beenproposed

s-CIELABiCAM

Many othermetrics

SSIM

69

Perceptual image difference metrics

Image difference maps

S-CIELAB iCAM CIELAB

ReproductionOriginal

70

Perceptual image difference metricsWe found no correlationbetween current image difference metrics and perceptually evaluatedimage differencesHigher-level cognitiveprocesses come into playCurrent active researcharea

E. Bando, J.Y. Hardeberg, D. Connah, Can Gamut Mapping Quality be predicted using Colour Image Difference Formulae?, SPIE Proc. 5666, 2005

71

Perceptual image difference metricsResearch project funded by theResearch Council of Norway

Ca 5.4 MNOK over 4 years 2007-2011

Related project funded by Océ PrintLogic Technologies (Paris, France)

Ca 1.4 MNOK over 4 years 2007-2011

New metrics development/evaluationImplement and evaluate existingmethodsNew metrics based on improvedcolour difference metricsNew metrics based on perceptualpredictors

Such as RetinexNew metrics based on saliency maps

Eye tracking experimentsNew metrics based on multilevelmethods

Using metrics to improvereproduction&representation

Standard optimization methodsHeuristic methodsVariational methods

ApplicationsImage compressionHigh dynamic range tone mappingColour gamut mappingHalftoning

72

Perspectives on image quality research”Most studies on image quality employ subjectiveassessment with only one goal – to avoid it in the future”

Yendrikhovskij

CIQmachine

CIQ=100!CIQmachine

73

OutlineIntroductionColor ScienceColor ManagementColor Image Quality

13

74 75

The Norwegian Color Research Laboratory

Research group at the Faculty of Computer Science and Media Technology at Gjøvik University College

Rooted in graphic arts industry needs but now serving the converging media technology industry

Great people3 permanent faculty3 associated faculty4 doctoral students and 1 postdoctoral researcher1 color management engineerBachelor and master students

Well equipped laboratory facilitesColor and spectral measurement, Color managementsoftware, viewing booths, image acquisition and reproduction devices, monochromator, etc. >200m2 lab space

Research in color science and image processing

Color Management and Device CharacterizationColor Image QualityColor Gamut Mapping and VisualizationMultispectral Color Image Acquisition and ReproductionFundamental Colorimetry and Psychophysics

Strong ties to the faculty’s media related bachelor and master programs

http://www.hig.no/mediahttp://www.master-erasmusmundus-color.eu/

Rich national and international network with industry and academiahttp://www.colorlab.no

76

Colorlab Research OverviewApplied research

Graphic arts industryColor managementPrint quality evaluationPSO color certification

Media technology industryDisplay technologyMovie productionPhoto managementTeleconferencing (?)

Computer visionRecycling sorting

Security applicationsPeople countingIntelligent videosurveillanceDocument and imageforensics

Fundamental researchMultispectral colorimaging

NFR/SHP 2003-20074.8 MNOK

Perceptual image difference metrics

NFR/SHP 2007-20115.51 MNOK

Color and perceptionColor memoryEye movements

In-between…Gamut mappingPrint and image qualityDevice modelingFace detection

77

CREATEColor Research for European AdvancedTechnology Employment

http://www.create.uwe.ac.uk/Series of conferences and research courses

Print-oriented courses in the UK, fall 2008Large final conference in Gjøvik in 2010

EU- funded through Marie Curie programFree participation (including travel) for selectedyoung researchers

Partners:Università degli Studi di Milano, Italy; Gjøvik University College, Norway; University of Leeds, UKUniversity of Ulster in Belfast, Northern Ireland, Universidad Autónoma de Madrid, Spain; Université de Reims Champagne-Ardenne, France; University of Pannonia, Hungary.

78

CIMET key factsMSc programme: Color in Informatics and Media TechnologyCollaborative effort supported by the EU

In Master Mundus programme: bringing together the best Europan forces to attract the best brains to EuropeUniversity of Saint-Etienne (France, coordinator), the University of Granada (Spain), the University of Joensuu(Finland), and Gjøvik University College (Norway)

Prestigeous for institutionsOnly three selected programmes in Norway in 2007

Prestigeous for studentsGenerous grants from EU

To cover tuition fees (EUR 10.060 per year)Highly competitive selection

Only 30 students per year

http://www.master-erasmusmundus-color.eu/

79

Questions?E-mail:

jon.hardeberg@hig.nojon@hardeberg.com

Web:http://www.hig.nohttp://color.hardeberg.comhttp://www.colorlab.no

Thanks to:Current and past Colorlab faculty and students

Upcoming events:Open Colorlab Workshops

4.6.08: Psychophysical experiments and statistics19.6.08: Face detection and recognition

Gjøvik Color Imaging Symposium 2009Immediately following SCIA09