Adaptive Spectral Projection

Adaptive Color Display via Perceptually-driven Factored Spectral Projection

Adaptive Color Display via Perceptually-driven Factored Spectral Projection Isaac Kauvar, Samuel J Yang, Liang Shi,Ian McDowall, Gordon WetzsteinStanford University

Testing.1

v

Three of the most important qualities of a display 2

v

Resolutions have substantially improved, from VGA to HD to 4K UHD and beyond, where for many displays, the pixels are no longer visible to the human eye. 3

v

Contrast has also been increasing, both through improvements in the black levels of displays, as well as, through new High dynamic range displays. 4

v

Color, on the other hand, remains an unsolved problem. Large gamuts have yet to be fully realized for general consumption.

( The newRec20:20 specification for UHD calls for a very large gamut, but this gamut, as well as even larger color gamuts have yet to be fully realized for general consumption. ) 5

v

We focus on this problem in the work that I will present today. 6

Conventional displays produce a limited set of colors.

The problem of color reproduction is highlighted by the fact that most displays can only produce a limited set of colors.7

Conventional displays produce a limited set of colors.

displays gamutchromaticity diagram

The gray horseshoe shape represents all colors the standard human observer can see, whereas the green triangle represents the subset of colors that conventional displays (shown here with the common sRGB gamut) are able to produce.8

Limited gamuts prevent accurate color reproduction.

The issue is that limited gamuts prevent accurate color reproduction. 9

This was highlighted when just recently, I was in kyoto, and I visited the ginkakuji silver temple. The garden there is covered with different mosses in every concievable shade of green.I wanted to capture and display these images so that I could share them with my friends, but no matter how hard I tried, my iPhone camera and its display could not achieve the colors that I wanted. There are, of course, two issues at hand: the camera, and the display. 10

What about content?Multi-spectral cameras

MicaSense RedEdge

Ximea xiSpec

Computer generatedimagesBig Buck Bunny

There are multiple approaches for overcoming the camera limitation: first, computer generated images can possess any desired gamut,and second, there has recently been much commercial development of cheap and accessible multi-spectral cameras. These cameras are capable of capturing all desired color content. It remains a challenge, however, to display such content. 11

Flexible gamut.

softwarehardware+

To solve the display problem, we advocate the use a flexible gamut, one whose color primaries depend on the content to be displayed. This enables one to efficiently use the available hardware capabilities. We developed a hardware prototype that enables flexible gamut selection. And we developed an optimization algorithm that chooses the best primaries for a given image.12

Flexibility yields performance.

sRGB visualizationof multispectraltarget image

The result is that we can obtain a much better match between the target image and the image reproduced by the projector, shown in this example. As our input image, we use a multispectral image, whose sRGB representation is visualized here.

To demonstrate the performance of our algorithm, we compare the results of our and legacy algorithms based on the error between the perceived image color and the target image color. As our input image, we use a multispectral image, whose sRGB representation is visualized here (obviously, the projector in this room is not capable of showing the full multispectral 13

Flexibility yields performance. Lab

CIELAB-76 error from target


As our perceptual error metric, we use the euclidean distance between the target and the result in CIELAB-76 color space. 14


LabStandard fixed gamut



With a standard fixed gamut and gamut mapping algorithm, there is a large error represented by the warm colors in this error plot. 15


LabStandard fixed gamutLegacyflexible gamut



Using a flexible gamut algorithm that was proposed recently but that does not optimize for smallest perceptual error, we see better performance but still some error. 16


LabStandard fixed gamutLegacyflexible gamutOur newflexible gamutalgorithm



Finally, with our perceptually-driven algorithm, we see nearly zero error between the target and the output, as indicated by the blue images in the bottom row. 17


LabStandard fixed gamutLegacyflexible gamutOur newflexible gamutalgorithm



Hardware implementation.

This new design is more robust, easier to align, cheaper, and more manufacturable. And is the design that we used throughout the rest of the paper. 19

Works with human users.

Finally, we found that with our algorithm, human users were better able to see and distinguish colors that were indistinguishable when using legacy techniques. 20

What limits the gamut of conventional projectors?

Okay, so what actually limits the gamut of conventional projectors. 21

++=

Multiplexed color primaries.

Many projectors temporally multiplex red, green, and blue images that are then averaged together by your brain. When designing the RGB filters, tradeoffs have to be made between the brightness of the projector and color purity of the primaries.

22

Limitations of current displays. Human flicker fusion rate: 60 fpsMax frame speed of standard display: 180-240 fpsMaximum number of primaries per image: 3-4=

Because of the limited speed of current displays, for each full color image, it is only possible to show 3 or 4 primaries, such as red green and blue.23


Columbia multispectral dataset

Fortunately, as it turns out, most natural images can be well represented by just three primaries. We did an experiment here to validate that previously discovered claim. 24


Columbia multispectral datasetBut the primaries are differentfor each image!

Importantly, however, the 3 primaries are different for each image. This insight inspired our flexible gamut approach, where we are able to choose different primaries for different images. 25

Related work.Gamut mapping

Banterle, et. al. 2011

Before describing our work, it is worth discussing previous related work. The gamut mapping problem of squeezing image content into the gamut of a particular display has received a lot of attention, and these days every company has their own secret sauce. 26

Related work.Gamut selectionGamut mapping


Long and Fairchild, 2011

Gamut selection is also essential to the design of current displays, where the gamut must be chosen and permanently fixed during the manufacturing process. 27

Related work.Gamut selectionLi, et. al. 2015MultipleprojectorsGamut mapping



There has also been work to extend color gamuts by overlaying multiple projectors that each have different gamuts. This is a strong approach, but requires careful calibration and is not the route we decided to pursue. 28

Related work.Gamut selectionJoint primary selectionand gamutmappingBen-chorin and Eliav, 2007Li, et. al. 2015MultipleprojectorsGamut mapping



Instead, we chose to solve the problem of joint gamut mapping and primary selection:For each image, we simultaneously determine both the best gamut and gamut mapping that are possible within the constraints of our hardware. The small amount of previous work on this problem of joint selection has been limited to using non-negative matrix factorization in a non-perceptually accurate color space. I will explain more specfiically what that means over the next few slides. 29

It is easiest to describe our algorithm in the context of the actual hardware implementation that we used. In particular, we had an LED light engine with, in our case, six LEDs, coupled into a DMD based grayscale projector. 30

gamut selection

Gamut selection consisted of choosing, for each time point, the linear combination of LED intensities yielding a color primary, which corresponds to one corner of the gamut triangle.31

gamut selection gamut mapping

The gamut mapping problem is then equivalent to choosing the grayscale image to display on the DMD corresponding to each color primary. 32

Thus, for each time point, we have a vector of 6 LED coefficients, and the grayscale value of each pixel. Because the three time points are averaged together by the observer, the image formation can be represented as a matrix multiplication. 33

The joint gamut mapping and gamut selection problem then consists of choosing the entries of these matrices, which we call G and H. 34

The matrix product of G and H yields for each pixel a 6-component color vector representing the averaged color of that pixel. 35

We project this six-component color vector onto the spectral sensitivity responses of the retinal cone cells. The result is a three-coordinate color in CIEXYZ color space for each pixel. Now, given a target multispectral image, which can also be represented in XYZ color space, again,our goal is to find G and H that results in the best match to the target image. 36

This optimization can be represented by a non-negative matrix factorization problem, non-negative because we can only have positive pixel and LED values. The error metric here is the euclidean difference between the target and result image in XYZ color space. 37

But human color perception is not linearin CIEXYZ space!

However, the previous description assumes that color differences are linear in XYZ space. Unfortunately, human color perception is in fact not linear in this space.38

MacAdams 1942Ellipses scaled by 10.

CIEXYZcolor spaceBut human color perception is not linearin CIEXYZ space!

This is represented by the fact that in XYZ color space, the size of the Just Noticeable Difference region is not constant. 39

MacAdams 1942Ellipses scaled by 10.

Jung 2011

CIELAB76 color spaceNonlineartransform

CIEXYZcolor space

By switching to CIELAB76, we can obtain something that is a much better approximation to human perception. While not a perfect representation, locally it is nearly linear, and our assumption is that it is more perceptually relevant. We note that there are even more modern color spaces, some of which we were also able to include in our optimization. See the paper for details. 40

So, how do we incorporate the CIELAB color space into our optimization?41

We do so by including the nonlinear function that converts from XYZ to LAB.42

Conversion to CIELAB (nonlinear)

The addition of this term makes the optimization problem significantly harder. Whereas before, the problem was bi-convex, the addition of this nonlinear term makes the problem nonconvex. 44

Conversion to CIELAB (nonlinear)Nonconvex!

We initially tried levengerb-marquardt update rules (i.e. Alternating least squares), but it was far too slow and offered no promise of substantial speedup.

Note you cannot just apply the inverse of phi to Ilab, because then your metric is still in linear space

45

Reformulate the problem.

Instead, to efficiently solve this nonconvex problem, we reformulated it in a way that allowed us to split up the easy and hard parts of the problem.

46

Reformulate the problem.

Instead, to efficiently solve this nonconvex problem, we reformulated it in a way that allowed us to split up the easy and hard parts of the problem.

47

Standard ADMM update rules.

Loop until convergence:

This formulation then fits directly into the Standard ADMM update rules. 48


Loop until convergence:Solve nonlinear problem on per-pixel basis

The key is that rather than solving a nonlinear levenberg marquardt across the whole image simultaneoulsy, we instead solve a bunch of small, pixelwise problems in parallel. 49


Loop until convergence:Solve nonlinear problem on per-pixel basisSolve bi-convexproblem(standard NMF)

Then, we solve the standard NMF problem; it has been shown previously that this can be implemented on a GPU in real time. 50

Convergence.

The implementation is Efficient and also displays smooth convergence, across all of the images we tested.

Although the run time for our implementation was very slow, about 2 hours per image, but it was entirely implemented in matlab and no effort was made to speed it up by porting to a GPU.51

Performance.

++=

What does the output of our algorithm look like? Here, represented in false color, we see the output pixel values corresponding to the three primaries for an example multispectral dataset. 52

Performance.

++=Resulting gamut

The gamuts themselves looked like this.53

Performance.

++=Resulting gamut

Tradebrightness for color spread

Note that the algorithm made a tradeoff in the full 3D color space between color coverage and maximum image brightness. 54

Performance.

++=

Error in LAB coordinatesResulting gamut

Tradebrightness for color spread

We can observe the performance of the algorithm by looking at the final residual. 55

Performance.

++=

Error in LAB coordinatesError of legacy method

If we compare this to the legacy adaptive gamut algorithm, however, we see a stark improvement. 56

Optimization significantly reduces LAB error.

These results held across many tested images. Our algorithm is the bottom row.

Further, our algorithm outperformed both the legacy flexible gamut algorithm and the fixed-gamut gamut mapping.These results held across many tested images. Our algorithm is the bottom row. 57

Optimization consistently reduces LAB error.

Pay attention to the bottom row, and the lack of error in the blue images. 58



Our algorithm

This performance improvement can be summarized across all tested multispectral images. 60

It also has better performance with other metrics.

Additionally, even though we optimized in LAB94 color space, our results were high performing in additional, more complicated and more perceptually accurate color spaces such as sCIELAB and LAB200061

Design of a flexible gamut projector.

Finally, I want to briefly discuss our hardware implementation. Given an optimization algorithm that is doing its job to select the best color gamuts for a given image, how to do we actually display the image using this gamut?62

We used a Lumencor light engine as the illumination source for a modified Texas Instruments LightCrafter DMD projector. 63

We synchronized the DMD with the light engine using a National Instruments Data Acquisition board. 64

Projector calibration.

We calibrated the projector by measuring the spectrum of each LED in the light engine. 65

Projector calibration.

We could then calculate the gamut of the projector, the solid black line, which could cover nearly the entire 2D chromaticity diagram. Thus, we have a lot of room to work around in and play with when determining the optimal 3-primary gamut for an image. 66

Empirical results.

Our projector was able to successfully display vivid color images, as shown in this photograph of the setup. 67

Empirical results.

While I cant show you the results using the projector in this room, because the color gamut of this projector is not as wide as that of our projector, I can at least show you that our projector and the entire optimization pipeline was capable of producing artifact free images.

You will have to rely primarily on the error plots of the previous few slides as a measure of the performance of our algorithm. We also performed a user study that attempted to support the fact that our algorithm is better at maintaining color differences. 68

Metamer user study corroborates our results.

Our algorithm

We designed this user study to corroborate the effectiveness of our approach, while holding off for now on a full user study that would rely on subjective evaluation of image quality. We chose three pairs of colors, and tested whether unknowing users could distinguish those pairs by asking them how many circles they saw displayed in an image, interleaving the images generated by the three different algorithms: fixed gamut, non-perceptual flexible gamut, and our perceptual flexible gamut.The results highlight that our algorithm better preserves perceptual color differences than both a fixed gamut as well as the legacy adaptive gamut algorithm. Thus, in conclusion we have developed an algorithm and hardware implementation that ensures that the colors displayed by your projector, match the colors you perceive in the world.

1) A flexible gamut has the ability to adapt more readily in situations which cause a fixed gamut to fail; 2) PNMF can maintain perceptual color differences better than NMF; 3) Our hardware prototype is capable of conveying the results of the flexible gamut algorithm. 69

Gordon WetzsteinSamuel J YangIan McDowallcomputationalimaging.orgwww.stanford.edu/~ikauvar

Liang Shi

Wetzstein group homepage:Isaac Kauvars homepage:

Thank you. 70

5 primaries

5 primaries

Gauss newton step with CIELAB-76

NMF

References Y. J. Jung, H. Sohn, S. Lee, Y. M. Ro, and H. W. Park, Quantitative Measurement of Binocular Color Fusion Limit for Non-spectral Colors, Optics Express, vol. 19, no. 8, pp. 7325-7338, 2011MOHAN, A., RASKAR, R., AND TUMBLIN, J. 2008. Agile spectrum imaging: Programmable wavelength modulation for cameras and projectors. Computer Graphics Forum 27, 2, 709717.RICE, J. P., BROWN, S. W., ALLEN, D. W., YOON, H. W., LITORJA, M., AND HWANG, J. C. 2012. Hyperspectral image projector applications. vol. 8254, 82540R82540R8.AJITO, T., OBI, T., YAMAGUCHI, M., AND OHYAMA, N. 2000. Expanded color gamut reproduced by six-primary projection display. In Proc. SPIE 3954, 130137.BEN-CHORIN, M., AND ELIAV, D. 2007. Multi-primary design of spectrally accurate displays. Journal of the SID 15, 9, 667677.BANTERLE, F., ARTUSI, A., AYDIN, T. O., DIDYK, P., EISEMANN, E., GUTIERREZ, D., MANTIUK, R., AND MYSZKOWSKI, K. 2011. Multidimensional image retargeting. In SIGGRAPH Asia 2011 Courses, 15:115:612.MACADAM, D. L. 1942. Visual sensitivities to color differences in daylight. OSA JOSA 32, 5, 247273.LI, Y., MAJUMDER, A., LU, D., AND GOPI, M. 2015. Content- independent multi-spectral display using superimposed projec- tions. Computer Graphics Forum (Eurographics).CHIAO, C.-C., CRONIN, T. W., AND OSORIO, D. 2000. Color signals in natural scenes: characteristics of reflectance spectra and effects of natural illuminants. OSA JOSA A 17, 2, 218224.DANNEMILLER, J. L. 1992. Spectral reflectance of natural objects: how many basis functions are necessary? OSA JOSA A 9, 4, 507515.

Adaptive Spectral Projection

Science

Transcript of Adaptive Spectral Projection