ImageEd: Technical Overvie · Color adjustment in 3-D space to utilize gamut optimally 1-D versus...

18
- 1 - ImageEd: Technical Overview Purpose of this document This paper is meant to provide insight into the features where the ImageEd software differs from other image-editing programs. The treatment is more technical than in the User Guide. The reader is assumed to be familiar with the functions of ImageEd, and to have basic knowledge of digital imaging, photography, and color models. Advanced technology behind simple interface Today almost every image viewer program has basic functions for image adjustment. Since ImageEd has few controls, it might seem at first sight that ImageEd belongs to the same category with some cheap or freeware programs; this is a totally false impression. ImageEd has few controls because it is the very idea of ImageEd to do as much as possible with as few controls as possible. In simple image adjustment programs, a control works in exactly the same way for every image. In other words, such programs have no intelligence at all. There must be a separate control for every numerical parameter that is needed to fully define an adjustment operation. In ImageEd, in contrast, the meaning of the controls is different for each individual image. Based on advanced image analysis, ImageEd is able to define the meaning of each control optimally for the particular image being processed, thereby reducing the degrees of freedom left to the user. A further unique feature of ImageEd is that all controls are applied simultaneously rather than sequentially. This is illustrated in Fig 1. Suppose that you first adjust color balance and then contrast using a simple sequential program. Then you change your mind and want to undo the color balance adjustment. To do this, you have to undo both contrast and color balance, and then do contrast again. Furthermore, you probably have to select slightly different parameters for the contrast this time, because the input image to the contrast adjustment has changed: at the first time it was the image with color balance applied; now it is the original image. In the basic adjustment interface of ImageEd there is no similar order dependency. The user can move the sliders in any order and undo or redo the adjustments freely. For example, to undo color balance adjustment you only need to reset the color balance slider, no matter which other sliders you have moved. The result depends on the final slider positions only. Of course, the processing in ImageEd also consists of several sequential stages in a fixed order. However, the sliders are order independent to the user because the adjustment effect produced by a certain slider position is not fixed but depends on the positions of other sliders; a change in one slider is internally taken into account by ImageEd when it calculates the effect of other sliders. Color adjustment in 3-D space to utilize gamut optimally 1-D versus 3-D adjustments The simplest way of adjusting the colors of an RGB image is to process the R, G, and B components separately (Fig. 2). Such an operation is the combination of three one-dimensional adjustment operations. This method is often sufficient for color balance and underexposure adjustment. For example, to correct greenish color cast, G is multiplied with a constant smaller than 1, while R and B are multiplied by a constant greater than 1. To correct underexposure, all three components are multiplied by the same constant greater than 1. Since RGB adjustments are very fast, ImageEd also uses them whenever possible. It is very common have both underexposed and overexposed areas in the same image. This occurs in scenes with a very large intensity range. To enhance the dark regions in such images, a nonlinear adjustment curve is needed. Applying a single nonlinear curve separately to each RGB component is the simplest possibility, but this is usually not a good solution because the operation may affect the color saturation and hue in an unwanted way. Besides nonlinear lightness adjustment, saturation adjustment is an operation that cannot be done with one-dimensional operations for R, G, and B. A conventional scheme to do such adjustments is shown in Fig. 3. The colors are first transformed to another coordinate system such as HLS (hue, lightness,

Transcript of ImageEd: Technical Overvie · Color adjustment in 3-D space to utilize gamut optimally 1-D versus...

Page 1: ImageEd: Technical Overvie · Color adjustment in 3-D space to utilize gamut optimally 1-D versus 3-D adjustments The simplest way of adjusting the colors of an RGB image is to process

- 1 -

ImageEd: Technical Overview

Purpose of this document This paper is meant to provide insight into the features where the ImageEd software differs from other image-editing programs. The treatment is more technical than in the User Guide. The reader is assumed to be familiar with the functions of ImageEd, and to have basic knowledge of digital imaging, photography, and color models.

Advanced technology behind simple interface Today almost every image viewer program has basic functions for image adjustment. Since ImageEd has few controls, it might seem at first sight that ImageEd belongs to the same category with some cheap or freeware programs; this is a totally false impression. ImageEd has few controls because it is the very idea of ImageEd to do as much as possible with as few controls as possible.

In simple image adjustment programs, a control works in exactly the same way for every image. In other words, such programs have no intelligence at all. There must be a separate control for every numerical parameter that is needed to fully define an adjustment operation. In ImageEd, in contrast, the meaning of the controls is different for each individual image. Based on advanced image analysis, ImageEd is able to define the meaning of each control optimally for the particular image being processed, thereby reducing the degrees of freedom left to the user.

A further unique feature of ImageEd is that all controls are applied simultaneously rather than sequentially. This is illustrated in Fig 1. Suppose that you first adjust color balance and then contrast using a simple sequential program. Then you change your mind and want to undo the color balance adjustment. To do this, you have to undo both contrast and color balance, and then do contrast again. Furthermore, you probably have to select slightly different parameters for the contrast this time, because the input image to the contrast adjustment has changed: at the first time it was the image with color balance applied; now it is the original image.

In the basic adjustment interface of ImageEd there is no similar order dependency. The user can move the sliders in any order and undo or redo the adjustments freely. For example, to undo color balance adjustment you only need to reset the color balance slider, no matter which other sliders you have moved. The result depends on the final slider positions only. Of course, the processing in ImageEd also consists of several sequential stages in a fixed order. However, the sliders are order independent to the user because the adjustment effect produced by a certain slider position is not fixed but depends on the positions of other sliders; a change in one slider is internally taken into account by ImageEd when it calculates the effect of other sliders.

Color adjustment in 3-D space to utilize gamut optimally

1-D versus 3-D adjustments

The simplest way of adjusting the colors of an RGB image is to process the R, G, and B components separately (Fig. 2). Such an operation is the combination of three one-dimensional adjustment operations. This method is often sufficient for color balance and underexposure adjustment. For example, to correct greenish color cast, G is multiplied with a constant smaller than 1, while R and B are multiplied by a constant greater than 1. To correct underexposure, all three components are multiplied by the same constant greater than 1. Since RGB adjustments are very fast, ImageEd also uses them whenever possible.

It is very common have both underexposed and overexposed areas in the same image. This occurs in scenes with a very large intensity range. To enhance the dark regions in such images, a nonlinear adjustment curve is needed. Applying a single nonlinear curve separately to each RGB component is the simplest possibility, but this is usually not a good solution because the operation may affect the color saturation and hue in an unwanted way.

Besides nonlinear lightness adjustment, saturation adjustment is an operation that cannot be done with one-dimensional operations for R, G, and B. A conventional scheme to do such adjustments is shown in Fig. 3. The colors are first transformed to another coordinate system such as HLS (hue, lightness,

Page 2: ImageEd: Technical Overvie · Color adjustment in 3-D space to utilize gamut optimally 1-D versus 3-D adjustments The simplest way of adjusting the colors of an RGB image is to process

- 2 -

saturation). L and S (perhaps even H) are then adjusted with one-dimensional curves, and the result is transformed back to RGB.

The problem with this scheme is that the adjusted values of L and S may no longer be inside the original RGB gamut. The inverse coordinate transform must map the out-of-gamut values back to the boundary of the RGB gamut. This may destroy details in the image, as pixels that originally had different colors may have equal colors after the mapping. Of course, an adjustment program can warn of this situation by using some visual means to indicate the out-of-gamut pixels. However, preventing out-of-gamut pixels may also prevent doing the adjustment entirely. In other words, the 1-D operations may be simply too limited for getting the desired result.

ImageEd avoids the gamut problem by not confining itself to this kind of 1-D adjustments. ImageEd performs the lightness and saturation adjustments as 3-D operations applied to the RGB triplets (Fig. 4). Since the calculations are done in RGB space, the gamut boundaries are easily taken into account. The colors near the gamut boundary are adjusted less than colors deep inside the gamut, thus no out-of-gamut pixels occur in the result.

The following subsections further illustrate 3-D adjustments in various cases.

Lightness adjustment without saturation loss

Fig. 5a shows a nonlinear adjustment curve, and in Fig. 5c, the solid arrows illustrate schematically how colors are changed when the curve is applied separately to all three RGB components. Although the figure is 2-dimensional, the principle is the same in three dimensions. The straight dotted lines connect locations in color space where the perceived saturation, (actually, chroma in CIELAB space) is constant. (These lines are not exactly straight in reality but have been drawn here as straight for simplicity.)

When a color X, which is originally rather dark, is moved in the arrow direction, it encounters chroma lines that are farther from the gray axis, meaning that its chroma increases. On the other hand, a lighter color such as Y crosses chroma lines closer to the gray axis, i.e., the chroma decreases. The colors in typical natural images are so light on the average that the overall effect of such a simple lightness adjustment would be unacceptable saturation loss.

ImageEd avoids the saturation loss by analyzing what the saturation effect would be for the majority of the image colors. For images where almost all colors belong to the “X” category, ImageEd performs the lightness adjustment with simple 1-D curves. The perceived saturation then increases somewhat, but this is usually acceptable because most images tolerate slight saturation enhancement.

For images with an appreciable number of pixels belonging to the “Y” category, ImageEd performs the lightness adjustment as a 3-D operation in such a way that the colors stay farther from the gray axis (see Fig. 5d). The parameters of the adjustment are determined on a per-image basis so that the overall chroma is roughly preserved. The adjustment is defined so that no color moves outside gamut. For colors near the gray axis, the lightness change is the same as with 1-D curves. For saturated colors the lightness change is less than it would be with 1-D curves (compare points Y´ in Figs. 5b and 5c). This kind of compromise between lightness and saturation is a necessary consequence when we want to increase the lightness but prevent saturation loss. ImageEd handles this compromise by doing the adjustment in RGB space where the gamut is a simple cube, rather than switching to another space where the gamut boundary is a complex shape.

Saturation adjustment taking gamut into account

In addition to preserving saturation in lightness adjustment, ImageEd also has a separate saturation adjustment function, whose meaning is to increase only saturation while keeping the lightness constant. Fig. 6a illustrates the gamut problem that occurs if the saturation adjustment is done as a 1-D operation for the CIELAB chroma or a similar component. In a 1-D operation, the adjusted chroma is fully determined by the original chroma. Therefore color points X and Y, which are initially on the same chroma line C1, are on the same chroma line C2 after the adjustment. The color Y has moved outside the gamut.

The gamut problem could be avoided by defining the lightness component in a different way, as depicted in Fig. 6b. However, although this computational lightness is preserved, the true lightness decreases and thus the saturation adjustment could partially cancel the effect of lightness adjustment, making the lightness and saturation controls interdependent.

Page 3: ImageEd: Technical Overvie · Color adjustment in 3-D space to utilize gamut optimally 1-D versus 3-D adjustments The simplest way of adjusting the colors of an RGB image is to process

- 3 -

To overcome these problems, ImageEd performs the adjustment as in Fig. 6c. The colors move along lines of constant lightness; therefore, the lightness is preserved. For light colors near the gamut boundary such as Y, the saturation is adjusted less to prevent out-of-gamut results. Again, the adjustment is a 3-D operation in RGB space, which allows computationally efficient handling of the gamut restriction.

The saturation adjustment in ImageEd has another unique feature: the adjustment is smaller for dark colors. The visually most significant colored objects usually have their lightness in the mid-tone range or higher; hence, there is no need to increase saturation in the dark tones. Increasing saturation of dark areas would make colored shadows more colored and might increase the visibility of image noise, both of which are mostly unwanted effects.

Color balance adjustment with overexposed regions

The colors recorded in a digital picture depend both on the reflection properties of the photographed objects and the color of the lighting under which the picture was taken. The effect of the lighting may be perceived as an overall “color cast” that needs correction.

A simplified way of modeling the cast is to assume that the effect of the lighting is to multiply each of the R, G, and B components by some numbers depending on the lighting color. For example, under reddish light the multiplier for R is greater than that for B. In Fig. 7a, the left part shows an image taken under reddish lighting and the distribution of pixels in the R-B plane. The balance can be corrected by undoing the effect of the light color, i.e., by multiplying R and B with constants such that the multiplier for B is greater. This results in the linear, one-dimensional adjustment curves shown in the middle part of Fig. 7a. The right part shows the resulting image and its pixel distribution, which is now concentrated near the diagonal line.

Often the intensity in the scene varies in such a high range that the camera exposure must be chosen to overexpose the brightest spots in the scene, in order to get enough exposure for the less illuminated areas. In the overexposed regions the RGB values recorded by the camera are no longer proportional to light intensity but saturate to a constant maximum value. Then the multiplicative color balance correction is no longer appropriate. (Do not confuse the two meanings of “saturation”. In the previous sections saturation was used in meaning “color purity”.)

If the scene contains light sources, the intensity of the light sources is typically very much higher than the intensity elsewhere in the scene. Then, it may be the case that in the light source region all color components saturate, while in the rest of the image no color component saturates. The brightest spots thus appear white but the rest of the image has a colorcast. The balance can still be corrected with one-dimensional curves, but now the curves have to be nonlinear as in Fig. 7b, for they must preserve the whiteness of the bright spots.

Often the situation is not that simple. Fig. 7b shows what the image of part (a) would have looked like if a higher exposure setting had been used. With this exposure, the lightest regions in the scene produce pixels where only the red component has saturated to 255 but blue varies, as indicated by the oval. There are small regions where both R and B have saturated and thus the color is white. If per-component adjustment curves were used, both the R and B curves would have to be nonlinear so that they go through (255, 255), to keep the highlights neutral white. However, this 1-D adjustment preserves R in the entire partially saturated region, so it does not correct balance in this region, which results in a severe pink artifact. Alternatively, by leaving R unchanged and increasing B, the balance could be adjusted even in the indicated region, but this is not a good method either: it would make the region constant white (i.e., variation in B would disappear), meaning that details in this region would be lost.

In the presence of white areas like in Fig. 7b, ImageEd does not automatically correct the color balance because it is very difficult for a computer to determine with certainty whether a white region is an overexposed region under colored light, or a truly white object under neutral light. However, overexposure situations have been taken into account in how the color balance control operates if the user moves it past the automatic position. ImageEd handles the case of Fig. 7b by performing a 3-D operation in RGB space (Fig. 7c). For dark colors this operation works like the linear per-component curves in Fig. 7a. For bright colors it works in a truly 3-D way, taking the saturated pixels inside the gamut while preserving white. This operation can correct the balance even in the partially saturated region.

Page 4: ImageEd: Technical Overvie · Color adjustment in 3-D space to utilize gamut optimally 1-D versus 3-D adjustments The simplest way of adjusting the colors of an RGB image is to process

- 4 -

Contrast and lightness adjustment without hue changes

In RGB space, the hue of a color depends on which of the three RGB components is the largest and which is the smallest, and of the relative differences between the smallest, middle, and largest components. Colors with equal smallest and middle components have one of the primary hues red, green or blue. Colors with equal largest and middle components have one of the secondary hues cyan, magenta or yellow.

When the same linear curve is applied to all three RGB components, all the component differences increase (or decrease) by the same factor, thus hues are preserved.

The automatic contrast adjustment in ImageEd mainly aims to expand the tonal range, which may be too narrow because of underexposure or overexposure. This expansion is almost linear, thus the hues are preserved.

If the user moves the contrast slider past the automatic position, ImageEd begins to apply an adjustment that resembles an S-shaped curve like in Fig. 8. However, the operation is again 3-D rather than component-based.

Increasing overall contrast requires that the adjustment work like an S curve for the neutral colors (gray axis), but there are several possibilities to define the adjustment for other colors. One possibility is to apply the same S curve for each RGB component, which has the advantage that gamut problems are avoided, just like in the lightness case. However, this would distort the hues severely if the adjustment is strong. Assume a color whose every component is in the lower part of the S curve (Fig. 8). The curve causes the middle component to become relatively closer to the smallest component, thus producing a shift towards primary hues. Then assume a color whose every component is in the upper part of the S curve. Now the middle component becomes relatively closer to the largest component, causing a shift towards the secondary hues.

These hue shifts could distort e.g. “memory colors” such as those of skin and sky to an unacceptable extent. Therefore, the contrast adjustment operation in ImageEd is a three-dimensional modification of the per-component adjustment, defined in a way that preserves hues.

What is said here about hue distortion would also apply to lightness adjustment if it were done with component curves. ImageEd does not have this problem since the lightness adjustment is 3-D, as described previously. This operation also takes the hue preservation into account. An important thing is, however, that the RGB-based hue described above is not quite the same as perceived hue, so preserving the RGB hue exactly can actually change the perceived hue. For this reason, the lightness adjustment in ImageEd allows small changes to the RGB-based hue so that the perceived hue is better preserved.

Adaptive sharpening

Adaptive unsharp masking

Image sharpening is an operation that amplifies the high spatial frequencies present in the image while preserving the low frequencies. As a result, edges and small details are enhanced.

Sharpening is usually performed as unsharp masking (USM), where a blurred version is calculated from the original image then the blurred version is subtracted from the original; the difference is multiplied by a constant then added back to the original image. Assuming a gray level image, the formula for unsharp masking is

)),(),((),(),( yxIyxIAyxIyxI boos −⋅+= , (1)

where

),( yxIo = original gray value at pixel ),( yx ,

),( yxIb = value of blurred image at pixel ),( yx ,

A = the constant multiplier,

),( yxIs = value of sharpened image at pixel ),( yx .

In this conventional scheme, the sharpening operation is controlled by the following parameters:

Page 5: ImageEd: Technical Overvie · Color adjustment in 3-D space to utilize gamut optimally 1-D versus 3-D adjustments The simplest way of adjusting the colors of an RGB image is to process

- 5 -

• The general shape of the filter mask used for blurring, e.g. a Gaussian shape is often used.

• The width of the filter mask used for blurring (referred to as the “radius” of the USM and determines the size of details affected by the sharpening).

• The multiplier A, referred to as the “amount” of the USM. This is often expressed in percent, with 100% corresponding to A = 1.

Every image contains more or less noise, and a general problem with sharpening is that the high frequency components in the noise are amplified as well. To avoid this problem, unsharp masking usually has a further parameter in addition to the above: a “threshold”. If the gray level variation in the neighborhood of pixel (x, y) is smaller than the threshold, sharpening is not done at that pixel (A = 0 in Eq. (1)), or is done with A smaller than the nominal value.

How the local gray level variation is measured depends on the implementation; in the simplest case it might be just the difference ),(),( yxIyxI bo − . Also, the exact way how the threshold and local variation determine the multiplier A is implementation dependent. They essential thing is that when a threshold is used, A is a function of (x, y) rather than a constant. Instead of Eq. (1), we then have a formula for spatially adaptive sharpening:

)),(),((),(),(),( yxIyxIyxAyxIyxI boos −⋅+= (2)

Sharpening in ImageEd

In ImageEd, the user need not select all the sharpening parameters described above. By analyzing the image, the software automatically selects the parameters according to the following principles:

• The radius depends on how much the image is originally blurred. When the image appears only slightly blurred, ImageEd uses a small sharpening radius, while for more blurred images the radius is larger. ImageEd measures blurring by detecting edge features in the image and analyzing the gray levels in the neighborhood of edges. Fig. 9 shows how an edge cross section depends on the blurring.

• The amount of sharpening (A) is spatially adaptive as described above. ImageEd estimates the overall noise level in the image by measuring the variation in gray levels within small regions and analyzing the statistical distribution of this variation. The sharpening amount is then set large at those pixels where the local variation is significantly larger than the estimated noise level. The sharpening amount is zero or almost zero at pixels where the local variation is on the order of the noise level. In other words, the estimated noise level sets a “threshold” parameter for the sharpening. (However, speaking about “threshold” is slightly misleading here because in ImageEd the transition between non-sharpened and fully sharpened pixels is smooth rather than crisp.)

• The maximum A is reduced from its normal value if the image appears to be already sharpened. When analyzing the edge cross sections to determine blurring, ImageEd detects edges that look like Fig. 9 d. If the image contains many such edges, ImageEd treats it as already sharpened.

• The maximum A is also reduced from its normal value if ImageEd finds that a strong JPEG compression artifact is present in the image. JPEG compression divides the image into small blocks (usually 8 × 8 for the luma component) and compresses each block separately. These blocks are visible in highly compressed JPEG images, and sharpening in the normal way would make the blocks even more visible.

In interactive mode, ImageEd allows the user to control the amount and the threshold parameters. Instead of having two separate controls for these parameters, there is a single control that affects both parameters simultaneously in a balanced way.

Detection of unusual images prevents spoiled colors

1-D and 3-D histograms

A commonly used color adjustment method is to expand the tonal range of each of the RGB components separately. The principle of such and operation is shown in Fig. 10a. This operation is easy to implement as an “automatic” function. It just involves calculating the tonal distribution, often called histogram, of each component, using the histogram to find the lowest and highest tones that occur in

Page 6: ImageEd: Technical Overvie · Color adjustment in 3-D space to utilize gamut optimally 1-D versus 3-D adjustments The simplest way of adjusting the colors of an RGB image is to process

- 6 -

the image, and applying an adjustment curve that maps the lowest tone to 0 and the highest tone to the maximum RGB value (255 for 8-bit color data).

Such an operation can correct both bad exposure and color cast. Some image adjustment programs have this kind of function, called “quick fix”, “instant fix”, “auto levels”, or something similar. For many images this works well and is the only operation needed. However, this simple method totally destroys the colors of some images and, therefore, it is much too unreliable to be taken as the basis of automatic processing. The one-dimensional histograms it uses contain too less information of the image. Fig. 10 b displays an example of an image where the simple method fails.

ImageEd also calculates the one-dimensional histograms, because it is necessary to know the endpoints of the tonal range in order to determine the amount which colors can be adjusted without bringing some colors outside the gamut. However, ImageEd also uses many other image properties as the basis of the adjustment. ImageEd analyzes the color distribution in three dimensions, which is used, for example, to find the lightest and darkest colors occurring in the image.

The 3-D color analysis is also the basis for automatic color balance adjustment. If a colorcast is present, the 3-D color distribution has a dominant direction different from the gray axis (cf. Fig. 7). Adjusting color balance in the wrong way is a main reason for destroyed colors, so ImageEd pays particular attention to ensuring that color balance is changed only when this is reliable. From the 3-D color distribution, ImageEd measures several features that are indications of colorcast, and combines these measurements in a manner that results in balance adjustment only when the measures agree. Fig. 11 shows an example of an image where ImageEd avoids destroying colors thanks to its careful analysis.

Spatial analysis

The general challenge in automatic image enhancement is to infer whether a measurable image property is an indication of a flaw that needs correction, rather than a property of the photographed scene. For example, an image can be almost monochrome because it is a close-up of a single colored object, not because is has colorcast. Automatic enhancement must be able to prevent colorcast correction in such a case. (Of course, even a monochrome close-up image may have colorcast, but there is no method for automatic correction using the image information alone, because it is impossible to separate the contribution of the lighting from that of the object color in this case.)

To prevent spoiled colors, ImageEd uses several image measures to detect images that are “unusual” in the sense that automatic adjustment cannot be reliably done in the normal manner. Some unusual images contain only a few colors and are therefore detected using the 3-D color analysis mentioned above. However, the color distribution of unusual images may be quite similar to that of normal images; therefore spatial information is also needed.

The spatial analysis in ImageEd involves measuring the amount of details in different parts of the image and classifying the image as special if the details are distributed over the image in a way that is uncommon. Fig. 12 illustrates how the spatial analysis affects the automatic adjustment. The image in part (b) has been made artificially by dividing the image in part (a) in small blocks and reordering the blocks randomly. Therefore, both images have identical histograms, but due to the spatial analysis, the scrambled image is detected as special and adjusted with caution.

Fuzzy classification logic

Of course, natural images cannot be strictly classified as either “normal” or “special”; some images are more special than other images. In an “either-or” classification scheme, the “dividing line” between normal and special images would have to be drawn rather arbitrarily, and if an image falls close to this line, a small variation in the image can take it to the opposite side. This is undesirable because it would make the adjustment to behave in a “random” manner for some images.

ImageEd uses fuzzy logic to avoid such a lack of stability. Fuzzy logic means that an object (an image in this case) can belong simultaneously to several classes. The object has a degree of membership in each class. For example, a “very unusual” image has unity membership in the “unusual” class and a zero membership in the “normal” class. An image that is “rather unusual” has a nonzero membership value in both classes. For images belonging to several fuzzy classes, ImageEd does the automatic adjustment as a weighted combination of several methods, the relative weights being determined by the memberships. This ensures that small variations in the input image cause only small changes in the result image.

Page 7: ImageEd: Technical Overvie · Color adjustment in 3-D space to utilize gamut optimally 1-D versus 3-D adjustments The simplest way of adjusting the colors of an RGB image is to process

- 7 -

Training based on a large image set The automatic functions in ImageEd have been developed using several thousand sample images varying in source and content. For this development, special computer programs have been made to display images to users and to let them manually adjust the images to optimum, or to select the best-looking image among several candidates. These programs have recorded the users’ choices, and based on these results, the automatic operations have been “trained” so as to simulate the human observer’s preferences. Several persons have participated in this subjective testing, so the behavior of ImageEd reflects the “average” preferences of observers.

The internal computation in ImageEd is partly based on artificial neural networks. A neural network can be regarded as a black box that takes certain input values and produces some outputs. In this case, the inputs are properties measured from the original image, and the outputs are parameters for color adjustment. The network has been trained by presenting it with a large number of sample images and the desired output values, which are obtained from the subjective testing. The training adjusts the internal parameters of the black box so that the trained network produces outputs close to human preferences.

Special adjustments work locally in color space In addition to the Basic Adjustments done with slider controls, ImageEd has “Special Adjustments” for color fine-tuning.

The principle of the special adjustments is shown in Fig. 13. (This does not apply to the Overall tint adjustment, which is global in nature, resembling the color balance adjustment.) When the user selects a point in the picture, the colors of this point and its neighborhood define a local color space region where the adjustment is effective. The selected color and its nearby colors are moved slightly towards a target color. To avoid producing artificial contours in the image, the distance moved in color space must depend on the color in a smooth manner. Therefore, the amount of adjustment decreases when the distance from the selected color increases, and the adjustment diminishes to zero at the boundary of the local region.

If the image contains several spatial regions having a color similar to the selected color, the adjustment affects all these regions at once. Therefore, the user need not point to every region separately, as would be the case with mask-based local adjustments.

To determine the shape and extent of the affected color space region, ImageEd analyzes the image data in the spatial neighborhood of the selected point. In this analysis, ImageEd finds the spatial extent of the “object” to which the selected point belongs. The affected color space region is then determined so that it roughly comprises the colors occurring in the object. For this reason, it does matter which one of similarly colored objects the user selects, even though the adjustment itself affects all these objects equally. If there are several possible objects to select, it is best to select the one whose color varies the most, to ensure that the affected color space region become large enough.

The target color which the arrows are directed to in Fig. 13 depends upon the type of the adjustment.

• For Sky, Skin, and Grass, the target color is taken from a set of representative colors that have been obtained by measuring the color of blue sky, human skin, and green grass over a large set of sample images. From these target colors, ImageEd selects one whose lightness corresponds to the color that the user selected from the image. Therefore, these adjustments affect mainly hue and saturation but not lightness.

• For Local tint decrease, the target color is the neutral color (gray tone) whose lightness is the same as that of the selected color. For Local tint increase, the roles of the source and target colors are interchanged. Therefore, Local tint affects saturation and hue while preserving lightness.

• For Local lightness, the target color is a color whose saturation and hue are roughly the same as those of the selected color, but whose lightness is higher or lower, depending on the adjustment direction. Therefore, this adjustment affects mainly the lightness.

ImageEd displays a “prohibitory sign” if the color under the pointer is not an appropriate source color. There are several possible reasons for this situation:

1. The color is already so close to the target color that the adjustment would not make any visible change.

Page 8: ImageEd: Technical Overvie · Color adjustment in 3-D space to utilize gamut optimally 1-D versus 3-D adjustments The simplest way of adjusting the colors of an RGB image is to process

- 8 -

2. The color is almost black, almost white, or otherwise close to the RGB gamut. ImageEd does not allow adjusting such colors because this could decrease the overall contrast or saturation. The Special Adjustments enable small color changes while taking care that the global color properties do not change much from the result of Basic Adjustments.

3. With Local tint decrease, you can adjust colors towards neutral, but you cannot make them exactly neutral. Therefore, the prohibitory sign appears even though the color under the mouse is not yet neutral. This is because ImageEd takes care of preserving neutral colors neutral, in order not to produce an unwanted colorcast in regions other than the selected one. Since neutral colors are not allowed to change, smoothness of the adjustment requires that colors close to neutral must not change either.

Page 9: ImageEd: Technical Overvie · Color adjustment in 3-D space to utilize gamut optimally 1-D versus 3-D adjustments The simplest way of adjusting the colors of an RGB image is to process

- 9 -

Figure 1

(a) Sequential adjustment steps in conventional programs

(b) Basic adjustments in ImageEd

Figure 2

One-dimensional color adjustments in RGB space

Original image

Adjusted image 1

Adjusted image 2

Final image

User parameters

User parameters

User parameters

User parameters

...

Original image

Adjusted image

Color balance Lightness Sharpness

Contrast Saturation

R

G

B

Adjusted RGB image

Original RGB image

1-D adjustment curves

Page 10: ImageEd: Technical Overvie · Color adjustment in 3-D space to utilize gamut optimally 1-D versus 3-D adjustments The simplest way of adjusting the colors of an RGB image is to process

- 10 -

Figure 3

One-dimensional color adjustments in a transformed space

Figure 4

Three-dimensional color adjustments in RGB space

L

S

HLS image

1-D adjustment curves

Adjusted RGB image

Original RGB

image

AdjustedHLS

image

RGB→HLS transform

HLS→RGB transform with mapping of

out-of-gamut colors

Original RGB image

3-D adjustment operation

R, G, B R´

Adjusted RGB

image G´

R, G, B

R, G, B

Page 11: ImageEd: Technical Overvie · Color adjustment in 3-D space to utilize gamut optimally 1-D versus 3-D adjustments The simplest way of adjusting the colors of an RGB image is to process

- 11 -

Figure 5

(a) A 1-D lightness adjustment curve (b) An original image

(c) The effect of applying the 1-D curve to B and R, shown in the B-R plane

(d) The 3-D lightness adjustment in ImageEd

L

Another line of constant chroma (C > 0)

R

Gray line (chroma C = 0)

B X

Y

black

white

R

B black

white

Y Y´

X X´

Lightness change of color Y

Lightness of color Y is increased less to preserve chroma

Page 12: ImageEd: Technical Overvie · Color adjustment in 3-D space to utilize gamut optimally 1-D versus 3-D adjustments The simplest way of adjusting the colors of an RGB image is to process

- 12 -

Figure 6

(a) 1-D chroma adjustment with gamut problem, shown in the B-R plane

(b) 1-D chroma adjustment with lightness defined differently to avoid the gamut problem

(c) The 3-D saturation adjustment in ImageEd

Chroma line C = C1

R

Gray line (chroma C = 0)

B

X

Y

black

white

Chroma line C = C2

Lines of constant lightness

Chroma line C = C1

R

Gray line (chroma C = 0)

B X

Y

black

white

Chroma line C = C2

Lines of constant (computational) lightness

R

Gray line

B X

Y

black

white

Lines of constant lightness

Page 13: ImageEd: Technical Overvie · Color adjustment in 3-D space to utilize gamut optimally 1-D versus 3-D adjustments The simplest way of adjusting the colors of an RGB image is to process

- 13 -

Figure 7

(a) Color balance adjustment with linear component curves

(b) Attempt to apply nonlinear component curves to an image where R has saturated

(c) Manual color balance adjustment of the image in (b) with ImageEd

R

B

R component clipped to maximum

R

B

R

B

R

B

R

B

Page 14: ImageEd: Technical Overvie · Color adjustment in 3-D space to utilize gamut optimally 1-D versus 3-D adjustments The simplest way of adjusting the colors of an RGB image is to process

- 14 -

Figure 8

Hue changes when the same nonlinear curve is applied to R, G, and B

Figure 9

Cross sections of edges

(a) An ideal, unblurred edge (b) A slightly blurred edge

(c) A substantially blurred edge (d) A sharpened edge

R´, G´, B´

R, G, B

Dark reddish brown, R becomes more dominant→ hue shift towards red

R1 G1 B1

R2G2B2Light reddish brown, R becomes less dominant → hue shift towards yellow

Gray level

Pixel coordinate

Page 15: ImageEd: Technical Overvie · Color adjustment in 3-D space to utilize gamut optimally 1-D versus 3-D adjustments The simplest way of adjusting the colors of an RGB image is to process

- 15 -

Figure 10

(a) Principle of simple automatic color adjustment based on 1-D histograms

(b) A close-up image adjusted with the simple automatic method and with ImageEd

R

G

B

Histogram based

ImageEd

Page 16: ImageEd: Technical Overvie · Color adjustment in 3-D space to utilize gamut optimally 1-D versus 3-D adjustments The simplest way of adjusting the colors of an RGB image is to process

- 16 -

Figure 11

A close-up image adjusted automatically with two methods

Photoshop CS2 Auto Color

ImageEd

Page 17: ImageEd: Technical Overvie · Color adjustment in 3-D space to utilize gamut optimally 1-D versus 3-D adjustments The simplest way of adjusting the colors of an RGB image is to process

- 17 -

Figure 12

(a) Automatic adjustments applied to a real image

(b) Automatic adjustments applied to an image having the same pixels in different order

Figure 13

Principle of special adjustments G

Y

R

M

B

C Color of selected point

Target color

Page 18: ImageEd: Technical Overvie · Color adjustment in 3-D space to utilize gamut optimally 1-D versus 3-D adjustments The simplest way of adjusting the colors of an RGB image is to process

- 18 -

Photo copyrights Jussi Heikkinen: Figures 10 a, 12

Juha Katajamäki: Figures 7, 10 b, 11

Antti Taskinen: Figure 5

Anygraaf USA, Inc. 5235 Westview Drive, Suite 100 Frederick, MD 21703 (240) 379-6620 http://www.anygraaf.com [email protected]