On Comparing Color Spaces for Food...

On Comparing Color Spaces for Food Segmentation

Sinem Aslan, Gianluigi Ciocca, and Raimondo Schettini

Università degli Studi di Milano-Bicocca

Outline

1. Motivation

2. Employed scheme

▪ Food Dataset

▪ Workflow of the segmentation technique (JSEG)

3. Parameter setting for JSEG

4. Experimental results

5. Conclusions

Sinem Aslan ([email protected]) 2

mailto:[email protected]

A general pipeline of calorie calculation by vision-based measurements

PreprocessingFood

segmentation

Food

recognition

Calorie

measurement

Greatly influence

the accuracy of

the subsequent

stages

•Noise & blur

removal, color

correction,

cropping, resizing

•Determining the

boundaries of

food regions

•Feature

extraction and

classification

•Estimating the

mass of the food

and calorie

calculation

Food

segmentation

I. Motivation

We employed color features



Literature works on food segmentationPublication Color space Segmentation scheme

Shroff et al. [1] Gray-scale Adaptive thresholding

He et al. [2] Gray-scale Active contours, normalized cuts, and local variation

performed on the detected foreground.

Zhu et al. [3] Gray-scale Normalized cuts performed on the foreground region,

that is tuned regarding to the resulted food recognition

performance.

Anthimopoulos et al. [4] CIELAB Mean-shift performed on the plate region which is

detected by RANSAC.

Bettadapura et al. [5] CIELAB and gray-scale Hierarchical segmentation with the GPS-based location

heuristics providin assumptions for segmentation.

Ciocca et al. [6] CIELUV JSEG on the detected food regions.

Matsuda et al. [7] CIELUV and gray-scale Regions that are segmented by Deformable Part

Model, a circle detector and JSEG are integrated

according to the score of classification accuracy they

provide.

We need to compare a variety of color spaces:

• On the same dataset

• Using the same segmentation scheme

automatically cropped UNIMIB2016 images [6]

JSEG [8]

I. MotivationSinem Aslan ([email protected]) 4


Food dataset: Automatically cropped UNIMIB-2016 images

▪ Wide variety of food types:

▪ 1,027 tray images (2629 cropped images) including 73 food categories;

▪ Evaluation with more precise ground truth:

▪ In addition to the bounding box annotations, polygon annotations are published.

▪ Challenging for segmentation.

multiple food in a plate

foods in same color with the plates

illumination changes

“noise” objects around ROI

II. Employed schemeSinem Aslan ([email protected]) 5


Segmentation scheme: JSEGmentation

• JSEG is a automatic color image segmentation method which

follows spatially guided approach:

• objective is to form pixel groupings that are homogeneous

from a spatial standpoint.

• Employs protocols involving region growing and merging.

• Works at perceptually uniform CIELUV color space.

We have preferred JSEG, since:

• It has been used successfully in many literature works

• Published source code yields modification on the

method conveniently



Parameters specified by the user:

• TQ : Color quantization threshold




Parameters specified by the user:

• TQ : Color quantization threshold

• N : Number of scales of J-Images

• TM : Region merge threshold




▪ User specified parameters directly influence the segmentation results:

▪ Low values of TQ and TM encourage over segmentation.

▪ Finer details are segmented with higher values of N and vice versa.

▪ Suggested default values [8] are TQ = 250 (CIELUV), TM = 0.4, N:automatic.

• Transforming the input images to other color spaces requires to

update the fixed value of TQ, while N and TM would not get affected

from this operation.

• a new criterion for color quantization: resulting number of

clusters (TC) after merging operation (instead of minimum

distance (TQ) between quantized colors).




Parameter setting for JSEG

▪ Default setting at the original implementation:

▪ TM = 0.4, N: automatic

▪ We have followed two approaches for setting of TC:

1. Fixed scheme of parameter setting. Fix the TC as the performance approaches to the default parameter setting, i.e., TQ = 250, for images in CIELUV color space [11].

2. Optimized scheme of parameter setting. Learn the value of TC from a training set (200 randomly sampled images) for each color space individually.

3. Parameter setting for JSEGSinem Aslan ([email protected]) 10


1. Fixed scheme of parameter setting

Performance results, in terms of (Fboundary + Fregion)/2, that are obtained with default setting of TQ and different settings of TC.

In comparison with TQ = 250 setting, the closest and slightly better performance is obtained with TC = 4.

4. Experimental resultsSinem Aslan ([email protected]) 11


• Boundary based Fscores: Y′DbDr = CIELUV > rgb in both image sizes.• Covering, PRI and VOI scores: Y′DbDr > CIELUV > rgb in both image sizes. • Among all CIEXYZ is the worst in all experiments.

1. Fixed scheme of parameter settingResults



• Boundary-based performance is improved for most of the color spaces,

• e.g., 6%, 5%, and 2% improvement for rgb, CIELUV and Y’DbDr.

• Region-based scores are improved significantly for all color spaces,

• e.g., around 15%, 16%, 10% and 20% improvement for rgb, CIELUV,

Y’DbDr and CIELAB.

2. Optimized scheme of parameter settingComparison with fixed scheme



• Boundary-based Fscrore:

• rgb = CIELUV > Y’DbDr (128pix); rgb > CIELUV > Y’DbDr (256 pix).

• Region-based scores: rgb > CIELUV > Y’DbDr

2. Optimized scheme of parameter settingComparison of color spaces



• Optimized scheme outperforms benchmark in the rates of 6% and 10% at

boundary-based Fscore.

• Improvement in region-based performance is even more remarkable, i.e.,

in the rates of 20%.

2. Optimized scheme of parameter settingComparison with benchmark



Conclusions

▪ In this paper we focused on color space selection for food segmentation.

▪ More precisely, an extensive comparative evaluation of ten color encoding scheme and spaces is made by using the well-known JSEG segmentation algorithm.

▪ We have investigated the optimal parameter setting for JSEG to work in different color spaces.

▪ Our main outcome is that significant improvements in segmentation can be achieved with a proper color selection, and by learning the proper setting of segmentation parameters from a training set for jseg

▪ Experimental results show that representations in rgb Y’DbDr is to be preferred for food segmentation.



Thank you



References

1. Shroff, G., Smailagic, A., Siewiorek, D.P.: Wearable context-aware food recognition for caloriemonitoring. In: Proc 12th IEEE Int. Symp. on Wearable Computers (ISWC'08). (2008)

2. He, Y., Khanna, N., Boushey, C., Delp, E.: Image segmentation for image-based dietary assessment: A comparative study. In: Proc. IEEE Int. Symp. on Signals, Circuits and Systems (ISSCS'13). (2013)

3. Zhu, F., Bosch, M., Khanna, N., Boushey, C.J., Delp, E.J.: Multiple hypotheses image segmentation and classication with application to dietary assessment. IEEE J. Biomed. Health Inform. 19(1) (2015)

4. Anthimopoulos, M., Dehais, J., Diem, P., Mougiakakou, S.: Segmentation and recognition of multifood meal images for carbohydrate counting. In: Proc. IEEE 13th Int. Conf. on Bioinformatics and Bioengineering (BIBE'13). (2013)

5. Bettadapura, V., Thomaz, E., Parnami, A., Abowd, G.D., Essa, I.: Leveraging context to support automated food recognition in restaurants. In: Proc. IEEE Winter Conf. on Applications of Computer Vision (WACV'15). (2015) 580{587

6. Ciocca, G., Napoletano, P., Schettini, R.: Food recognition: A new dataset, experiments, and results. IEEE J. Biomed. Health Inform. 21(3) (2017) 588{598

7. Matsuda, Y., Hoashi, H., Yanai, K.: Recognition of multiple-food images by detecting candidate regions. In: Proc. IEEE Int. Conf. on Multimedia and Expo (ICME'12). (2012) 25{30

8. Deng, Y., Manjunath, B.: Unsupervised segmentation of color-texture regions in images and video. IEEE Trans. Pattern Anal. Mach. Intell. 23(8) (2001) 800{810



An additional slide: Evaluated color spaces

Color space Explanation

Y′IQ

Y′CbCr

Y′PbPr

Y′DbDr

Taking advantage of human vision’s sensitivity to changes on the

luminance component

CIEXYZ

CIELAB

CIELUV

Device independent

O1O2O3 O1 and O2 are independent of highlights, but sensitive to surface

orientation, illumination direction and illumination intensity, O3 has no

invariant property.

I1I2I3 Color information is separated into three approximately orthogonal

components at I1I2I3 and it is reported as useful for segmentation

rgb invariant to surface orientation, illumination direction and intensity


On Comparing Color Spaces for Food...

Documents

Transcript of On Comparing Color Spaces for Food...