Food Recognition and Leftover Estimation for Daily Diet Monitoring Gianluigi Ciocca, Paolo Napoletano, Raimondo Schettini

@MADiMa 2015, Genova, Italy, September 8th 2015

Automatic dietary monitoring Nowadays, technology can support the users in keep tracks of their food consumption in a more user friendly way allowing for a more comprehensive daily dietary monitoring.; Recent findings showed that computer vision techniques can help to automatically recognize food and estimate its quantity

Automatic dietary monitoring of canteen customers (based on robust computer vision techniques for food recognition and leftover estimation)

Proposed System // Workflow

Automatic dietary monitoring of canteen customers (based on robust computer vision techniques for food recognition and leftover estimation)

Proposed System // Food recognition

Hough transform

From each patch, a visual descriptor is extracted and submitted to a pre-trained k-NN classifier in order to receive a classification label. The labels of the the patches are then post-processed to remove spurious labels in order to have more homogeneous groups of labels that correspond to the food regions.

40 × 40 pixels.

Automatic dietary monitoring of canteen customers (based on robust computer vision techniques for food recognition and leftover estimation)

Proposed System // Leftover estimation

The personnel of the canteen is bounded to follow the regulations provided by nutritionists in the form of nutritional tables, and to serve a specific amount of food (that depends on its calories and nutrients). This somewhat simplifies the problem of the estimation of the food quantity.

Once we have identified this ratio, the corresponding amount of calories is deduced by the precompiled nutritional tables.

Monitored and recorded the meal of 1000 customers of a real canteen (1000 customers correspond to 2000 tray images, 1000 before and 1000 after the meal) •  Customer behavior: each customer selected 3 dishes from the daily menu that included

15 different dishes; •  Tray acquisition: automatic photographic system that includes a raspberry motherboard,

an embedded camera and a motion sensor. The system automatically detects when the tray has to be acquired.

•  Ground-truth: The annotations have been created using the IAT – image annotation tool

[2], that permitted to draw a polygon around the food:

Experiments // Canteen scenario

before after

2 metrics for food recognition: Standard Accuracy (SA) and the Macro Average Accuracy (MAA)

Experiments // Measures

Denoting NPc the number of positives, i.e., the number of times the class c occurs in the dataset; TPc the number of true positives for class c, i.e., the number of times that the system recognizes the dish c; C the number of classes,

leftover estimation: Error

where wc is the class weight and I is the number of test customers. The class weight is defined as the number of elements of the class divided by the total number of elements.

Automatic dietary monitoring of canteen customers (based on robust computer vision techniques for food recognition and leftover estimation)

Experiments // Results

The system is capable of estimating the relative quantity of eaten food with an average error of about 15 percentage points, with the best and worst cases being 7 and 34 percentage points respectively.

Results achieved on a real canteen scenario are promising with an average accuracy in recognition of about 99%, and and average error in food estimation of 15 percentage points. The proposed food recognition and leftover estimation system can serve multiple purposes: 1.  at the check-out station, the food recognition allows to keep track the eaten food and the

user’s dietary habits;

2.  using the list of recognized foods, an automatic billing procedure can be activated speeding up the check- out;

3.  by evaluation the leftovers, we can better estimate the food intakes in terms of calories ingested.

As a future work, 1.  we will experiment the system on large scale food datasets

2.  we will experiment CNN based features

Conclusions //