1

iLog: An Intelligent Device for Automatic FoodIntake Monitoring and Stress Detection in the IoMTLaavanya Rachakonda, Student Member, IEEE, Saraju P. Mohanty, Senior Member, IEEE and Elias Kougianos,

Senior Member, IEEE.

Abstract—Not knowing when to stop eating or how muchfood is too much can lead to many health issues. In iLog, wepropose a system which can not only monitor but also createawareness for the user of how much food is too much. iLogprovides information on the emotional state of a person alongwith the classification of eating behaviors to Normal-Eatingor Stress-Eating. Chronic stress, uncontrolled or unmonitoredfood consumption, and obesity are intricately connected, eveninvolving certain neurological adaptations. We propose a deeplearning model for edge computing platforms which can auto-matically detect, classify and quantify the objects from the plateof the user. Three different paradigms where the idea of iLogcan be performed are explored in this research. Two differentedge platforms have been implemented in iLog. The platformsinclude mobile, as it is widely used, and a single board computerwhich can easily be a part of network for executing experimentswith iLog-Glasses being the main wearable. The iLog model hasproduced an overall accuracy of 98% with an average precisionof 85.8%.

Index terms— Smart Home, Smart Healthcare, Smart Liv-ing, Food Intake Monitoring, Stress-Eating, Stress-Level Anal-ysis, Internet of Medical Things (IoMT), Machine Learning

I. INTRODUCTION

Food intake monitoring applications which allow an individ-ual to track the consumption of food have been of substantialinterest in the technology community [1]. Long term effectsof overeating on humans include obesity and polycystic ovarysyndrome (PCOS). Obesity occurs due to caloric imbalance inone’s diet. i.e., the amount of calories consumed exceeds theamount of calories burned. It is a condition where a person’sweight is higher than the normal weight in regards to theirheight [2]. One of the main reasons for this excess calorieintake is overeating or not timing the food intake with properintervals during the day.

Indirect factors of overeating include depression, imbalancein emotions and many other psychological dysfunctions suchas stress. When a person is stressed, the human body releases ahormone called cortisol which will increase the appetite of theperson. This hormone creates cravings for sugar rich foods andcomfort foods [3]. Uncontrollable over consumption of highcaloric foods is termed as Stress-Eating in iLog.

The Internet of Medical Things (IoMT) or Internet of HealthThings (IoHT) is a collection of multiple medical devices

L. Rachakonda is with the Dept. of Computer Science and Engineering,University of North Texas, E-mail: rl0286@unt.edu.

S. P. Mohanty is with the Dept. of Computer Science and Engineering,University of North Texas, E-mail: saraju.mohanty@unt.edu.

E. Kougianos is with the Dept. of Electrical Engineering, University ofNorth Texas, E-mail: elias.kougianos@unt.edu.

independently connected to the Internet through wireless com-munication, with capability to exchange data with cloud basedservers. IoMT which has simple sensors or medical devices asthe basic elements, is the backbone of smart healthcare. Anautomatic IoT Edge level system to help a person differentiatebetween Stress-Eating and Normal-Eating is presented in iLog.The basic overview of the iLog system is shown in Fig. 1.

IoMT

Cloud

Stress State

Analysis

Automatic

Processing

Fig. 1. System Level Overview of iLog.

iLog provides an automated stress level detection method-ology, as explained in Section IV, that considers food habits,thereby helping users stay healthy. The main objectives of iLogare the following:

1) User’s Convenience: There are many applications whichhelp in monitoring direct food intake, as mentioned inSection II-A. However, there are very few options thathave the user’s convenience as the main motivation.

2) No User Input: The state-of-art in this area addressessemi-automated methods of monitoring food, as ex-plained in Section II-A but the quantity of the consumedfood must be always provided by the user. Proposing ano user input system was our second objective.

3) User Education: Educating the user on when to eat,what to eat, and how much to eat is our next objective.Helping the user understand is important because thishelps in the reduction of consumption for high caloricfoods.

4) Comprehend the Phrase “Stress-Eating”: Chronic stressis one of the factors that may contribute to the develop-ment of obesity [2], [4]. If the stress persists chronically,the appetite levels along with the cortisol levels remainunchanged. Additionally, when a person is stressed, thegut microbiota in the stomach secrete hormones whichincrease cravings for sugar-rich foods [5]. Food con-sumption under such uncontrollable cravings is referredto as “Stress-Eating.” Helping the user comprehend thedifference between Stress-Eating and Normal-Eating is

2

the main objective.The organization of the paper is as follows: Section II

discusses existing related research and the novel contributionsof the current for its advancement. Section III provides a broadoverview of the proposed food monitoring system. Section IVoutlines the learning algorithm and automatic processing ofiLog. Section V provides the automatic stress state analysisand the data considered in iLog. Section VI presents theimplementation of the model of the proposed system. SectionVII is used to validate the individual results and also providea comparative analysis. The paper concludes in Section VIII.

II. RELATED PRIOR RESEARCH AND CONTRIBUTIONS OFTHE CURRENT PAPER

A. Related Prior Research

There are a number of mobile or web applications andelectronic gadgets that attempt to help users to monitor theirdaily food intake [6]. Some are presented in Table I. A semi-automatic IoT-enabled system that considered current foodintake and predicted future food intake to maintain healthydiet is proposed in [7]. However, the automatic quantificationalong with the stress state relationship is not observed.

There are few visual approaches which use external cameracaptures and processes the food volume and food weightestimation [8], [9]. However, these approaches do not use theIoT and also fail to explain the relationship between stressand food consumed. Though [10] uses a visual approach, theautomatic food quantification is not provided. Another semi-automatic system is proposed in [11], which allows users toobtain nutritional information before consuming the food. Thisrequires manual input and no information about stress and foodalready consumed is provided.

A vibration-based detecting system using a microphoneembedded in an ear pad for food classification is providedin [12], [13] which however, is not accurate and there is nocorrelation to stress.

Gyroscope and accelerometers have been used in [14] and[15]. However, the precision of the project is not high, andthe classification and weight of the food consumed are notaccurate. This also did not correlate with stress levels. Theswallowing signal of the food is recorded and evaluated by theacoustic method [16], [17]. This is not accurate in calculatingthe food weight. Also, the classification of food, relationshipwith stress and food is not present.

The jaw movements of the person is detected, the force usedby the jaw is sent to a piezoelectric film sensor and the amountof food consumed is detected [18]. By swallowing detection,the motion of throat skin is captured [19]. However, this doesnot classify the type of food consumed and the relationshipwith the stress is missing.

The major issues that are not properly addressed in most ofthe current research are:• Users are not educated regarding the relationship between

Stress-Eating vs Normal-Eating.• Automatic food classification and quantification methods

were not provided. For example, in classification, even

after the related food was listed, the user is required tomanually select the item from a list of items.

• Analysis of the variations of stress level with the foodconsumption habits is not presented.

• Techniques to control the variations of stress levels of theuser are not provided.

• A convenient food monitoring wearable was not pro-posed.

• Automatic leftover food consideration is not provided.• Fully-automated systems were not proposed by any of

the articles cited above.

B. Proposed Solutions of the Current Paper

iLog provides solutions to the research objectives stated byproviding:• Automatic continuous monitoring of food detection, clas-

sification and quantification consumed with no user input.• A method which educates the users on the difference

between Stress-Eating and Normal-Eating.• Techniques to control the variations of stress.• An interface representing the stress state analysis of the

user by showing the time of last meal, total number ofcalories consumed, and the number of calories that couldbe consumed for that day.

• Different approaches where a mobile application is usedas a front end for user access to the data, while theprocessing is done in the edge device.

• Allowing the user to access his/her own predicted stresslevels from the previous days are provided with access todatabase storage.

C. Novelty and Significance of the Proposed Solutions

The novelty and significance of the proposed solutions are:

• An automated food logging method without the user’smanual entry by proposing iLog-Glasses which capturethe images of food throughout the course of the meal.This is convenient by not having to take pictures of thefood for every meal, for which there is high chance ofgenuine forgetfulness.

• Automatic food quantification without the user having toenter the quantity of food consumed for every meal.

• Providing a stress state analysis by considering the factthat the user may or may not have completed the foodon the plate, i.e. the leftover food is also quantified toproduce accurate analyses.

• Allowing the user to fully analyze and comprehendhis/her stress levels based on previous logged datathrough an interface.

• Providing a fully-automated edge level device to monitorstress behavioral variations with no user input will helpin promoting a healthy life style.

A detailed representation of iLog is provided in Fig. 2. Thecamera with which the images of food are taken can be placedanywhere according to the comfort of user.

3

TABLE IMOBILE APPLICATIONS FOR FOOD INTAKE MONITORING.

App Name Manual Im-age Entry

Manual DataEntry

Bar-codeScanning

CaloriesConsumed

Drawbacks Stress-EatingClassification

See How You Eat Yes Yes No Yes No automatic classification and requiresmanual entry of portion size

No

Lose it Yes Yes Yes Yes Requires manual food portion size andno automatic classification

No

YouAte Yes No No No Automatic Food classification and man-ual entry of portion size

No

MealLogger Yes Yes No Yes Needs manual entry of portion size andlacks automatic classification

No

BiteSnap Yes No No Yes Automatic Food classification is notprovided and manual entry of portionsize is needed

No

iLog (Proposed) No No No Yes None Yes

Reference

Image

Edge Platform

Classification

Feature

Extraction

Object

Detection

Nutrition Dataset

Stress-Eating

Analysis

Mobile

Application

Interface

Cloud

Platform

Database

Fig. 2. A Detailed Representation of iLog.

III. ILOG: A SYSTEM LEVEL OVERVIEW OF ANIOMT-BASED APPROACH TO RECOGNIZE AND MONITOR

STRESS-EATING

The system level overview of the concept proposed in iLogis shown in Fig. 1. The camera which is attached to theglasses, and takes continuous images once the power supply isenabled. These captured images are sent to a platform wherethe automatic processing of the images will be done. After theprocessing, the stress state analysis of the person takes place.The analyzed outcome is sent to the mobile phone where theprocessing is done. This analyzed data is also sent to the cloudserver for storage. The architecture of the iLog platform isshown in Fig. 3. The processing of iLog can be performed notonly on high computing devices such as In-Cloud units [10]but can also be performed on edge level devices as discussedin Section III-B and in very low performance computingparadigms like sensors and single board computers.

Cloud

Services

Input Unit

iLog as In-Sensor Unit

Local Area Network (LAN)

User

Interface

(UI) Unit

End/Sensing Devices

Edge / Fog Plane

Router/

Gateway

Edge Data Center (Edge Router)

12

3iLog as In-Edge Unit

iLog as In-Cloud Unit

Internet

Fig. 3. An Architectural Overview of iLog Showing 3 Computing Paradigms:(1) Cloud-based, (2) Edge-based, and (3) Sensor-based.

A. Relation between Stress and Food Consumption

The behavioral relationship between stress and food con-sumption is compulsive by considering various factors likethe type of eating, metabolic rate, role of insulin, and foodaddictions [20]. Similarly, when a person is under stress itis likely to have a tendency to eat high calorie “comfort”foods [21]. Chronic stress affects not only the quantity andtiming of food intake but also the choice of foods, whichmay even lead to depression and inflammation and influencemetabolic responses a human body undergoes in stress [22].The quantities and nutrition facts with which the analysis ofStress-Eating is done is explained in Section V-A.

B. Edge Platform based Approach for Automatic Processingin iLog

The In-Edge based approach for stress state analysis of theperson considering the eating habits is done by analyzing theimage data from the camera through Wi-Fi. The received datais automatically processed inside the edge level device. Thecomputations are all performed in the device and the stressstate analysis is performed 4. The analyzed data is sent to theInternet cloud for storing and the user is also notified throughthe notification in mobile application. The daily statisticsof food intake along with their respective nutritional factsconsidered in analyzing (see Sec V) are presented in themobile application as an interface for the user’s convenience.

C. Edge platform based Board Approach for Automatic Pro-cessing in iLog

The In-edge board approach works with the data from thecamera which is considered as an input. The images are sent tothe wireless embedded platform where automatic processingis done. Subsequently, the stress state analysis of the person isperformed and is sent to the user’s mobile as a text message,as represented in Fig. 4.

4

Cloud

Storage

Image

Input Unit

Stress State

Analysis

Unit

Object

Classification

Unit

Object

Detection

Unit

Object

Quantification

Unit

Wi-Fi

Module

Automatic Processing

Fig. 4. Automatic Processing in iLog.

IV. THE PROPOSED TRAINING METHODOLOGY FORMACHINE LEARNING MODELS OF ILOG

A. Design of the Learning Model

The design of the learning model that is used in iLog isrepresented in Fig. 5. The package printed nutritional facts ofthe food to be consumed can be considered as inputs throughOptical Character Recognition (OCR) or bar-code scanning.These detected images are classified and then quantified usingthe same model. After the quantification of food is done, thecalorific information that is consumed is calculated along withthe time of the meal.

Automatic food quantity estimation

Nutrition Facts of the Food item

Start

Calculate caloric information of all the

food items in the meal using a machine

learning model

Classify the food items using a machine learning model

Computer

Vision

Methods

using

Machine

Learning

Models

Perform accurate analysis of stress level

to distinguish between Normal-Eating

and Stress-Eating

Food Items

Fig. 5. Proposed food intake identification and quantification flow in iLog.

B. Automatic Classification of Food from the Plate

Once the images are obtained, the next important stageis to classify the objects in the images. For this, graphicalannotation tools are used with which the detected objects arelabeled to certain food items. The classification using this toolis shown in Fig. 6. For some images where the object is interestis being covered by another random subject, suppressing theinitial errors and image subtraction techniques are used. Thus,the background object is segmented. From that, the mappingfunction is used to restore the image.

C. Automatic Object Detection From Plate

Once the images have been obtained, they undergo segmen-tation where the pixels of the image are segmented to groupsin order to better identify the object. After the segmentationphase is done, the pixels that belong to the same set of visualdescriptors are arranged together by various methods. PrincipalComponent Analysis (PCA), is one of the methods that is used

Fig. 6. Automatic Classification of Targeted Images Using Graphical Annota-tion Tools for Stress Analysis in iLog. Top Row: Single Object Classification;Bottom Row: Multiple Object Classification.

to cluster the data set by finding a principal set of dimensions.Subsequently, the images are labeled and classified in themachine learning model used for system. For the process ofautomatic object detection, an object detection API is used.In the model, assessment of a classification program alsoknown as training is done with a set of pre-labeled images. Ingeneral, a multi layer topology of nodes with weights and biasvalues are considered in the learning stage of the model. Theseweights, bias values and the activation functions involved ateach node and layer, help in predicting the value at the nextlayer or node as shown in Eq. (1). Given a layer i and itsvalues (x)i, the next layer j with values (h)j can be derivedas:

(h)j = f((W )j , i · (x)i + (b)j , i), (1)

where f is the activation function, (W )j , i is the weight matrixand (b)j , i the bias. The selection of the activation functiondepends upon the type of application the model is being trainedfor. The process of object detection along with classificationis presented in Algorithm 1.

D. Automatic Calorie Quantification

The detected, classified and labeled images are assignedwith a unique identification number. All the nutrition in-formation regarding the food items being detected is storedin a database. Based on the repetition of the objects inthe plate, the identification number is iterated. The calorieinformation of the consumed food is derived based on theidentification number. If the identification number is repeated,the nutrition information is also iterated with respect to theidentification number. Thus, the quantities of foods consumedcan be processed without any manual inputs. The processedinformation is then analyzed according to Table II, as shownin Fig. 7. The method of quantification is also presented inAlgorithm 2 under the assumption of a person eating a donutand a bagel in a meal.

E. Metrics for Automatic Processing in iLog

The metrics considered for the model are its Precision,Recall, Accurate Precision (AP), and Confidence [23]. In orderto understand these, basic concepts such as True Positive (TP),

5

Algorithm 1 Process of Object Detection and Classificationused in iLog Model

1: Images which are to be used for testing and training themodel are collected.

2: The formats of the images are converted from JPEGto XML after creating bounding boxes by using anygraphical image annotation tool.

3: Multiple bounding boxes in various images for the samefeature, which are called priors are also created in thesame annotation tool.

4: Using box-coder, the dimensions of priors are made equal.5: By considering the concept of IOU (Intersection Over

Union), the matched and unmatched thresholds for match-ing the ground truth boxes to priors are set. This ismandatory as the model will not be ready for trainingif the match hasn’t been made.

6: Images in XML format are made equal in size either byusing reshape or resize functions.

7: Using convolution and rectified linear functions, the fea-ture maps are assigned to every image sent to the model.

8: Based on these features, the images are either sent toregression or classification where the objects are detectedthrough boxes in the images.

9: Repeat the above steps for all the images.

Images

Object Detection

Object Classification

Assign food item ID

Setup Counter

Number

Check the

calorie count

Does ID

repeat?Multiply

Increment

Counter Number

w.r.t ID number

Is Object

Detection

done?

Stress State Analysis Unit

Multiply

Check the

calorie count

No

No

Yes

Yes

Fig. 7. Automatic Calorie Quantification of Classified Objects for StressAnalysis in iLog.

False Positive (FP), False Negative (FN) and True Negative(TN) need to be defined. In the following discussion, theintersection over union (IOU) is a measure that evaluatesthe overlap between two bounding boxes. It is given by theoverlapping area between the predicted bounding box and theground truth box divided by the area of union between them.• True Positive (TP): A correct detection. This occurs when

the detection with IOU is greater than or equal to the setthreshold value.

• False Positive (FP): A wrong detection. This occurs whenthe IOU is less than the set threshold value.

• False Negative (FN): This occurs when a ground truth isnot detected.

• True Negative (TN): This is the possible outcome wherethe model correctly predicts the ground false. This is not

Algorithm 2 Process of Automatic Calorie Quantification usedin iLog Model

1: Initialize the random variables D1 and B1 to zero.2: Obtain the detected, classified object and assign it to a

variable or ID number D1 (assumption that the object isa donut).

3: Assign value count of D1, vD1 to one.4: Retrieve the nutrition fact information stored with ID D1

from database. Therefore, calories associated with that ID,D1 (cD1) are obtained.

5: Continue to the next object that is classified.6: if object is already assigned to ID then7: iterate the variable vD1 one time8: Multiply vD1 with calorie information i.e., cD1

9: else10: Assign new ID for the detected classified object B1

(assumption that the new object is a bagel)11: Repeat step 4 with variable B1 .12: end if13: Repeat the above steps for all the classified images.14: Update the total calculated calorie (tc) count for each

individual ID along with the time of consumption in thedatabase.

considered in object detection classifiers because thereare many possible bounding boxes that should not bedetected within an image.

1) Precision: The ability of a model to identify only therelevant objects from an image is known as its precision (P ):

P =

(TP

TP + FP× 100%

). (2)

2) Recall: The ability of a model to identify all the relevantcases from the detected relevant objects is called recall (R):

R =

(TP

TP + FN× 100%

). (3)

3) Confidence: Confidence is used to rank the predictionsand quantify the relationship between prediction and recall aswe consider increasing numbers of lower ranked predictions.Instead of presenting a single error code, a confidence intervalCI is calculated:

CI = z

√(α · (1− α)

Nsample

), (4)

where Nsample is the sample size, z is a critical value fromthe Gaussian Distribution, and α is the accuracy obtained.

4) Accuracy: The accuracy of a model can be defined asthe ratio of correct detections made by the model to all thedetections made by the model:

α =

(TP + TN

TP + TN + FN + FP

)× 100%. (5)

5) Precision-Recall Curve: The performance of the objectdetector can be evaluated with the use of the precision-recallcurve as the confidence is changed by plotting a curve for eachobject class. An object detector model is considered good if

6

the precision and recall values remain high even after there isa change in the confidence threshold.

6) Average Precision: Average Precision (AP ) is the preci-sion averaged across all the recall values between 0 and 1. APhelps in evaluating the performance of a model by calculatingthe area under the curve of the precision-recall curve. In orderto obtain this, interpolation is performed, as discussed in [24].n point interpolation summarizes the shape of the precision-recall curve by averaging the precision at a set of 11 equallyspaced recall levels [0, 0.1, 0.2, ...., 1], as defined in Eq. (6)where ρ(r′) is the measured precision at recall r′:

AP =

(1

Nsample

) ∑rε0,0.1,0.2,..,1

ρinterp(r). (6)

=

(1

Nsample

) ∑rε0,0.1,0.2,..,1

maxr′.r′>r

ρ(r′). (7)

V. AUTOMATIC STRESS LEVEL DETECTION FROMFOOD-INTAKE MONITORING

Once the calorie quantification is derived, the data is com-pared with the threshold values for distinguishing betweenStress-Eating and Normal-Eating, as shown in Fig. 8.

Classified

Object Data

Total Calorie

Value

Calories

>2330 & time

interval

<6hours?Normal-Eating:

Update status in

mobile application

Stress-Eating:

Update status in

mobile applicationSend to

server for

storage

Timestamp of

the last meal

Repeat the

process

NoYes

Fig. 8. An Example for Automatic Stress State Analysis of Classified Objectsfor Stress Analysis in iLog for Men.

A. Factors Considered in the Analysis of Stress State Detec-tion

In order to analyze the eating behavior of the person, thefollowing data are considered:

1) The type of foods that are consumed during the awakeperiod is very important as it defines the emotional stateof the person. When analyzing the emotional behaviorof the user with respect to food consumption, pleasuringfoods such as sugar, salt and fat are instantly gratifyingbut have negative effect on mood and temper during thecourse of the day [25].

2) The amount of food that is being consumed by theperson is another important factor along with the typeof food as uncontrollable eating behavior is not onlyconsidered as an abnormality but also as a symptom forstress-eating.

3) The time at which the food is being consumed is alsoimportant. Irregularity between meals can be used as afactor for Stress-Eating.

4) The gender of a person is also equally important toanalyze Stress-Eating behaviors because the body re-quirements change accordingly.

B. Food Data Collection for Stress State Detection

The recommended calorie intake per day is in the range of1600-2000 for women and 2000-3000 for men, respectively.This also includes calories which are generated from sugars,carbohydrates, proteins, etc. [26]. Each gram of sugar, carbo-hydrate and proteins yields 4 calories/gram and each gram offat yields 9 calories/gram [27]. An excess of 20 calories/dayleads to an increase of 1 kilo by the end of the year [28]. Also,the average time for the food to pass through the stomach,small and large intestine is 6-8 hours. So preferably a timegap of 6-8 hours is required for the digestive system to passalong the food [29]. Considering all these factors, the thresholdvalues set to analyze Stress-Eating are represented in Table II.

TABLE IIANALYSES OF STRESS-EATING

RecommendedCalories/day

Sugars (gm/day) TotalCalories

Timeinterval(hours)

Stress-Eating

Men: 2330 37.5grams of sugaror 150 calories

2500 6 Stress-Eating

Women:1830

37.5grams of sugaror 150 calories

2000 5 Stress-Eating

The data that is used to train the iLog model is taken froman online database [30]. The sample information that is beingconsidered in iLog is described in Table III.

TABLE IIINUTRITION FACTS CONSIDERED TO ANALYZE STRESS-EATING

Food Item Food Type Portion AddedSugars

Calories

Jelly Donut Dessert 1 51.70 221

ChocolateCupcake

Dessert 1 49.70 240.64

French Toast Dessert 3.609”across

15.45 157.95

Muffin Dessert 1 3.714 136.30

Pancake Dessert 1medium5” across

0 183.20

Waffle Dessert 1 round4” across

3.15 102.96

Coffee Cake Dessert 1 slice 29.20 159.80

ChocolateChipCookies

Dessert 1 2”across

12.57 48.10

Brownie Dessert 1 2” sq 47.98 128.86

Ice cream Dessert 1 cup 95.20 267.30

7

VI. IMPLEMENTATION OF ILOG USING EDGE COMPUTINGPLATFORMS

The system level block diagram of iLog is shown in Fig. 9.The input JPEG format images will be captured at the iLogglasses and through Wi-Fi connectivity, the images are sentto the edge platform where the classification of features, andtheir extraction is done. Using the extracted features, objectsfrom the images are detected. Once the object detection isperformed, the data is sent and compared in the databasewith the nutrition dataset. From here, using cloud platformas base, the analyzed data is sent to the edge platform wherein the stress-state analysis is done and the data are sent tothe mobile application. A large variety of foods along withtheir nutritional values are taken from [30] for the analysisof Stress-Eating in this model. A total of 130 classes havebeen defined in order to classify the detected food objects.The model has been trained with 1000 images. These imageswere all license-free and are collected from websites whichprovide copyright-free images such as Pixabay and Freepik.Out of 1000 images, 800 images are used for training themodel while the remaining 200 are used for testing.

The collected classified data is sent to the Firebase databasewhere the analysis of the data is done according to Table II.

Inputs Images

from Camera

Wi-Fi

connectivity

Edge

PlatformClassification

Feature

Extraction

Object

Detection

DatabaseNutrition

Dataset

Stress-

Eating

Analysis

Cloud

Platform

Mobile

Application

Fig. 9. System Level Representation of iLog.

A. Mobile Computing Platform

As one of the edge computing platforms, a mobile phoneis considered for availability and usability to the users. Theimages from iLog glasses are sent to the mobile phone throughWi-Fi connectivity where the model will be implemented. Thedetection and classification of objects from a plate along withtheir confidence rates are shown in Fig. 10.

Fig. 10. Object Detection Result.

The gender of the user can be noted while setting-up thewearable. Based on the received information and analysis, the

eating behaviors of the user are reported accordingly to theuser using a mobile application as an interface. The log of theuser’s meals are provided along with the eating behaviors andsuggested remedies if the user is experiencing Stress-Eating.The data is also stored in the cloud platform where the usercan login to have an access to the previous data, as shown inFig. 11.

Carrier 8:45PM

Login

iLog

07/05/2019

9:00 AM

• Pancake - 2 - 340cal

• French Toast-1-70cal

• Coffee- 1- 42cal

12:00 PM

• Noodles - 1 - 250cal

• Yogurt- 1- 240cal

• Ice cream- 1- 300cal

7:00 PM

• Burger - 1 - 350cal

• Yogurt- 1- 240cal

• Ice cream- 1- 300cal

Total:2132cal.

Review

07/05/2019

0

Calories Left:

Techniques:

Stress-Eating

Carrier 8:45PM

Log

07/05/2019

07/04/2019

07/03/2019

Logout

Carrier 8:45PM Carrier 8:45PM

Logout

Welcome Sarah!

Fig. 11. In Edge Stress-Eating Demonstration for a Female User.

B. Single Board Computer Platform

As one possible edge platform, single board computers(SBC) can be used. The camera is considered as the inputsource for images and the data is automatically processed inthe SBC. The size of the processor used prevents it frombeing placed on the pair of glasses, but the glasses can bea part of the wearable network connected the with Internet,as represented in Fig. 1. The object detection result using anSBC is shown in Fig. 12

Frame

Pudding: 95%

Sherbet: 94%

Whipped Cream: 97%

Fig. 12. Object Detection Result using SBC.

C. Characteristics of iLog

The average measuring details of automatic processing isrepresented in Table IV.

VII. VALIDATION OF ILOG

The TensorFlow Object Detection API is used for thetraining and testing the model. SSD (Single Shot MultiBoxDetector) MobileNet is used as the classifier as the frameper second capability is very much higher than the Region-Convolutional Neural Network (RCNN) classifier. The totalnumber of layers defined inside this model is 7 with a batch

8

TABLE IVACCURACY RESULTS OF ILOG

Food Item CI(%) for 5 foldcross validationwith 15 epochs

CI for 5 Repeated5 fold cross valida-tion with 15 epochs

Pancake 92 97

Donut 94 98

French Toast 89 95

Ice Cream 90 97

Brownie 91 95

Waffle 88 96

Muffin 90 98

Cake 92 96

Chocolate 91 95

Cookies 90 95

size of 24 images for every iteration of training at an initiallearning rate of 0.004. The activation function used at eachlayer is ReLu i.e., Rectified Linear Unit function.

The total accuracy of iLog is obtained as 97% approxi-mately. The pattern of maintaining the accuracy and trackingthe loss in detecting the images is shown in Fig. 13. Lossis representing the rate at which the system is optimized. Theaccuracy curve shows the pattern in which the system is trainedand has reached the highest level of detecting the images.

TensorBoard SCALARS GRAPHS INACTIVE

Tooltip sorting method:

Smoothing

Horizontal Axis

Runs

=training\.

=training:logdir

Accuracy

1

Loss

Show data download links

Ignore outliers in chart scaling

default

0.679

STEP RELATIVE WALL

Write a regex to �lter runs

TOGGLE ALL RUNS

0.000.1000.2000.3000.4000.5000.6000.7000.8000.9001.001.10

0.000 500.0 1.000k 1.500k 2.000k 2.500k 3.000k 3.500k 4.000k

0.500

0.600

0.700

0.800

0.900

1.00

(a) Accuracy

TensorBoard SCALARS GRAPHS INACTIVE

Tooltip sorting method:

Smoothing

Horizontal Axis

Runs

=training\.

=training:logdir

Loss 1

Loss

Show data download links

Ignore outliers in chart scaling

default

0.679

STEP RELATIVE WALL

Write a regex to �lter runs

TOGGLE ALL RUNS

0.00

0.200

0.400

0.600

0.000 1.000k 2.000k 3.000k 4.000k

0.5000.5500.6000.6500.7000.7500.8000.8500.9000.9501.001.05

0.000 500.0 1.000k 1.500k 2.000k 2.500k 3.000k 3.500k 4.000k

(b) Loss

Fig. 13. Accuracy and Loss of iLog.

A. Validation of In-Edge based Approach using Mobile PhoneIn-Edge implementation of the iLog is presented here. With

a total of 1000 images, 800 images are used for training while

the remaining are used for testing the model. With 300 objectsbeing detected from 200 testing images, the precisions andrecalls are calculated within the model, while the precision-recall curve for In-Edge implementation is provided in Fig.14.

Using Eqn. (6), the average precision of the In-Edge imple-mentation is 81.8%. The total accuracy of the model withinEdge is 98% approximately.

B. Validation of Single Board Approach

The SBC implementation of the iLog is represented here.With 300 objects being detected from 200 testing images, theprecision and recalls are calculated within the model while theprecision-recall curve for SBC implementation is provided inFig. 14.

In-edge

00.20.40.60.811.2

0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45P

reci

sio

nRecall

(a) Precision x Recall for In-Edge Implementation

In-sensor

0.40.50.60.70.80.911.1

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7

Pre

cisi

on

Recall

(b) Precision x Recall for Single Board Implementation

Fig. 14. Precision x Recall for iLog.

Using Eqn. (6), the average precision of the In-Sensorimplementation is 85.8%. The total accuracy of the modelwithin Sensor is 97% approximately. While the accuracy of themodel is independent of the precision of the model, the averageprecision and accuracy can be increased with an increasednumber of training and testing images or the dataset beingused to train and test the model.

C. Comparison of the Approaches

A brief comparison of the approaches proposed by iLog andStress-Log [10] is presented in Table V. The Table presentsthe type of automation, processing capabilities and systemcharacteristics.

D. Comparison with Existing Research

Though there are several studies conducted to prove andverify the connection between eating behaviors to stress of aperson, as explained in Section II-A, there are very few studies

9

TABLE VSPECIFICATIONS OF ILOG FOR DIFFERENT PLATFORMS

Characteristics MobilePlatform

SBC-1 SBC-2

Camera 16MP;1080x2340pixels

5MP;640x480

5MP;640x480

Accuracy 98% 97% NA

Average Precision 81.8% 85.8% NA

Object Detection Yes Yes Yes

Object Classifica-tion

Yes Yes Yes

Object Quantifica-tion

Yes Yes Yes

Input Type Image Image Image

Automation Fully Auto-mated

Fully Auto-mated

Fully Auto-mated

which provide a continuously monitoring system in order tokeep track of day-to-day stress. Some of these studies, alongwith this work are presented in Table VI.

TABLE VICOMPARATIVE ANALYSIS OF SELF-MONITORING SYSTEMS

Research Inputs DevicePrototype

Self-Analysis

Fully Auto-mated

Harrison,et.al [31]

Pictorialstroop task,Emotionrecognitionfrom images

No Notpossible

NA

Beijbo, et.al[32]

Images No Cannot beachievedcompletely

Semi-automatic

Jiang, et.al[11]

Images No Not muchhelpful

semi-automatic

Taichi, et.al[33]

Images No No nutritionInfo.

No, requiresmanual in-put

Pouladzadeh,et.al [34]

Images No Not muchhelpful

Semi-automatic,no quantifi-cation

Rachakonda,et.al [10]

Images Yes Yes Semi-automatic

iLog (Cur-rent Paper)

Food intake Yes,iLog-glasses

Yes FullyAutomaticsystem

VIII. CONCLUSIONS AND FUTURE RESEARCH

Stress monitoring is one of the most important aspects ofsmart healthcare for lifestyle management, considering the im-pact of stress on overall health and well-being of individuals.

The approach presented here provides an extension to themonitoring systems by focusing on the eating behaviors of theusers and analyzing if the behavior is Stress-Eating or Normal-Eating. The proposed idea of iLog is implemented usingedge platforms, a mobile platform and an SBC platform. Thefoods from the images are automatically detected, classifiedand quantified. The quantified foods are then compared tothe stored database of the nutrition in Firebase and the useris provided with feedback using a mobile application as aninterface. The average precision and accuracy of the twofully automated approaches are 81.8%, 98% and 85.8%, 97%,respectively.

The use of glasses and the embedded platform friendlysoftware allows an advancement in the state of the art. iLogcould be the answer to the need for watching the foodbehaviors and their impact on overall physical and mentalhealth. Also, the three different paradigms, In-Cloud, In-Edgeand In-Sensor, based on computational capability and perfor-mance are explored in iLog. Further light weight machinelearning models might be needed to implement iLog in micro-controllers with very less computing capabilities. In-Sensorprocessing of deep learning technologies could be consideredas future work.

IX. ACKNOWLEDGMENTS

The authors acknowledge the assistance of Texas Academyof Math and Science (TAMS) student Arham Kothari andUniversity of Pittsburgh faculty Dr. Madhavi Ganapathirajufor their help on the preliminary version of this work.

REFERENCES

[1] J. Wei and A. D. Cheok, “Foodie: Play with Your Food Promote Inter-action and Fun with Edible Interface,” IEEE Trans. Consum. Electron.,vol. 58, no. 2, pp. 178–183, May 2012.

[2] K. A. Scott, S. J. Melhorn, and R. R. Sakai, “Effects of Chronic SocialStress on Obesity,” Curr. obese. repp., vol. 1, pp. 16–25, Mar. 2012.

[3] S. D. Hewagalamulage, T. K. Lee, I. J. Clarke, and B. A. Henry, “Stress,cortisol, and obesity: a role for cortisol responsiveness in identifyingindividuals prone to obesity,” Dom. Anim. Endo., vol. 56, pp. S112 –S120, 2016.

[4] N. Rasheed, “Stress-associated eating leads to obesity,” Int. J. HealthSci., vol. 11, no. 2, pp. 1–2, 2017.

[5] R. J. Seeley, A. P. Chambers, and D. A. Sandoval, “The Role of GutAdaptation in the Potent Effects of Multiple Bariatric Surgeries onObesity and Diabetes,” Cell Metabolism, vol. 21, no. 3, pp. 369–378,Mar. 2015.

[6] T. Vu, F. Lin, N. Alshurafa, and W. Xu, “Wearable Food IntakeMonitoring Technologies: A Comprehensive Review,” Comp., vol. 6,no. 1, pp. 1–28, 2017.

[7] P. Sundaravadivel, K. Kesavan, L. Kesavan, S. P. Mohanty, andE. Kougianos, “Smart-Log: A Deep-Learning based Automated Nutri-tion Monitoring System in the IoT,” IEEE Trans. Consum. Electron.,vol. 64, no. 3, pp. 1–1, August 2018.

[8] M. Sun, L. E. Burke, Z. Mao, Y. Chen, H. Chen, Y. Bai, Y. Li, C. Li,and W. Jia, “eButton: A Wearable Computer for Health Monitoring andPersonal Assistance,” in Proc. 51st ACM/EDAC/IEEE DAC, 2014, pp.1–6.

[9] F. Zhu and M. Bosch and C. J. Boushey and E. J. Delp, “An ImageAnalysis System for Dietary Assessment and Evaluation,” in Proc. IEEEICIP, 2010, pp. 1853–1856.

[10] L. Rachakonda, A. Kothari, S. P. Mohanty, E. Kougianos, and M. Gana-pathiraju, “Stress-Log: An IoT-based Smart System to Monitor Stress-Eating,” in Proc. IEEE ICCE, 2019, pp. 1–6.

[11] H. Jiang, J. Starkman, M. Liu, and M. Huang, “Food Nutrition Visual-ization on Google Glass: Design Tradeoff and Field Evaluation,” IEEEConsum. Electron. Mag., vol. 7, no. 3, pp. 21–31, May 2018.

10

[12] O. Amft and M. Kusserow and G. Troster, “Bite Weight Prediction FromAcoustic Recognition of Chewing,” IEEE Trans. Biomed. Eng., vol. 56,no. 6, pp. 1663–72, Jun. 2009.

[13] O. Amft, “A Wearable Earpad Sensor for Chewing Monitoring,” in Proc.IEEE Sensors, 2010, pp. 222–227.

[14] Y. Dong, A. Hoover, J. Scisco, and E. Muth, “A New Method for Mea-suring Meal Intake in Humans via Automated Wrist Motion Tracking,”App. Psych. and Biofeedback, vol. 37, no. 3, pp. 205–215, Sep 2012.

[15] Thomaz, Edison and Essa, Irfan and Abowd, Gregory D., “A PracticalApproach for Recognizing Eating Moments with Wrist-mounted InertialSensing,” in Proc. of the ACM Int. Joint Conf. on Pervasive and Ubi.Comput., 2015, pp. 1029–1040.

[16] M. Farooq, J. M. Fontana, and E. Sazonov, “A Novel Approach for FoodIntake Detection using Electroglottography,” Physch. Meas., vol. 35,no. 5, pp. 739–51, 2014.

[17] K. Kohyama, Y. Nakayama, I. Yamaguchi, M. Yamaguchi, F. Hayakawa,and T. Sasaki, “Mastication Eff. on Block and Finely Cut Foods Stud.by Electro.” Food Quality and Preference, vol. 18, no. 2, pp. 313–320,2007.

[18] M. Farooq and E. Sazonov, “Detection of Chewing from PiezoelectricFilm Sensor Signals using Ensemble Classifiers,” in Proc. 38th Anl. Int.Conf. of the IEEE EMBC, 2016, pp. 4929–4932.

[19] H. Kalantarian and N. Alshurafa and M. Sarrafzadeh, “A WearableNutrition Monitoring System,” in Proc. 11th Int. Conf. on Wear. andImplant. Body Snsr. Networks, 2014, pp. 75–80.

[20] Y. H. C. Yau and M. N. Potenza., “Stress and Eating Behaviors.”Minerva Endo., p. 255267, 2013.

[21] L. Sominsky and S. J. Spencer., “Eating Behavior and Stress: A Pathwayto Obesity,” Front. in Psych., vol. 5, no. 434, pp. 1–8, 2014.

[22] J. K. Kiecolt-Glaser, “Stress, Food, and Inflammation: Psychoneuroim-munology and Nutrition at the Cutting Edge,” Psych. Med., vol. 72,no. 4, pp. 365–369, 2010.

[23] J. Davis and M. Goadrich, “The Relationship Between Precision-Recalland ROC Curves,” in Proc. 23rd Int. Conf. on Mach. Learn., 2006, pp.233–240.

[24] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zis-serman, “The Pascal Visual Object Classes (VOC) Challenge,” Int. J. ofComp. Visn., vol. 88, no. 2, pp. 303–338, Jun 2010.

[25] N. D. Volkow, G.-J. Wang, and R. D. Baler, “Reward, Dopamine andthe Control of Food Intake: Implications for Obesity,” Tren. in CognitiveSci., vol. 15, no. 1, pp. 37–46, 2010.

[26] Nordqvist, Christian, “ How much food should I eat each day?” Med.News Today. MediLexicon, Int.,, 2018.

[27] R. K. Johnson, L. J. Appel, M. Brands, B. V. Howard, M. Lefevre,R. H. Lustig, F. Sacks, L. M. Steffen, and J. Wylie-Rosett, “Dietarysugars intake and cardiovascular health: a scientific statement from theAmerican Heart Association.” Am. Heart Ass. Nut. Comm. of the Councilon Nut., Phy. Act., and Metab. and the Council on Epidem. and Preven.,vol. 120, no. 11, pp. 1011–20, 2009.

[28] G. Taubes, “The science of obesity: what do we really know about whatmakes us fat? An essay by Gary Taubes,” BMJ, vol. 346, 2013.

[29] B. E. Goodman, “Insights into Digestion and Absorption of MajorNutrients in Humans,” Adv. in Phys. Edu., vol. 34, no. 2, pp. 44–53,2010.

[30] Data.GovData.Gov, “Food-a-pedia,” Oct. 2016, data Retrieved fromDatasets, Data.Gov. [Online]. Available: https://catalog.data.gov/dataset/food-a-pedia

[31] A. Harrison, S. Sullivan, K. Tchanturia, and J. Treasure, “EmotionalFunctioning in Eating Disorders: Attentional Bias, Emotion Recognitionand Emotion Regulation,” Psych. Med., vol. 40, no. 11, pp. 1887–1897,2010.

[32] O. Beijbom, N. Joshi, D. Morris, S. Saponas, and S. Khullar, “Menu-Match: Restaurant-Specific Food Logging from Images,” in Proc. IEEEWinter Conf. on App. of Comp. Visn., 2015, pp. 844–851.

[33] J. Taichi and K. Yanai, “A food image recognition system with MultipleKernel Learning,” in Proc. 16th IEEE ICIP, 2009, pp. 285–288.

[34] P. Pouladzadeh, S. Shirmohammadi, A. Bakirov, A. Bulut, and A. Yas-sine, “Cloud-based SVM for food categorization,” Multimedia Tools andApp., vol. 74, no. 14, pp. 5243–5260, Jul 2015.

Laavanya Rachakonda (S’18) earned her Bach-elors of Technology (B. Tech) in Electronics andCommunication from Jawaharlal Nehru Technolog-ical University (JNTU), Hyderabad, India, in 2015.She is currently Ph.D. candidate in the researchgroup at Smart Electronics Systems Laboratory(SESL) at Computer Science and Engineering at theUniversity of North Texas, Denton, TX. Her researchinterests include artificial intelligence, deep learningapproaches and application specific architectures forconsumer electronic systems based on the IoT.

Saraju P. Mohanty (SM’08) received the bachelor’sdegree (Honors) in electrical engineering from theOrissa University of Agriculture and Technology,Bhubaneswar, in 1995, the masters degree in Sys-tems Science and Automation from the Indian Insti-tute of Science, Bengaluru, in 1999, and the Ph.D.degree in Computer Science and Engineering fromthe University of South Florida, Tampa, in 2003. Heis a Professor with the University of North Texas.His research is in “Smart Electronic Systems” whichhas been funded by National Science Foundations

(NSF), Semiconductor Research Corporation (SRC), U.S. Air Force, IUSSTF,and Mission Innovation. He has authored 300 research articles, 4 books, andinvented 4 U.S. patents. His Google Scholar h-index is 34 and i10-index is125 with 5000+ citations. He was a recipient of 11 best paper awards, IEEEConsumer Electronics Society Outstanding Service Award in 2020, the IEEE-CS-TCVLSI Distinguished Leadership Award in 2018, and the PROSE Awardfor Best Textbook in Physical Sciences and Mathematics category from theAssociation of American Publishers in 2016 for his Mixed-Signal SystemDesign book published by McGraw-Hill. He has delivered 9 keynotes andserved on 5 panels at various International Conferences. He is the Editor-in-Chief (EiC) of the IEEE Consumer Electronics Magazine (MCE).

Elias Kougianos (SM’07) received a BSEE from theUniversity of Patras, Greece in 1985 and an MSEEin 1987, an MS in Physics in 1988 and a Ph.D. in EEin 1997, all from Louisiana State University. From1988 through 1998 he was with Texas Instruments,Inc., in Houston and Dallas, TX. In 1998 he joinedAvant! Corp. (now Synopsys) in Phoenix, AZ as aSenior Applications engineer and in 2000 he joinedCadence Design Systems, Inc., in Dallas, TX as aSenior Architect in Analog/Mixed-Signal Custom ICdesign. He has been at UNT since 2004. He is a

Professor in the Department of Electrical Engineering, at the University ofNorth Texas (UNT), Denton, TX. His research interests are in the area ofAnalog/Mixed-Signal/RF IC design and simulation and in the development ofVLSI architectures for multimedia applications. He is an author of over 120peer-reviewed journal and conference publications.