GOAL-SEEKING DECISION SUPPORT SYSTEM TO EMPOWER PERSONAL WELLNESS

GOAL-SEEKING DECISION SUPPORT SYSTEM TO EMPOWER PERSONAL

WELLNESS MANAGEMENT

A Dissertation

Presented to

The Graduate Faculty of The University of Akron

In Partial Fulfillment

of the Requirements for the Degree

Doctor of Philosophy

Mukesh Kumar Chippa

December, 2016

GOAL-SEEKING DECISION SUPPORT SYSTEM TO EMPOWER PERSONAL

WELLNESS MANAGEMENT

Mukesh Kumar Chippa

Dissertation

Approved:

AdvisorDr. Shivakumar Sastry

Committee MemberDr. Nghi H. Tran

Committee MemberDr. Igor Tsukerman

Committee MemberDr. William H. Schneider IV.

Committee MemberDr. Victor Pinheiro

Accepted:

Interim Department ChairDr. Joan Carletta

Interim Dean of the CollegeDr. Donald P. Visco

Dean of the Graduate SchoolDr. Chand K. Midha

Date

ii

ABSTRACT

Obesity has reached epidemic proportions globally, with more than 1 billion adults

overweight - at least 300 million of them clinically obese and is a major contributor

to the global burden of chronic disease and disability. This can also be associated

with the rising health care costs with in USA alone accounting for more than 75% of

health care costs dedicated to Diabetes and Hypertension. While there are various

technological advancements in building various fitness tracking devices such as fitbit,

etc, and more and more corporations offering wellness programs, they have not been

able to create a long term change in the life style of its users. One of the primary

reasons, such devices fail to create an impact is that these devices are not personalized.

The challenge in keeping healthy people healthy and making them intrin-

sically motivated to manage their own health is at the center of Personal Wellness

Management. In this dissertation, this problem is presented as a decision making

under uncertainty where the participant takes an action at each discrete time steps

and the outcome of the action is uncertain. In this setting, under reasonable set of

assumptions the problem is formulated as a Completely Observable Markov Decision

process and a Partially Observable Markov Decision Process. The results presented

in this highlights the advantages and disadvantages of using each of these frameworks.

One of the major challenges in formulating the wellness problem in the above

mentioned frameworks is the need for clinically validated data. Also, the solutions

proposed for solving the formulated problem, solve for an optimal solution. While it

iii

may be unrealistic to find such experimentally validated data, it is also known that

in solving complex problems such as the PWM, good enough solutions are sufficient.

In this dissertation, Goal-Seeking framework is presented as an alternative to the

above frameworks. Bulk of the thesis is dedicated to document how the Goal-Seeking

framework is different from other frameworks.

This dissertation identifies each of the artifacts in formulating the problem

of Personal Wellness Management. A software system architecture is laid out and

many of the existing software technologies such as Object Oriented design, MySQL,

RESTful API’s etc are leveraged in the implementation of this framework.

iv

ACKNOWLEDGMENTS

I express deep gratitude to my research advisor Dr. Shivakumar Sastry for his solid

mentorship, regular guidance in both technical and non-technical aspects, and inspi-

rational discussions during my course of doctoral studies at the University of Akron. I

am extremely grateful to my dear parents, my wife for their live, courage and endless

support.

I sincerely acknowledge the support from Dr. Victor Pinheiro and Dr. Ju-

dith A. Juvancic-Heltzel in helping me understand various human behavior models

and also in providing resources and coordinating participants during the exercise

performance data collection phase of the project.

I am thankful to all my colleagues at the Complex Engineering Systems

Laboratory(CESL) at the University of Akron, especially Arijit Ghosh, for helping me

understand Bayesian update methods, Hemanth Pidaparthy, Prakash Gaddam and

Sriharsha Vankamamidi, for their continuous support and useful discussions during

various phases of this project. My sincere gratitude also goes to the faculty members

of the Electrical and Computer Engineering department, University of Akron, for

strengthening my knowledge and skills via graduate level courses and workshops.

Without the solid foundation laid by my past teachers and mentors at the

high school level and undergraduate studies, this doctoral dissertation would not have

been a success. Finally, I sincerely acknowledge financial support from the University

v

of Akron, US National Science Foundation (NSF) under the grant #IIS-1237069 that

immensely helped to financially support my doctoral studies.

vi

TABLE OF CONTENTS

Page

LIST OF TABLES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix

LIST OF FIGURES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi

CHAPTER

I. INTRODUCTION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

1.1 Goal-Seeking Approach . . . . . . . . . . . . . . . . . . . . . . . . 3

1.2 Software System Design . . . . . . . . . . . . . . . . . . . . . . . . 4

1.3 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

II. BACKGROUND . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2.1 Personalized Medicine . . . . . . . . . . . . . . . . . . . . . . . . . 7

2.2 Human Weight Dynamics . . . . . . . . . . . . . . . . . . . . . . . 7

2.3 Energy Balance Model . . . . . . . . . . . . . . . . . . . . . . . . 10

2.4 Human Behavior Models . . . . . . . . . . . . . . . . . . . . . . . 12

2.5 Microsoft Kinect Camera . . . . . . . . . . . . . . . . . . . . . . . 14

2.6 Classical Sequential Decision-Making Frameworks . . . . . . . . . 15

2.7 Goal-Seeking Paradigm . . . . . . . . . . . . . . . . . . . . . . . . 20

2.8 Notation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

III. WELLNESS MANAGEMENT FORMULATIONS IN CLASSICALFRAMEWORKS FOR SEQUENTIAL DECISION-MAKING . . . . . . 25

3.1 Markov Decision Processes . . . . . . . . . . . . . . . . . . . . . . 25

3.2 Partially-Observable Markov Decision Processes . . . . . . . . . . 35

vii

3.3 Examples of Wellness Interventions . . . . . . . . . . . . . . . . . 39

3.4 Modeling Issues in Sequential Decision Making Frameworks . . . . 51

IV. GOAL-SEEKING FORMULATION FOR PERSONALIZED WELL-NESS MANAGEMENT . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

4.1 Detailed Goal-Seeking Formulation of PWM . . . . . . . . . . . . 52

4.2 Reflection Mapper Design . . . . . . . . . . . . . . . . . . . . . . . 58

4.3 Bayesian Update of Human Motivation Distribution . . . . . . . . 60

4.4 Simulation Results . . . . . . . . . . . . . . . . . . . . . . . . . . . 62

V. SYSTEMS ARCHITECTURE AND SOFTWARE DESIGN . . . . . . . 68

5.1 Overview of the PWM Software Architecture . . . . . . . . . . . . 68

5.2 Versatile and Scalable Deployment Architecture . . . . . . . . . . 73

VI. EVALUATING AND GUIDING EXERCISE BEHAVIORS . . . . . . . 80

6.1 Supporting Exercise Performance using Kinect Camera . . . . . . 80

6.2 Multiple Kinect Cameras . . . . . . . . . . . . . . . . . . . . . . . 87

6.3 Estimating Ground Reaction Forces during a Vertical Jump . . . . 103

VII. DISCUSSION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106

VIII. CONCLUSIONS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109

BIBLIOGRAPHY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110

viii

LIST OF TABLES

Table Page

2.1 Notation used throughout the thesis. . . . . . . . . . . . . . . . . . . . 23

2.2 Notation used in the MDP Formulation in Chapter 3. . . . . . . . . . . 23

2.3 Notation used in POMDP Formulation in Chapter 3. . . . . . . . . . . 24

2.4 Notation used in Goal-Seeking Formulation in Chapter 4. . . . . . . . . 24

3.1 State Space for Illustrative Example . . . . . . . . . . . . . . . . . . . . 33

3.2 Example Actions for Illustrative Example . . . . . . . . . . . . . . . . . 33

3.3 Policy Computed by Value Iteration . . . . . . . . . . . . . . . . . . . . 34

3.4 Policy Computed by Policy Iteration . . . . . . . . . . . . . . . . . . . 35

3.5 State Space for Illustrative Example . . . . . . . . . . . . . . . . . . . . 36

3.6 Three of the Nine Optimal Policies computed for MDP with re-stricted action sets shown in Figure 3.3. . . . . . . . . . . . . . . . . . . 40

3.7 State Transition Matrix used for Low Intensity Actions . . . . . . . . . 46

3.8 State Transition Matrix used for Medium Intensity Actions . . . . . . . 46

3.9 State Transition Matrix used for High Intensity Actions . . . . . . . . . 46

3.10 Observation Matrix used for Low Intensity Actions . . . . . . . . . . . 47

3.11 Observation Matrix used for Medium Intensity Actions . . . . . . . . . 47

3.12 Observation Matrix used for High Intensity Actions . . . . . . . . . . . 47

3.13 Snapshot of the POMDP policy. The policy had over 250 vectorsto represent the states and only a few are shown in this table as anillustration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

ix

4.1 The motivation of the participant affects the variance of the nutritionand exercise actions as shown below. . . . . . . . . . . . . . . . . . . . 55

5.1 The Goal Seeking Artifacts and the corresponding classes implemented 77

6.1 An illustration of the first 18 values of the 150-dimension featurevector for the Jumping Jacks exercise. . . . . . . . . . . . . . . . . . . . 85

6.2 Potential Errors in Jumping Jacks HICT Exercises . . . . . . . . . . . . 91

6.4 Potential Errors in Wall Sit HICT Exercises . . . . . . . . . . . . . . . 92

6.6 Potential Errors in Push Up HICT Exercises . . . . . . . . . . . . . . . 93

6.7 Potential Errors in Abdominal Crunch HICT Exercises . . . . . . . . . 94

6.8 Potential Errors in Step-up on to Chair HICT Exercises . . . . . . . . . 95

6.9 Potential Errors in Squat HICT Exercises . . . . . . . . . . . . . . . . . 96

6.10 Potential Errors in Triceps dip on Chair HICT Exercises . . . . . . . . 97

6.11 Potential Errors in Plank HICT Exercises . . . . . . . . . . . . . . . . . 98

6.12 Potential Errors in High Knees/Running in Place HICT Exercises . . . 99

6.13 Potential Errors in Lunge HICT Exercises . . . . . . . . . . . . . . . . 100

6.14 Potential Errors in Push-Up HICT Exercises . . . . . . . . . . . . . . . 101

6.15 Potential Errors in Side Plank HICT Exercises . . . . . . . . . . . . . . 102

x

LIST OF FIGURES

Figure Page

1.1 Personalized Wellness Management . . . . . . . . . . . . . . . . . . . . 2

2.1 Joint locations obtained from the Microsoft Kinect 2.0 cameras afterexecuting the skeletal extraction algorithm. . . . . . . . . . . . . . . . . 14

2.2 Overview of MDP. Decisions are made in discrete time epochs basedon the observed state (weight). At each epoch, the weight of theindividual is measured and a corresponding action is recommended.The implementation of the action results in a reward to the indi-vidual and the time advanced to the next epoch. . . . . . . . . . . . . . 17

2.3 Overview of POMDP. Decisions are made in discrete time epochsbased on an estimate of the state of the system. In each state ofthe system the system is said to emit an observation stochastically.The observation and the estimated state is mapped to an actionand recommended. The implementation of the action results in areward to the individual and the time advanced to the next epoch. . . . 19

2.4 Overview of the Goal-Seeking Paradigm. Decisions are made indiscrete time epochs. Since there is no attempt to model the statesof the system, the individual provides a set of feasible actions tothe decision-maker. The decision maker selects one of the feasibleactions based on the Reflection Mapper and establishes a limit forperformance that can be achieved in the interval of time to the nextepoch. At the next epoch, the actual performance is evaluated.If the difference between the expected performance and the actualperformance is within the tolerance bound for that individual, thedecision-maker continues to recommend actions. . . . . . . . . . . . . . 21

3.1 Quantized Action Space used for the MDP. Each action represents aspecific choice of caloric intake (nutrition) and caloric expenditure(physical activity level). . . . . . . . . . . . . . . . . . . . . . . . . . . 27

xi

3.2 An example state transition matrix computed with h = 0.8. . . . . . . . 34

3.3 POMDP Action Space . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

3.4 Block diagram on how POMDP and MDP frameworks are used toselect an action from the action set. After constructing the pomdppolicy, at each time step, depending on the observation, selects thebest intensity level of the action. This intensity level is used tofilter the action set for the MDP framework. Using the humanweight dynamics model and user preferences, the MDP frameworkselects one action among many actions of the same intensity level. . . . 39

3.5 Shows all possible states from Initial to Target. For each state, thebars represent the action recommended. The exercise scale (red) ison the right (PAL) and nutrition (blue) scale is on left (cals/day).This policy was computed using a Linear Reward function. Therecommendation is to eat less and work out more. . . . . . . . . . . . . 42

3.6 The expected weight loss trajectory for a participant who initiallyweighs 120 Kg and follows the policy in Figure 3.5. . . . . . . . . . . . 43

3.7 Actions computed from a policy that rewarded a target rate of weightloss. Higher or lower loss is penalized. The target loss can beachieved via multiple actions. The figure illustrates one stochasticchoice that helps the participant achieve target weight. . . . . . . . . . 43

3.8 Weight loss trajectory and uncertainty regions when executed thepolicy computed for recommended weight loss. . . . . . . . . . . . . . 44

3.9 MDP policy obtained using a reward function that rewarded actionsthat were within 30% of baseline exercise activity level. . . . . . . . . . 45

3.10 Weight loss trajectory for participant not willing to perform highintensity exercises but willing to go on a calorie-restricted diet. . . . . . 45

3.11 Desire to perform : Nutrition : 0.2, Exercise : 0.8 . . . . . . . . . . . . 49

3.12 The weight trajectories for different probability of adherence levelsand its corresponding variations in time to reach target weight . . . . . 50

4.1 Detailed View of Goal-Seeking Interactions . . . . . . . . . . . . . . . . 57

4.2 The Energy Balance Model . . . . . . . . . . . . . . . . . . . . . . . . . 59

4.3 The Reward function used in the goal seeking framework to assigncost (negative rewards) to consequences of each action that is esti-mated through reflection mapper. . . . . . . . . . . . . . . . . . . . . 60

xii



4.6 Expected weight trajectory and its error bars for a participant withNutrition Adherence Level : 0.1 and Exercise Adherence Level: 0.1.Since the adherence level is too small, the participant does not ad-here to the recommended actions and therefore is expected to in-crease weight and never reach the target weight. . . . . . . . . . . . . . 66

4.7 Expected weight trajectory and its error bars for a participant withNutrition Adherence Level : 0.5 and Exercise Adherence Level: 0.5.Since the participant adheres to the recommended action 50% of thetime, the participant neither increases weight nor decreases weightas is evident by the large error bars. . . . . . . . . . . . . . . . . . . . 66

4.8 Expected weight trajectory and its error bars for a participant withNutrition Adherence Level : 1.0 and Exercise Adherence Level: 1.0.In this scenario, the participant adheres to each recommended ac-tion and therefore after few time steps, the goal seeking frameworkrecommends high intensity actions. Since the adherence levels arehigh, the error bars in estimating the wight trajectory to the rec-ommended action is small. . . . . . . . . . . . . . . . . . . . . . . . . . 67

5.1 The Personal Wellness Management System Architecture Design . . . . 69

5.2 Activity Flow in the Participant Dialog Module . . . . . . . . . . . . . 70

5.3 The class diagram of the implementation of Three compartmentmodel of the human body weight dynamics. . . . . . . . . . . . . . . . 71

5.4 Implementation of the three compartment human weight dynamicmodels in JAVA. Simulating a hypothetical Participant aged 27years, 100KG and 1.7 m height, sedentary life style, eating 2000Calories/day and doing moderate intensity exercises will weigh ap-proximately 88.5 Kgs in 180 Days. . . . . . . . . . . . . . . . . . . . . . 72

5.5 Multitier Structure of Database Storage in PWM. . . . . . . . . . . . . 74

5.6 RESTful API for Scalable Deployment of PWM. The web servicesprovided for Goal Seeking resides in the server along with otherservices in the PWM wellness software suite. The client and serverinteract with each other using the JSON data format. . . . . . . . . . . 78

xiii

5.7 An example of client software initiating with a HTTP request touse the reflection mapper web service hosted on the server modulethrough RESTful api. The data required for the server is formattedinto a JSON object and is inserted in post request. The serverafter processing the request, formats its response into another JSONobject and places it into the HTTP Response that is then sent tothe client. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79

6.1 Four Exercises that were selected from the HICT Suite for this study. . 82

6.2 Sample time-series data captured from the Kinect camera for thex-coordinate of a few joints is as shown on the left. The figure onthe right shows the trajectories of the (x, y) coordinates for eachjoint when the Jumping Jacks exercise is performed. . . . . . . . . . . . 83

6.3 Projection of all the joints on the Y Z-Plane for the Jumping Jacksexercise. This was obtained by drawing a line between the aver-age starting position and the average finish position of each jointillustrated in Figure 6.2. . . . . . . . . . . . . . . . . . . . . . . . . . . 84

6.4 Confusion Matrix achieved using the projections-based feature vector . 84

6.5 Comparison of the Standing shoulder adduction exercise performedby an expert participant(on the left) and a beginner participant(onthe right). Observe lot of variations in the wrist position of thebeginner participant. This shows that the beginner participant isnot in control of the exercise and is prone to injury. . . . . . . . . . . 86

6.6 The 3D Joint coordinates from each camera are provided by consid-ering the location of the camera as the origin. When data for thesame exercise are collected using multiple cameras, it is necessaryto translate and/or rotate the coordinates from one camera to theframe of reference of the other camera. . . . . . . . . . . . . . . . . . . 88

6.7 Demonstrating the need of multiple Kinect camera. While a par-ticipant performing Lunges exercise, the Kinect camera 1 has un-tracked frames of ankle joint during brief moments when the par-ticipant bent forward. During this period, Kinect camera 2 has noproblem tracking the joint as its positioned with an angle to theparticipant. Therefore Kinect camera 2 frames can be used in placeof Kinect camera 1, however they need to be transformed to Kinectcamera 1 frame of reference. . . . . . . . . . . . . . . . . . . . . . . . 89

xiv

6.8 Superimposed frames from Kinect camera 1 frames(blue data points)and transformed Kinect camera 2(green data points) data frames forthe ankle joint during a Lunges exercise. Observe the green datapoints cluster highlighted in the circle appear in the absence of bluedata points. The combined data can now be used to analyze if theparticipant made an error during the exercise activity. . . . . . . . . . . 90

6.9 Plot of spine base over a single jump. . . . . . . . . . . . . . . . . . . . 104

6.10 Impulse calculated using force plate, vicon and kinect data. Thegraphs shows that the impulse calculated using force plate data isapproximately equal to impulse calculated using kinect and vicondata. This shows that the change in momentum is conserved duringjumping phase. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105

xv

CHAPTER I

INTRODUCTION

Personalized Wellness Management (PWM) is a critical national priority that has the

potential to significantly reduce healthcare costs that are currently greater than 16%

of the nation’s GDP [1, 2]. Approximately 75% of total health care costs are associ-

ated with management of chronic illnesses such as Diabetes and Hypertension [3]. In

addition to the costs, there is loss of productivity and morale, and families are con-

fronting worsening Quality of Life [2]. To address these growing societal challenges,

the current reactive approach to medical practice , which is focused on disease man-

agement must be fundamentally transformed into a proactive approach that empowers

individuals to manage their personal wellness.

PWM is a complex multi-disciplinary problem that lies at the intersection of

areas such as psychology, exercise science, medicine, behavioral management, nutri-

tion and emerging computing/communications technologies [4, 5, 6, 7, 8, 9]. Although

there are several devices that can assist, monitor, track, or support individuals to

monitor their wellness, such devices have only offered small or incremental advances

in wellness management. As is evidenced by the escalating medical costs and obesity,

such technologies have not been able to create a paradigm shift in the state-of-the-

practice. To address this need, we propose a comprehensive decision-maker for PWM

that accounts for the diversity of individual choices, disease risks, socio-economic sta-

tus, ethnicity, genetic predisposition, and uncertainties in the operating environment

of individuals.

1

Figure 1.1: Personalized Wellness Management

Since PWM affects the health and well-being of individuals, it is very closely

related to the medical profession. In addition, wellness is a complex enterprise that

involves the confluence of many disciplines. For these reasons, it is necessary to

manage wellness in a rigorous and predictable framework. However, in order to

account for the diversity of the individuals and the disease risks, it is necessary to

personalize the wellness care. Ultimately, the objective for the PWM enterprise is

to encourage participants to become intrinsically motivated in the management and

sustenance of their own wellness. The goal-seeking framework we present achieves

this by relying on the principle of bounded-rationality [10], i.e., humans will make

rational decisions if they are presented with the right information that affects their

own wellness at the right time.

Several new approaches have emerged to assist participants in improving

their wellness. For example, the quantified self movement [11] has resulted in several

wearable devices and support systems to track physical activity. Similar advances

2

are being investigated to track and monitor nutrition activity. Classical models for

weight dynamics such as [12] have also been developed in the recent years. Our focus

is on investigating a goal-seeking approach [13, 14] in which there is no attempt to

optimize the exercise and nutrition actions.

1.1 Goal-Seeking Approach

We use a Goal-Seeking formulation as a basis to design a new decision-support to em-

power individuals to manage their personal wellness. The Goal-Seeking formulation

is grounded in a systems paradigm proposed by Mesarovic and Takahara [15] that

has been used extensively to study large-scale and complex systems [16, 17, 18]. In

recent years, this paradigm has been used as a basis to coordinate the management of

a collection of batteries [19, 13]. The highlights of this paradigm are that there is no

attempt to optimize the selection of actions and that uncertainty can be represented

and accounted for explicitly. This paradigm is described in more detail in Chapter 4.

Managing human wellness is viewed as an instance of sequential decision-

making. In this view, the system, i.e., the human, evolves by making a sequence of

actions. For this investigation, we assumed that the weight of a person, which can be

directly measured, is a strong indicator of the person’s wellness 1. Thus, in accordance

with the human weight dynamics model developed at the Diabetes Research Center

of the National Institute of Health [12], the two actions that critically affect an

individual’s weight are the nutrition intake (calories input) and the physical activity

(calories output). When these actions are executed, they result in consequences. The

uncertainties that affect the consequences are the internal metabolic constraints of

the individual and the motivation of the individual to adhere to the selected actions.

1In the future, this assumption can be relaxed by assuming other factors such as metabolism;however, one needs to have validated models for the weight dynamics before such assumptions canbe integrated in a formal decision-support framework such as the one proposed here.

3

Thus, this approach to wellness management fits well with the constructs of the goal-

seeking paradigm as is explained in more detail in Chapter 4. A complete formulation

of the wellness management problem in the Goal-Seeking framework required the

design of a reflection mapper that is described in Section 4.1.4 and a cost mapper ;

the selection of a cost mapper function is also described in the same section.

In order to understand how the goal-seeking approach compares with the

classical approaches to sequential-decision making reported in the literature, alter-

native formulations for wellness management were developed using the framework

of Markov Decision Processes (MDP). In contrast to the goal-seeking approach, the

MDP approach required the creation of a state-transition model for each individual.

An extended MDP framework called the Partially Observable Markov Decision Pro-

cess (POMDP) was used to represent the uncertainties in the formulation. The hu-

man weight dynamic models were incorporated into these formulations and “optimal”

policies were computed. These policies served as a basis for the selection of nutrition

and exercise actions in the MDP and POMDP frameworks. The selected actions and

their consequences were compared with the actions/consequences obtained using the

goal-seeking approach. Chapter 3 presents the MDP formulations and the results

of the comparison. The evaluation mappers that were required were collaboratively

developed with Master’s students in the team and are described in [20, 21, 22, 23];

these works are briefly described in chapter 6 for completeness.

1.2 Software System Design

Another important dimension of the goal-seeking formulation of the wellness man-

agement problem was that the formulation served as the basis for a scalable and

flexible systems architecture for the decision-support system. This robust architec-

ture provided the foundation for several investigations focused on personal wellness

4

management. The systems architecture is described in detail in Chapter 5. Many

elements of this comprehensive software system are being implemented and field trials

based on large populations of participants are being planned based on the decision-

support system that is described in this thesis.

1.3 Contributions

In summary, the contributions of this investigation are

1. The first Goal-Seeking Formulation for empowering Personalized Wellness Man-

agement (PWM).

2. Comparing Goal-Seeking approach and the classical approaches for sequen-

tial decision-making, i.e., Markov-Decision Processes and Partially Observable

Markov Decision Processes.

3. Design of Reflection Mapper for the Goal-Seeking framework for PWM. De-

signed and implemented a reference model for human weight dynamics in Java.

4. Selection of Cost function based on healthy weight loss rate for the Goal-Seeking

framework for PWM.

5. Design of Evaluation Mappers for evaluating and guiding exercise behaviors

6. Goal-Seeking system architecture, storage and software design.

7. Integration and implementation of Goal-Seeking software system ecosystem.

5

CHAPTER II

BACKGROUND

This chapter presents background that is necessary to describe the outcomes of

this investigation. Section 2.1 briefly describes trends in personalized medicine and

motivates the need for personalized wellness management. In order to make wellness

interventions more quantified and rigorous, we need a basis for these interventions.

Literature in the area of human weight dynamics is described in Section 2.2 and

the specific energy-balance model that was used in this investigation is described in

Section 2.3. The goal-seeking decision-support system must operate in the complex

domain of human behaviors. For this reason, well-known theories for human behavior

that have originated in psychology and social sciences are described in Section 2.4.

In order to evaluate the exercise recommendations, an evaluation tool was designed

based on the Microsoft Kinect Camera. Section 2.5 presents an overview of the

capabilities enabled by this device and a review of literature that has used this camera

for capturing and analyzing human motion. The goal-seeking approach that is the

basis for this investigation must be eventually compared to classical decision-making

frameworks. Two classical approaches for sequential decision-making are described

in Section 2.6. A more precise formulation of wellness management using these two

approaches is presented in Chapter 3. The goal-seeking paradigm is briefly presented

in Section 2.7 and a more precise formulation of wellness management based on this

paradigm is described in Chapter 4.

6

2.1 Personalized Medicine

Personalized wellness management that is envisaged by the proposed decision-support

tools is grounded in the idea of personalized medicine which dates back many hun-

dred of years. It is well-known that rapid developments in computational biology

and medical imaging have created new avenues for researchers to create and analyze

personalized diagnosis and treatments [24, 25, 26]. Despite such advances, there is

a long way to go in understanding why different individuals experience disease or

respond to treatment differently. This lack of knowledge has led clinicians prescribe

drugs to a patient based on general information about what might actually work for

that particular patient. If the medication does not work after a few weeks, the pa-

tient might be switched to another medication. This ”trial and error” approach can

lead to patient dissatisfaction, adverse drug interaction and side effects let alone the

money spent. Such a process, based on trial and error, is unfortunately the state-of-

the-practice today in the domain of exercise and nutrition. While there are general

guidelines and principles prescribed [27, 28], there are few quantitative methods that

can conclusively determine how much exercise an individual must perform and what

diet an individual must consume to remain healthy. Thus, the principal objective of

the decision-support system proposed in this thesis is to keep healthy people healthy

by designing personalized wellness prescriptions.

2.2 Human Weight Dynamics

Many factors that affect the wellness of an individual such as inherent disease risk

or metabolic issues cannot be precisely quantified or observed. However, the weight

of a person can be observed via simple measurement. For this reason, weight and

other metrics that can be derived from the weight such as Body Mass Index (BMI)

7

are often used as proxies for the wellness of a person. Using such metrics, recent

reports indicate that about two thirds of the US population is overweight [29]. Such

weight is correlated with chronic conditions such as diabetes [30], hypertension [31]

and heart disease [32]. For this reason, there has been considerable interest to develop

mechanisms and mathematical models that accurately predict outcomes of life style

changes and weight loss intervention strategies [33, 34]. These models offer opportu-

nities to understand how individuals lose and gain weight and can be used to help

patients manage their weight change and improve adherence to life style changes. In

this regard, many researchers have attempted to model human body weight change

dynamics as described in the following paragraphs.

A model proposed by Hill et. al [35] aims to provide a measure of average

daily energy imbalance. The authors assumed that a pound of weight represented

3500 kcal and that excess energy is converted to storage at 50% efficiency. Based on

historical trends, they estimated that the median daily energy imbalance gap between

intake and expenditure needed to explain a population weight gain of 1.8-2.0 lb per

year was about 30 kcal d−1. Another model developed by Swinburn et al. [34] used a

labeling technique to estimate the total energy expenditure (TEE) which the authors

assumed equals total energy intake (TEI) when the weight is stable. The authors

used TEE to estimate the difference in energy flux between states of weight balance.

From this model, Swinburn et al. [36] and Hall and Chow [37] concluded that the

increase in US food supply over the last 30 years was correlated with the increase

in weight in the US population. From such studies it was concluded that a TEI of

about 94KJd−1 would result in a weight increase of about 1 Kg for adults [38].

Weight change does not happen instantaneously. To calculate the time taken

for body weight change, dynamical system models have been developed that use food

8

intake and physical activity levels as model inputs, and calculate changes in energy

expenditure and body weight over time.

The first reference to a two-compartment body composition model that con-

siderd fat mass and lean mass tissue dates back to 1942 [39]. Here, the body mass

was considered to be either fat mass, which had a density of 0.9007 kg/liters, or

lean mass with a density of 1.1 kg/liters. This idea was further used by Forbes [40],

who analyzed body composition data collected across different studies and suggested

that the weight loss is exponential over time. This contribution was later refined

by Antonetti [41] to obtain a model for calculating weight change as a function of

time and caloric intake values. However, these models used constant values for the

Basal Metabolic Rate (BMR) 1 and the activity level; these values resulted in inaccu-

racies in the weight change predictions. Antonetti used an activity coefficient ka to

describe the physical activity performed by the participant [41]. More recently, this

idea is formalized by defining the Physical Activity Level (PAL), which is defined as

PAL = TEE ÷ BMR. Physical activity can also be measured using the Metabolic

Equivalent intensity levels(METs) [44] which expresses the energy expenditure of

physical activity as multiples of Resting Metabolic Rate (RMR) 2. Westerterp [45]

described another two-compartment model, where the changes in fat mass and fat-

free mass are driven by a system of two ordinary differential equations. The model

helped to predict how exercise affects weight gain, and how changes in caloric intake

factors in to the system. The model also accounted for how the body composition

changed over time affects weight change. Other models for human weight dynamics

include [46, 47, 48, 49, 50, 33]. A recent model incorporates both weight and height

dynamics for a wide range of ages and accounts for gender and race [51]. In [52], the

1The minimum rate of energy expenditure that is needed to support nominal body functions [42,43].

2RMR describes the energy needed to maintain the basic physiological processes [42].

9

authors develop a dynamic model for weight dynamics by incorporating an algebraic

relationship between FM and LM that is based on data collected by the Center for

Disease Control. This model is useful for estimating the change in weight as a result

of adherence to recommended nutrition intake.

The main idea in the two-compartment model is that the energy store of the

human body is either fat mass, FM , or lean mass LM . The daily energy balance,

EB(t), i.e., the difference between the calories consumed and the calories expended,

affects these stores as

d FM(t)

dt=

(1− p(t))EB(t)

ρFM, (2.1)

d LM(t)

dt=

p(t)EB(t)

ρLM, (2.2)

where ρFM and ρLM are energy densities and p(t) is the p-ratio that represents the per-

centage of the imbalance denoted by EB to the two compartments, respectively [53].

Hall [54] defined p using the Forbes formula [55] as p = CC+FM

, where C = 10.4 ρLM

ρFM.

2.3 Energy Balance Model

More recently, a three-compartment model was developed by Hall at the National

Institute of Health, Diabetes Research Center [56]. This model is particularly at-

tractive because it was calibrated using the data from the well-known Minnesota

Semi-Starvation Experiment [57] that used 32 healthy men. These men were placed

on a control diet for 12 weeks. They were then on a semi-starvation diet for 24 weeks.

They were then placed on a restricted rehabilitation diet for 12 weeks and on a unre-

stricted rehabilitation diet and 8 weeks. The models proposed by Antonetti [41] and

Hall [56] show a good agreement with the data collected from the Minnesota semi-

10

starvation study. Hall extended the two-compartment model to a more accurate

three-compartment model [37, 58, 59].

The key ideas in the three-compartment model are as follows. The normal

daily energy balance EB(t) is defined as

EB(t) = EI(t)− EE(t), (2.3)

where EI(t) is the energy consumed and EE(t) is the energy expended each day.

The daily energy intake, measured in kilo-calories (kcal), is calculated from the car-

bohydrate intake (CI), fat intake (FI) and protein intake (PI) for the day as

EI(t) = a1CI(t) + a2FI(t) + a3PI(t), (2.4)

where a1 = 4 kcal/gram, a2 = 9 kcal/gram and a3 = 4 kcal/grams. CI, FI and PI

are measured in units of grams/day. The energy expenditure expressed in kcal/day

is calculated as

EE(t) = TEF (t) + PA(t) +RMR(t) (2.5)

where the thermic effect of feeding TEF (t) denotes the energy expended in processing

food and PA(t) is the energy expended in physical activity. The value of TEF is

usually approximated to 10% of the energy intake. The energy expended per day,

PA(t), expressed in kcals, represents the energy expended as a result of conducting

work activities, household tasks, and physical exercise [60]. RMR can be estimated

using the Harris-Benedict prediction equation [61] as

RMR = 66 + (13.75× weight) + (5× height)− (6.76× age), and (2.6)

RMR = 655 + (9.56× weight) + (1.85× height)− (4.68× age), (2.7)

11

for men and women, respectively, where weight is in kg, height in cm, and age is in

years.

This weight dynamics based on this three-compartment model is described

in more detail in Section 4.2 of Chapter 4.

The human body cannot be viewed just as a rechargeable battery because

the nutrition and exercise actions are in fact complex behaviors that are affected by

several factors.

2.4 Human Behavior Models

Factors that affect human behavior have been studied extensively by psychologists

and social scientists and their insights have been codified into the following well-known

theories:

1. Theory of Planned Behavior (TPB) [62],

2. Social Cognitive Theory (SCT) [63],

3. Self Determination Theory (SDT) [64],

4. Transtheoretical Model (TTM) also known as the Stages of Change Model [65],

5. Theory of Reasoned Action (TRA) [66], and

6. Health Belief Model (HBM) [67].

Each of the above theories propose a set of constructs and explain how these con-

structs affect human behavior. These theories account for both environmental and

individual influences. While these theories have been used to explain human behav-

iors at an aggregate level, these models cannot be readily used to offer interventions

12

to individuals. Other models for human behavior are reported in [68]. Recent re-

ports in the literature have highlighted the complexities of using the weight dynamic

models in the context of human wellness programs [69, 70, 71, 72].

Among these theories, the Transtheoretical Stages of Change (SOC) model

was used in this investigation as a representative example in Chapter 3 and Chapter 4.

In the SOC view, the main constructs in SCT are

• Precontemplation

• Contemplation

• Preparation.

• Action.

• Maintenance.

Although these attributes sounds subjective, psychologists use validated in-

struments (surveys) to measure such attributes.

Get Healthcare Systems Book reference to show how this has been

used for a National study.

In the recent literature, human behavior models have been used to augment

the weight dynamic models described in the preceding section [62, 73].

When an individual carries out exercise behaviors as recommended, it is pos-

sible that the individual may get injured because of incorrect actions. It is also

necessary to accurately measure the ground reaction forces in such activities as de-

scribed in Chapter 6. For this reason, there have been several investigations carried

out to understand how an inexpensive, non-invasive device such as the Kinect camera

can be used [21, 20, 23]. The next section describes the Microsoft Kinect camera and

its role in motion capture and analysis.

13

2.5 Microsoft Kinect Camera

In recent years, the Microsoft Kinect Camera has been extensively used to study

human motion. In [74], the performance of dancers is evaluated by comparing it

against a reference standard. In [75, 76, 77, 78], the role of the Kinect camera in

rehabilitation is discussed. In [79], an exercise feedback system which recognizes

exercises using a classifier is discussed. The performance is compared to a reference

to provide real-time guidance and feedback in a tele-rehabilitation system. In [80],

the authors used the joint information to ensure anonymity of the people during

surveillance. In [81], the data obtained from a Kinect camera is analyzed to estimate

the anthropometry of participants. A good survey of the current research trends

using the Kinect camera can be found in [82].

Figure 2.1: Joint locations obtained from the Microsoft Kinect 2.0 cameras afterexecuting the skeletal extraction algorithm.

14

One of the novelties of the Kinect camera is that the standard skeleton track-

ing algorithm [83, 84] provides accurate 3D coordinates of a set of twenty five joints

that are illustrated in Figure 2.1. These data are collected at a rate of 30 frames

per second and hence, we can obtain a fairly accurate trajectory of the joints as the

exercises are performed. In recent work that was closely related to this investigation,

the accuracy of the Kinect camera was shown to be comparable with that of the

Vicon camera, which is considered to be a gold-standard for the capture and analysis

of human motion. Preliminary results that use the joint location data to estimate

ground reaction forces is discussed in [21]. The accuracy of the Kinect camera has

also been compared with other marker-based systems in [85, 86].

There are several reports in the literature that focus on recognizing human

actions using joint locations [87, 88, 89, 90, 91, 92, 93, 94]. Results for recognizing

exercises and errors that can occur during their performance is discussed in [20].

Recent results focus on fusing joint location data from multiple Kinect cameras to

mitigate the effects of occulsions [23].

As discussed in Section 2.2 and Section 2.3, human weight does not change

instantaneously. Thus, an individual participating in exercise and nutrition behaviors

is often faced with the question “What should I do now?”. Thus, the main purpose of

a decision-support system is to help the individual make such decisions. This thesis

presents a goal-seeking decision-support system. There are two classical models for

making such decisions in the literature and these are described in the next section.

2.6 Classical Sequential Decision-Making Frameworks

Markov Decision Processes (MDPs) and Partially-Observable Markov Decision Pro-

cesses (POMDPs) are two classical frameworks for sequential decision-making. In

a sequential decision making setting, the individual must make multiple decisions

15

at different times to achieve an objective. In the context of wellness interventions,

these decisions would correspond to nutrition and exercise actions that must be fol-

lowed at different times. As is well-known, there is considerably uncertainty when

a human being attempts to implement the recommended actions. The MDP and

POMDP frameworks have been designed for modeling decision-making under uncer-

tainty. Such frameworks provide a formal basis for the design of a decision-support

system. These ideas will be introduced in the next two sections and more precisely

defined in Chapter 3.

2.6.1 Markov Decision Processes

MDPs offer a rigorous framework for modeling decision-making under uncertainty [95,

96, 97, 98, 99] when the state of the system is completely observable. MDPs have

been used extensively in the variety of problem settings and applications including

robotics [100], automated control, economics and manufacturing [101, 102], health-

care [103, 104] An MDP formulation for wellness management was presented in [73].

Figure 2.2 illustrates the key ideas of how MDP is embodied in a decision-

support system. In this view, the individual engages in wellness activities by mak-

ing decisions at discrete time epochs and implementing these decisions between the

epochs. Consistent with the energy balance model discussed in Section 2.3, these

decisions involve the selection of an action for nutrition and an action for exercise.

Since the underlying models focus on energy, we use calories as the units for these

actions. Thus, every decision involves selecting an action for nutrition, i.e., number

of calories that must be consumed per day, and an action for exercises, i.e., number

of calories that must be expended per day through exercise. As illustrated in the

figure, the MDP formulation is embodied in the decision support subsystem as will

be explained in more detail in Chapter 3. The MDP formulation assumes that the

state of the system is completely observable. For this reason, we selected the weight

16

of the individual as the state. Thus, in this formulation, at each epoch, the weight

of the individual is measured and the decision-support system selects an action, i.e.,

nutrition-exercise choice, depending on the weight. The MDP formulation, however,

allows for uncertainty in the implementation of the actions. Thus, the weight of the

individual at the next time epoch is not necessarily known until that time to the next

epoch as elapsed.

DecisionSupport

Whatshould Idonow?

N EH

si aj

t

rijk

DecisionSupport

N EH

sk

t+1

Figure 2.2: Overview of MDP. Decisions are made in discrete time epochs based onthe observed state (weight). At each epoch, the weight of the individual is measuredand a corresponding action is recommended. The implementation of the action resultsin a reward to the individual and the time advanced to the next epoch.

A more precise formulation of the MDP for wellness management will be

presented in Chapter 3. The quantized action space and the support necessary to

enable transitions will be described more precisely. The actions that are recommended

in each time epoch are based on an optimal policy that can be obtained by solving

the MDP. An overview of the well-known solution methods and examples of optimal

policies will also be described.

17

2.6.2 Partially Observable Markov Decision Processes

In many applications, the state of the system cannot be observed. For example, a

POMDP is reported in [105] to monitor a production process that cannot be observed

directly and the status must be inferred by observing the quality of the output. A

POMDP is used to maintain a distribution of future energy prices and use this fore-

casted information to schedule the operation of appliances in homes [106]. There

are many such examples in a broad spectrum of applications that range from health-

care [107] to aircraft collision [108]. In this specific context of wellness management,

it is easy to note that the individual’s motivation affects the implementation of both

nutrition and exercise actions. In fact, the motivation of an individual to implement

exercise actions may be different than the motivation to implement nutrition actions.

An extension of MDP in which the state is only partially visible is called a Partially

Observable Markov Decision Process (POMDP).

Figure 2.3 illustrates the key ideas of POMDP. As in the case of MDP, the

POMDP framework would serve as the basis for designing the decision-support sys-

tem that helps the individual in each time epoch. The state is not fully observable

and in each state, the system is expected to emit an observation stochastically. In

this context of wellness management, we used the constructs embodied in the theo-

ries of human behavior described in Section 2.4 as states. Specifically, we used the

Social Cognitive Theory (SCT) as a representative example and used the constructs

supported in this theory as the states. The states used were precontemplation, con-

templation, preparation, action, and maintenance. An underlying state-transition

probability matrix determines how the states change stochastically in response to

an action recommended. In a POMDP model for a system, there is a observation

probability matrix, one for each action. These probabilities govern what observation

is emitted from which state after a specific action is performed. We viewed the ad-

18

herence or otherwise of the individual to a recommended action as two observations

— adhered and not adhered.

Since the state is not fully observable, the POMDP maintains a belief dis-

tribution over the states of the system. Based on its current estimated state and

the observation, it selects an action that is recommended to the individual. The

implementation of the action results in a reward to the individual.

DecisionSupport

Whatshould Idonow?

Estimate(si)aj

t

N EH

M

t+1

DecisionSupport

Estimate(sk)

N EH

M

rijk

O1

Figure 2.3: Overview of POMDP. Decisions are made in discrete time epochs basedon an estimate of the state of the system. In each state of the system the system issaid to emit an observation stochastically. The observation and the estimated stateis mapped to an action and recommended. The implementation of the action resultsin a reward to the individual and the time advanced to the next epoch.

A more precise specification of the POMDP for wellness management that is

based on SCT is presented in detail in Chapter 3. As in the case of MDP, we can

obtain an optimal policy for selecting actions in a POMDP. Examples of such policies

and a comparison of the POMDP policy with that obtained from an MDP is also

discussed.

19

2.7 Goal-Seeking Paradigm

The objective of this investigation was to design a Goal-Seeking decision-support

system.

The goal-seeking paradigm [15] is an approach to modeling and describing

systems that differs from the MDP and POMDP approaches briefly described above.

Both the MDP and POMDP frameworks assumed that states of a system are pre-

cisely describable; and the dynamics of the system are described by a state-transition

function. In the POMDP framework, the observations emitted depended on the state

of the system.

In contrast, there is no attempt to describe the system states in the goal-

seeking paradigm and, hence, the system model is simpler than what one may expect

when using a state-transition paradigm. Instead, the decision-making process is for-

mulated using the following constructs. There is a set of Alternate Actions, Π, from

which the decision-maker can select actions. In the wellness management context,

these actions are nutrition and exercise recommendations as already described in the

MDP and POMDP frameworks. Anticipated system perturbations and disturbances

are represented as a set of Uncertainties, ∆. If a given perturbation δi ∈ ∆ occurs,

it would impact the success of a selected action. Consequences are outputs that are

produced by the system; the set of Consequences, Ψ, includes all outcomes that may

result from the implementation of some action. The decision-maker uses a function

called Reflection, Ξ : Π ⊗ ∆ → Ψ, as its view of the environment. Suppose that

the decision-maker selects an action π1 ∈ Π; the decision-maker uses Ξ to estimate

the consequence, ψ1 ∈ Ψ, that π1 would produce if a given perturbation occurs. An

Evaluation Set, Λ, represents a Performance Scale that is used to compare the results

of alternate actions according to an Evaluation Mapping, Ω : Ψ ⊗ Π → Λ. That

is, if the decision-maker has the option to select one of two actions π1, π2 ∈ Π, and

20

these actions are expected to result in consequences ψ1, ψ2 ∈ Ψ, respectively, then the

decision-maker uses values of Λ as the metric to determine whether one of the two

actions is preferred over the other. Ω is also used to evaluate the actual measured

output of the system. A Tolerance Function, Γ : Π ⊗ Ψ → Λ provides a bound on

how much the performance can vary before being considered as unsatisfactory.

Figure 2.4: Overview of the Goal-Seeking Paradigm. Decisions are made in discretetime epochs. Since there is no attempt to model the states of the system, the indi-vidual provides a set of feasible actions to the decision-maker. The decision makerselects one of the feasible actions based on the Reflection Mapper and establishes alimit for performance that can be achieved in the interval of time to the next epoch.At the next epoch, the actual performance is evaluated. If the difference between theexpected performance and the actual performance is within the tolerance bound forthat individual, the decision-maker continues to recommend actions.

Using these artifacts and transformations, the task of the decision-maker may

be stated as

21

Continue to select an action π ∈ Π as long as the outcome is within

tolerance limits, i.e., Ω(π, ψ) > Γ(π, ψ), for any possible perturbation

δ ∈ ∆.

Rather than finding an optimal policy like the MDP or the POMDP, the

goal-seeking approach aims to find a satisfying solution that is within an acceptable

tolerance limit. Such an approach is useful when it is not possible, or desirable, to

construct a precise model of a system. Consequently, in this paradigm, the control of

a complex system does not require decision-maker that is based on complex models.

It is feasible to design algorithmic modules that offer systematic integration of multi-

disciplinary constraints to address the needs of complex issues such as personalized

wellness management.

In the next chapter, we present more precise MDP and POMDP formulations

for wellness management as envisaged in this investigation. A detailed goal-seeking

formulation is presented in Chapter 4.

2.8 Notation

The notations used throughout this thesis are presented in this section. Table 2.1

shows the notation that is used throughout the thesis.

Table 2.2 shows the notation used in the MDP formulation for wellness man-

agement that is described in Chapter 3.

Table 2.3 shows the notation used in the POMDP formulation for wellness

management that is described in Chapter 3.

Table 2.4 shows the notation used in the Goal-Seeking formulation for well-

ness management that is described in Chapter 4.

22

Table 2.1: Notation used throughout the thesis.

Symbol Descriptionci Nutrition action i. Each action identifies a specific (quan-

tized) caloric intake per day.lj Exercise action j. Each action identifies a specific (quan-

tized) physical activity level for exercise activities. Thislevel can be uniquely mapped to caloric expenditure perday.

< ci, lj > A wellness recommendation; identifies a specific nutritionaction and an exercise action.

t = 1, 2, · · ·NDecision epoch or ”tick” during which a decision is made.U [cmin, cmax] The Uniform distribution of dietary calories/day, that the

participant eats.U [lmin, lmax] The Uniform distribution of physical activity level, that the

participant exercises.

Table 2.2: Notation used in the MDP Formulation in Chapter 3.

Symbol DescriptionSM Set of states.AM Set of actions.pijk The probability of transition from si ∈ SM to sk ∈ SM using

aj ∈ AM .rijk The reward received for transitioning from state si ∈ SM to

sk ∈ SM using aj ∈ AM .Vt(s) The value function, i.e., sum of expected rewards accumu-

lated, when starting from state s ∈ SM and acting optimallyfor a horizon of t steps.

23

Table 2.3: Notation used in POMDP Formulation in Chapter 3.

Symbol DescriptionSP Set of states.AP Set of actions.pijk The probability of transition from si ∈ SP to sk ∈ SP using

aj ∈ AP .rijk The reward received for transitioning from state si ∈ SP to

sk ∈ SP using aj ∈ AP .DP Belief Distribution over POMDP states.O1 Observation that participant adhered to recommended ac-

tion in the POMDP Formulation.O2 Observation that participant did not adhere to recom-

mended action in the POMDP Formulation.poijk Probability of emitting an observation Oi after performing

action aj when the system is in state sk in the POMDPFormulation.

Table 2.4: Notation used in Goal-Seeking Formulation in Chapter 4.

Symbol DescriptionΠ Set of actions. Each πi ∈ Π corresponds to < ci, lj >.∆ Set of uncertainties. The occurrence of δi ∈ ∆ impacts the

consequence of a selected action.Ψ Set of consequences.Ξ Reflection Mapper. Ξ : Π⊗∆→ Ψ.Λ Performance Scale [0 . . . 100].Ω Evaluation Mapper. Ω : Ψ⊗ Π→ Λ.Γ Tolerance Function.DM Motivation Distribution that is used in the Goal-Seeking

Formulation.

24

CHAPTER III

WELLNESS MANAGEMENT FORMULATIONS IN CLASSICAL

FRAMEWORKS FOR SEQUENTIAL DECISION-MAKING

This chapter presents wellness management formulations in the two classi-

cal frameworks for sequential decision-making, namely Markov Decision Processes

(MDP) and Partially-Observable Markov Decision Processes (POMDP). Both these

frameworks require an underlying state-representation to support the decision-making.

In MDP, the state must be fully observable and in POMDP the state is not fully ob-

servable. Uncertainty is captured in both these frameworks by the stochastic transi-

tions between the states. This formulations in this chapter demonstrate how wellness

actions can be selected in these frameworks. The advantages and disadvantages of

these frameworks are discussed.

Section 3.1 presents the MDP formulation. Here, since the state of the system

must be observable, the weight of the individual was selected as a representation of the

state. In Section 3.2, the hidden states are based on the theories of human behavior

discussed in Section 2.4. As detailed in the discussion, the POMDP model maintains

a belief distribution over these states and actions are stochastically selected.

3.1 Markov Decision Processes

An MDP is defined by the four-tuple (SM ,AM , T, R), where SM denotes the states

of the MDP, AM is the Action Space, T is the state transition probability matrix and

25

R is the reward matrix. The construction of each of these artifacts, the computation

of an optimal policy and the use of this policy to select actions for a participant are

described in this section.

3.1.1 State Space

Since the state of an MDP must be completely observable, we used the weight of

the participant as the state. A range of [75 . . . 120] was selected as a representative

example and the weight of the individual was rounded to the closest integer in this

range.

3.1.2 Action Space

The Action Space, AM , was formulated by quantizing the nutrition actions and the

exercise activities. The caloric intake was assumed to be in the range [500, 5000]

calories, in steps of 250 calories. Exercise activity was assumed to be a physical

activity level in the range [1 . . . 3] in steps of 0.1. Recall, this is a dimensionless value

and can be mapped to caloric expenditure as illustrated in Equation 4.4.

Thus, as illustrated in Figure 3.1, the action space AM comprises the set of

quantized nutrition-exercise pairs. As a representative example for this investigation,

using the choice of ranges described above, we had 360 choices in the action space.

3.1.3 State Transition Probability Matrix

Uncertainty in the system was captured by designing stochastic transitions between

the states as described here. The probability of transitioning from state si ∈ SM to

state sk ∈ SM using action aj ∈ AM is denoted as pijk. The probability of transition

between si and sk were represented as a state transition probability matrix, Tj, one

for each action. The construction of Tj is described will be described in more detail.

Since MDP represents how the states of the individual changes, the three-

compartment weight dynamics model described in Section 2.2 was used to construct

26

Figure 3.1: Quantized Action Space used for the MDP. Each action represents aspecific choice of caloric intake (nutrition) and caloric expenditure (physical activitylevel).

T . The duration of time between two epochs was selected as 3 weeks (21 days). To

account for metabolic variability in individuals, we arbitrarily selected a threshold

value h = 0.8. We then assumed that this value represented the chance that the

individual would achieve the weight predicted by the weight dynamics model if the

corresponding action was selected. Since the weights were quantized in steps of 1 kg,

it was easy to identify neighboring states for an action as follows.

Starting in state si, we used the weight as input to the weight dynamics

model; the caloric intake and expenditure corresponding to action aj were also used

as inputs to the model. The model provided a prediction of the state sk that would

27

be attained if action aj was implemented for 3 weeks. We assigned

pijk = h;

the two neighboring states, i.e., sk+1 and sk−1 excluding boundary conditions, were

assigned probabilities as

pijk+1 =1− h

2,

and

pijk−1 =1− h

2.

3.1.4 Reward Matrix

In order to compute an optimal policy using the MDP, it was necessary to design

a reward for each transition. The reward obtained when transitioning from state

si ∈ SM to state sk ∈ SM using action aj ∈ AM was denoted as rijk. The reward is

assumed to be non-negative and bounded, i.e. 0 ≤ rijk ≤ rmax, ∀i, k ∈ SM , aj ∈ AM .

The reward matrix, one for each action aj ∈ AM is denoted as Rj.

Instead of computing a large number (360) of reward matrices, we designed

a two reward functions that returned a real value between 0 (undesirable) and 100

(desirable).

1. Linear Reward: Using the initial and target weights, wi and wt, for the

individual, the slope was selected by assuming that (wi, 0) and the (wt, 100)

were the end points of a line. Given si, sk ∈ SM , the linear reward function was

evaluated at sk.

2. Safe-Loss Reward: This reward function was designed to make sure that an

individual does NOT loose more than 1 kg per week; this is an example of a safe

weight loss policy to ensure that there are no adverse effects on the individual.

28

For this purpose, the reward function was

100− |sk − si − 3| × 10.

Note the reward is maximum if the weight loss in the time epoch is close to 3,

i.e., 1 kg per week.

3.1.5 Computing Optimal Policies

A policy is a function that maps the current state of the MDP to an action. An

optimal policy is one that maximizes the total reward that the individual can earn,

in an expected sense. The purpose of formulating a problem, such as the wellness

management problem, as an MDP is to obtain an optimal policy. The objective of this

investigation was to understand how the goal-seeking approach described in Chapter 4

compares with classical frameworks for sequential decision-making, we assumed that

the policies are stationary, i.e., the policy will not change with time. This means that

whenever the MDP is in a specific state, the action recommended and the next state

that is selected stochastically will not change.

Since the MDP is used make decisions at different (sequential) times, it is

important to specify how the current reward must be traded off with a potential

future reward that can be obtained by selecting the current action. This is captured

by using a discount factor, γ. Thus, given an MDP, the goal is to find a policy P

that maximizes the expected total discounted reward.

Suppose the MDP is allowed to evolve for infinite time, the value accrued by

using a policy P assuming that the MDP starts in a particular state s0 ∈ SM is

V π(s0) = E

[∞∑t=0

γtrijk

],

where 0 < γ ≤ 1.

29

Among all possible policies for an MDP, the policy that maximizes the ex-

pected total discounted reward is called the optimal policy, p∗, and accrued reward

by starting in state s0 is called the optimal value and is denoted as

V ∗(s0) = maxp∈P

V p(s0).

In this application of MDP for wellness management, since the policies are

assumed to be stationary, the optimal policies can be computed using two classical

algorithms, namely Value Iteration and Policy Iteration. Such a policy can be used

to select actions in each time epoch as will be explained.

3.1.5.1 Value Iteration

The key idea in value iteration is to compute the value accrued in each of the states

at time epoch t; for state si, the value of each possible action aj is examined and the

highest value is recorded

Vt+1(si) = maxaj∈AM

∑sk∈SM

pijkrijk + γ∑sk∈SM

pijkVt(sk)

, (3.1)

This expression can be understood by thinking “backwards”; Irrespective of

the value accrued in the preceding t steps, the best action in the step t+ 1 is the one

that provides the highest value. Since the classical value iteration algorithm assumes

the infinite horizon setting, the γ team appears in the above expression to discount

future rewards.

After computing the value for each state, the action that provides maximum

value is selected to be the action for that state in the current policy, i.e.,

π(si) = arg maxaj∈AM

Vt(si). (3.2)

30

The value iteration terminates when the maximum difference between two

successive value functions, called the Bellman Error Magnitude is less than a pre-

specified threshold value, ε). It is known [109] that there exists a t∗, which is poly-

nomial in the size of the state space, |SM | × |AM |, when the magnitude of the largest

value of the largest reward, rijk and 11−γ , are such that the action that maximizes V ∗t

is the optimal action.

Instead of calculating the bound on t in advance and carrying out value

iteration for that period, the iteration is terminated when

|Vt(s)− Vt−1(s)| < ε,∀si ∈ SM .

When the above bound is valid, it is known that [110]:

maxsi∈SM

|Vt(s)− V ∗(s)| < 2εγ

1− γ. (3.3)

3.1.5.2 Policy Iteration

In value iteration, the value of each state is updated in each iteration. In contrast,

policy iteration assumes a random policy for the initial step. Assuming that each

state si ∈ SM is the starting state, compute the value obtained by implementing the

current policy π until the value of each step converges as described the preceding

paragraph. For each state, the value is now

Vt(si) =∑sk∈SM

pijk[rijk + γVt(sk)] ∀si ∈ SM , (3.4)

where the action aj is specified by the current policy π.

31

The current policy is improved by selecting the action, for each state, that

maximized the value in Vt(.), i.e.,

π(si) = arg maxaj∈AM

Vt(si). (3.5)

The value function of a policy is the expected discounted reward that will

be gained at each state by executing that policy. Once the value of each state is

known under the current policy, the option of improving this value by changing the

first action taken is considered. If it can, the policy is changed to take the new action

whenever that state is encountered. This step is guaranteed to strictly improve the

performance of the policy. When no improvements are possible, then the policy is

guaranteed to be optimal. Since there are at most |A||SM | distinct policies, and the

sequence of policies improves at each step, the policy iteration algorithm terminates

in at most an exponential number of iterations. It is known that the running time

is pseudo polynomial and that for any fixed discount factor, there is a polynomial

bound in the total size of the MDP [111].

3.1.6 Illustrative Example

To illustrate the MDP formulation in this section, a specific example is now pre-

sented. In this example, the state space was selected have seven states as illustrated

in Table 3.1. This represents an individual whose weight is in the range 145 kg to

150 kg. The objective of this example is to demonstrate all the artifacts of the MDP

formulation and is not intended to be realistic.

The action space for this example is shown in Table 3.2. Note there are only

three possible actions as indicated in this table.

32

Table 3.1: State Space for Illustrative Example

State Weight (kg)s1 145s2 146s3 147s4 148s5 149s6 150

Table 3.2: Example Actions for Illustrative Example

Action Nutrition(Cals) Exercise(PAL)a1 2500 1.2a2 2750 1.3a3 2250 1.4

The state transition probability matrix was constructed using the three-

compartment human weight dynamics model and is illustrated visually in Figure 3.2.

Notice that there are seven states in Figure 3.2, one corresponding to each

state in Table 3.1. From each state, there are three possible transitions. With prob-

ability h = 0.8, the transition is to a state that is estimated by the weight dynamics

model. The other two states have been included with probability 0.1 each as explained

earlier.

This formulation was encoded in the input format specified by the BURLAP

MDP Solver [112]. The policy obtained using value iteration is shown in Table 3.3.

Similarly, the policy that was obtained using policy iteration is shown in Table 3.4.

Notice that both policies essentially provide the same mapping of the state to one of

the actions.

33

s1 s2 s3

s4 s5 s6

0.8

0.1

0.8

0.1

0.8

0.1

0.8

0.1

0.1

0.8

0.1

0.1

0.8

0.1

0.1

0.8

0.1

0.1

0.8

0.1

0.1

0.8

0.10.1

0.80.1

0.8

0.1

0.8

0.1

0.80.1

0.8

0.10.8

0.10.1

0.1

0.1

Figure 3.2: An example state transition matrix computed with h = 0.8.

3.1.7 Using the MDP for Action Selection

A policy that is obtained by solving an MDP can be used as follows. When the

participant is state si, the action that corresponds to this state in the policy is selected.

The individual is allowed to implement the action. When it is time to select a new

action, the individual’s weight is observed, i.e., measured. The action corresponding

to this state is again recommended. In the MDP framework, it is anticipated that

Table 3.3: Policy Computed by Value Iteration

State Actions1 a3s2 a3s3 a1s4 a1s5 a3s6 a3

34

Table 3.4: Policy Computed by Policy Iteration

State Actions1 a3s2 a3s3 a2s4 a1s5 a3s6 a1

any uncertainties that the individual may experience in implementing the action are

accounted for by the probabilities in the state transition matrix.

The next section presents a formulation of wellness management in the POMDP

framework.

3.2 Partially-Observable Markov Decision Processes

The MDP formulation assumed that the state is completely observable. For this

reason, a measurable attribute such as weight was selected to represent the state in

the preceding section. In reality, the state is not fully observable. For example, in the

domain of wellness management, there are several hidden factors such as metabolic

limitations, motivation and disease conditions that may cause the result of an action

to be different from that anticipated by the weight dynamics model.

A Partially-Observable Markov Decision Process (POMDP) is a tuple

(SP ,AP , T, R,Ψ,O),

where

• SP ,AP , T, R describe a Markov Decision Process,

35

• Ψ is a finite set of observations the agent can emit, and

• O : SP ⊗AP → Ψ is an observation function.

A POMDP is an MDP in which the agent is unable to observe the current

state. Instead, in the POMDP framework, an observation can be emitted in each

state. This observation provides some indication of the state of the POMDP. As

in the MDP, there is a reward associated with selecting an action in each state; an

optimal policy for a POMDP maximizes the total expected discounted reward and

the purpose of formulating a POMDP problem is to compute an optimal policy that

can guide action selection.

In addition to constructing the state-transition probability matrix and a re-

ward matrix for each action, it is necessary to design the observation probability

matrices for each action aj ∈ AP . Because this is a large space, the action space was

further abstracted as illustrated in Figure 3.3.

In order to formulate the wellness management problem in the POMDP

framework, a precise set of state are necessary. For this purpose, we selected the

states from the theory of planned behavior as illustrated in Table 3.5.

Table 3.5: State Space for Illustrative Example

State Descriptions1 Precontemplations2 Contemplations3 Preparations4 Actions5 Maintenance

Since these states cannot be directly observed, we maintain a belief distribu-

tionDP over these states. The key idea is that based on the current belief distribution,

the optimal policy helps to select an action. This action is implemented. Depending

36

Figure 3.3: POMDP Action Space

on the next state of the POMDP, an observation is emitted. This observation, the

recommended action and the current belief distribution is used to update the belief

distribution.

Let p(si) denote the probability assigned to si ∈ SP in DP . We know that

0 ≤ p(si) ≤ 1 ∀si ∈ SP and that∑

si∈SP p(si) = 1. To compute a new belief

state p′(si), given an old belief state p(si), an action aj, and an observation ψk, The

new degree of belief in some state s′, d′(s′) can be obtained from probability theory

37

as follows:

p′(si) = p(si|ψ, aj, DP )

=p(ψ|si, aj, DP )p(si|aj, DP )

Pr(ψ|aj, DP )

=p(ψ|si, aj)

∑sk∈SP Pr(si|aj, DP , sk)p(sk|aj, DP )

p(ψ|aj, DP )

=O(si, aj, ψ)

∑sk∈SP pkjip(sk)

p(ψ|aj, DP )

The denominator, p(ψ|aj, DP ), can be treated as a normalized factor, independent

of si, that causes si to sum to 1. The state estimation function SE(DP , aj, ψ) has

as its output the new belief state p′(si) for each state. Thus, the state-estimation

component of a POMDP controller can be constructed from a given model.

The POMDP uses its current belief distribution and a policy that is obtained

by solving the POMDP, to select actions. Each time the action is implemented, the

POMDP transitions to a new state and emits an observation. The belief distribution

is updated and the process continues.

Figure 3.4 presents an overview of how the POMDP framework is useful in

wellness management. As illustrated, once the problem is formulated in the POMDP

framework, one can compute an optimal policy. Since the states are not observable,

we maintain a belief distribution. Based on the belief distribution, the POMDP policy

recommends one of the actions shown in Figure 3.3. Since each of these high-level

actions can correspond to several choices of nutrition and physical activity level as

illustrated in Figure 3.1, we use an MDP to select the action based on the weight of

the individual.

38

Figure 3.4: Block diagram on how POMDP and MDP frameworks are used to selectan action from the action set. After constructing the pomdp policy, at each timestep, depending on the observation, selects the best intensity level of the action. Thisintensity level is used to filter the action set for the MDP framework. Using thehuman weight dynamics model and user preferences, the MDP framework selects oneaction among many actions of the same intensity level.

Using the restricted action set in each of the nine categories shown in Fig-

ure 3.3, we computed nine MDP policies. Three of these policies are illustrated in

Table 3.6. Note, the missing entries in the second column correspond to actions with

low caloric intake and low physical activity levels. The entries are missing because

the weight dynamic model could not find suitable actions in this range.

3.3 Examples of Wellness Interventions

This section presents results for a hypothetical participant with an initial weight of

120 kg. This person aspires a target weight of 80 kg. His height is 1.7 meters and age

is 29 years. The current physical activity level is 1.2. The state space for the MDP

39

Table 3.6: Three of the Nine Optimal Policies computed for MDP with restrictedaction sets shown in Figure 3.3.

Weight(kg) Low-Low Medium-Medium High-High120 n12, l7 n7, l8 n5, l12119 n12, l7 n10, l12 n5, l12118 n12, l7 n11, l11 n5, l12117 n12, l7 n7, l8 n5, l12116 n12, l7 n10, l10 n5, l12115 n12, l7 n8, l9 n5, l13114 n12, l7 n9, l8 n5, l12113 n12, l7 n11, l11 n5, l12112 n12, l7 n11, l11 n5, l12111 n13, l7 n8, l10 n5, l12110 n12, l5 n9, l8 n5, l12109 n13, l7 n9, l9 n5, l12108 n12, l6 n10, l10 n5, l12107 n12, l6 n8, l8 n5, l12106 n12, l6 n11, l12 n5, l12105 n12, l7 n9, l9 n5, l12104 n12, l7 n8, l8 n5, l13103 n12, l7 n10, l12 n5, l13102 n12, l7 n8, l8 n5, l12101 n12, l7 n9, l9 n5, l12100 n12, l7 n8, l8 n5, l1299 n12, l7 n9, l9 n5, l1298 n12, l7 n9, l10 n5, l1397 n12, l7 n8, l9 n5, l1396 n12, l7 n9, l11 n4, l1295 n12, l7 n10, l12 n4, l1294 n12, l7 n8, l8 n5, l1293 n12, l7 n8, l8 n4, l1292 n17, l0 n8, l8 n5, l1291 n8, l9 n4, l1290 n10, l12 n4, l1289 n9, l10 n5, l1388 n9, l11 n5, l1287 n7, l9 n5, l1286 n9, l12 n5, l1285 n8, l10 n5, l1284 n7, l8 n5, l1283 n8, l10 n5, l1282 n7, l9 n5, l1281 n7, l8 n5, l1280 n7, l8 n5, l1279 n9, l12 n5, l1278 n8, l10 n5, l1277 n9, l10 n5, l1276 n8, l12 n5, l1275 n10, l12 n1, l12

40

was discretized in steps of 1 kg. The minimum and maximum number of calories

consumed by the participant per day was in the range [500, . . . , 5000] in steps of 250

calories. Similarly, the physical activity level for the participant was restricted in the

range [1, . . . , 3] in steps of 0.1. All the possible combinations of nutrition and physical

activity in these ranges constituted the action space.

3.3.1 Results obtained using MDP Formulation

The state transition probability matrix was calculated as described earlier using hu-

man weight dynamics model. Given wt, the weight of the participant at time t, and an

action aj, the weight wt+1 was obtained using the weight dynamics model described

in Section 2.2. Suppose w′ represents the predicted weight, then The transition prob-

ability from state wt to state w′ was set as

P (w′∣∣w, a) =

0.8, if w′ = wt+1

0.1, if w′ = wt+1 − 2

0.1, if w′ = wt+1 + 2.

(3.6)

This means that the transition to the next state followed the weight dynamics model

80% of the time, but was off by ±2 Kg 20% of the time.

The Brown-UMBC Reinforcement Learning and Planning (BURLAP) Java

library [112] was used to obtain the optimal policies using different reward functions.

Figure 3.5 illustrates the policy obtained by solving the MDP using a linear

reward function; this reward encourages a participant to reach the target weight in

the fastest possible manner. This can be accomplished with high physical activity

levels and low caloric intake.

The weight trajectory is shown in Figure 3.6. Notice that the figure shows

that for a given number of days, there is a variability in the weight. These error

41

Figure 3.5: Shows all possible states from Initial to Target. For each state, the barsrepresent the action recommended. The exercise scale (red) is on the right (PAL) andnutrition (blue) scale is on left (cals/day). This policy was computed using a LinearReward function. The recommendation is to eat less and work out more.

bars show the minimum and maximum weights obtained for 50 different (stochastic)

executions of the MDP policy for the same participant.

Figure 3.7 illustrates the policy obtained by solving the MDP using the Safe-

Loss Reward. Higher reward is assigned to state transitions for actions that result in

weight change not more than healthy weight loss rate. It is to be noted that, in this

case we do not differentiate between two actions that result in same weight loss rate.

For example, low-calorie diet and low-intensity exercise may result in same weight

loss rate as high-calorie diet and high-intensity exercises.

The weight loss trajectory using the above policy is shown in Figure 3.8.

To illustrate how to capture participant preferences, we now show how to

compute a policy for a participant who does not wish to have physical activity level

that is more than 30% of the baseline activity level. This can be accomplished by

42

0 20 40 60

80

90

100

110

120

Time(Days)

Body

Wei

ght(

Kg)

grid

Figure 3.6: The expected weight loss trajectory for a participant who initially weighs120 Kg and follows the policy in Figure 3.5.

Figure 3.7: Actions computed from a policy that rewarded a target rate of weightloss. Higher or lower loss is penalized. The target loss can be achieved via multipleactions. The figure illustrates one stochastic choice that helps the participant achievetarget weight.

43

0 50 100 150 200 250 300

80

90

100

110

120

Time(Days)

Body

Wei

ght(

Kg)

Figure 3.8: Weight loss trajectory and uncertainty regions when executed the policycomputed for recommended weight loss.

designing a reward function that increases reward accordingly. Figure 3.9 and Fig-

ure 3.10 illustrate the policy and the weight loss trajectory obtained using such a

reward function, respectively. Notice that the caloric intake is higher in high weight

states and gradually decreases as the participant weight decreases. This is in ac-

cordance with the weight loss dynamics presented in section 4.2 that baseline calorie

intake is proportional to the total body mass, and therefore the participant is required

to consume more calories to support his weight. As the weight starts to decrease, the

baseline calories also decrease and so does the caloric intake. The expected weight loss

trajectory in this scenario is shown in figure 3.10 and can be seen that the expected

time to reach the target weight is now pushed further than the time in Figure 3.8;

this delay can be attributed to the fact that participant preference is now considered

in the policy.

44

Figure 3.9: MDP policy obtained using a reward function that rewarded actions thatwere within 30% of baseline exercise activity level.

0 100 200 300 400 500

80

90

100

110

120

Time(Days)

Body

Wei

ght(

Kg)

Figure 3.10: Weight loss trajectory for participant not willing to perform high inten-sity exercises but willing to go on a calorie-restricted diet.

45

Table 3.7: State Transition Matrix used for Low Intensity Actions

s1 s2 s3 s4 s5s1 0.75 0.25 0 0 0s2 0.25 0.5 0.25 0 0s3 0 0.1 0.8 0.1 0s4 0 0 0.1 0.8 0.1s5 0 0 0 0.1 0.9

Table 3.8: State Transition Matrix used for Medium Intensity Actions

s1 s2 s3 s4 s5s1 0.9 0.1 0 0 0s2 0.3 0.4 0.3 0 0s3 0 0.1 0.4 0.5 0s4 0 0 0.1 0.3 0.6s5 0 0 0.1 0.4 0.5

3.3.2 Results obtained using POMDP Formulation

In order to work with the abstracted action sets illustrated in Figure 3.3, it was nec-

essary to construct state-transition probability matrices for these abstracted actions.

Table 3.7, Table 3.8 and Table 3.9 show the transition probabilities that were con-

structed manually for low, medium and high intensity actions. Three such matrices

were constructed for nutrition actions, and three additional matrices were constructed

for exercise actions.

Table 3.9: State Transition Matrix used for High Intensity Actions

s1 s2 s3 s4 s5s1 1.0 0 0 0 0s2 1.0 0.0 0 0 0s3 1.0 0 0 0 0s4 0 0 0.2 0.4 0.4s5 0 0 0 0.2 0.8

46

Table 3.10: Observation Matrix used for Low Intensity Actions

Adhered O1 Not Adhered O2

s1 0.1 0.9s2 0.8 0.2s3 1.0 0s4 1.0 0s5 1.0 0

Table 3.11: Observation Matrix used for Medium Intensity Actions


s1 0.1 0.9s2 0.3 0.7s3 0.5 0.5s4 1.0 0s5 1.0 0

In addition to the state-transition matrices, it was necessary to construct

observation probability matrices corresponding to each action. Table 3.10, Table 3.11

and Table 3.12 show the corresponding observation probability matrices.

The optimal policy computed maps the belief distribution over the states to

actions. An example of a part of this map is illustrated in Figure 3.13. As in the

case of the MDP, an optimal policy for the POMDP is computed offline. During the

implementation of the policy, a belief distribution is maintained over the states of

Table 3.12: Observation Matrix used for High Intensity Actions


s1 0 1.0s2 0 1.0s3 0 1.0s4 0.6 0.4s5 0.8 0.2

47

the POMDP. At each step, the a dot product is computed using the current belief

distribution and a policy vector shown in Table 3.13. The action corresponding to

the policy vector that yields the highest value after the dot product is selected.

Table 3.13: Snapshot of the POMDP policy. The policy had over 250 vectors torepresent the states and only a few are shown in this table as an illustration.

Policy Vector Action Intensity3.90 11.65 0.88 0.81 0.26 Low0.80 1.48 18.16 25.85 2.56 Medium3.91 11.53 0.39 0.15 0.06 Low0.13 0.13 0.13 2.38 27.22 High0.80 0.90 15.15 27.18 2.97 Medium0.03 0.03 0.03 2.40 27.26 High0.73 1.53 18.26 25.88 2.59 Medium0.03 0.03 0.03 3.51 26.83 High3.86 11.40 0.25 0.27 0.90 Low0.16 1.26 18.71 26.02 2.74 Medium3.44 11.57 3.58 4.59 1.01 Low0.00 0.00 0.00 2.40 27.26 High

Figure 3.11 shows how the belief distribution is being updated when imple-

menting the POMDP policy.

As discussed earlier, the POMDP recommends one of the actions shown in the

abstracted action space in Figure 3.3. Each of these actions may in fact correspond to

several actions at the resolution of the action space illustrated in Figure 3.1. For this

reason, we used nine different MDPs, each with its own optimal policy as indicated

in Table 3.6. The weight of the participant was used as the observable state for the

MDP. The weight trajectory of the individual was computed by assuming different

adherence levels.

Figure 3.12 shows the weight trajectories when the optimal policy computed

from the above POMDP formulation is executed assuming different levels of adher-

ence. In this scenario, we have assumed that the adherence of the individual for

48

(a) Selected Action is (2200,1.5) (b) Selected Action is (2200,1.5)

(c) Selected Action is (2300,2.0) (d) Selected Action is (2300,2.0)

(e) Selected Action is (3900,2.0) (f) Selected Action is (3900,2.0)

Figure 3.11: Desire to perform : Nutrition : 0.2, Exercise : 0.8

49

0 500 1,000 1,500 2,000

80

100

120

Time(Days)

Body

Wei

ght(

KG

)

Nutrition=0.7;Exercise=0.7Nutrition=1.0;Exercise=1.0Nutrition=0.8;Exercise=0.5Nutrition=0.8;Exercise=0.8

Figure 3.12: The weight trajectories for different probability of adherence levels andits corresponding variations in time to reach target weight

a nutrition recommendation is different from the adherence for an exercise recom-

mendation. These adherence levels are used to stochastically generate observations,

which are used to update the belief distribution. Notice from the figure that as the

degree of adherence to nutrition and exercise decreases, the time taken to reach the

target weight increases. For example, if the adherence level for nutrition and exer-

cise is 1.0 each, it can be seen from the figure that target weight can be achieved

about 240 to 260 days. In this case the belief distribution of the motivation states

is pushed towards s5 and high intensity exercise and nutrition actions are selected

by the POMDP policy. However as the adherence level for nutrition and exercise is

decreased to 0.8 and 0.5 respectively, the belief distribution of the POMDP states

is shifted towards s2 and therefore the POMDP policy selects low intensity exercise

and nutrition actions. In such states, since the motivation of participant is low, the

participant ends up increasing weight and hence will take more time in reaching the

target weight (520-1660 days).

50

3.4 Modeling Issues in Sequential Decision Making Frameworks

Although the two classical decision-making frameworks provide a precise, mathe-

matical framework for selecting wellness actions, there are several modeling issues

that are inherent in this approach. For example, the design in order to design the

state-transition probability matrices, we would need detailed models of the patient’s

natural history without the wellness interventions. Accommodating preferences and

temporal constraints is challenging. Similarly, the design of the observation probabil-

ity matrix is challenging because there are no models from which these probabilities

can be derived. Perhaps, the most challenging aspect is the design of rewards. In

order to appropriately design rewards, it is necessary to trade off the current costs

with future benefits. Similar issues have been addressed in the context of medical

interventions [113, 114, 115, 116] and it is necessary to design suitable metrics for the

wellness domain.

The next chapter presents the Goal-Seeking approach to wellness manage-

ment.

51

CHAPTER IV

GOAL-SEEKING FORMULATION FOR PERSONALIZED WELLNESS

MANAGEMENT

This chapter presents a Goal-Seeking formulation for wellness management and de-

scribes the advantages and disadvantages of this approach when compared with the

formulations in the preceding chapter that were based on two classical frameworks

for sequential decision-making.

Similar to the decision-makers in the preceding chapter, the objective of a

goal-seeking decision-maker are to enable a participant to adopt healthy life style

choices by making exercise and nutrition recommendations that falls with in healthy

weight loss regime and also caters to the individual life style changes. The decision-

maker must select actions that has the highest probability of adherence to help indi-

vidual achieve target weight loss.

4.1 Detailed Goal-Seeking Formulation of PWM

In this section, the artifacts of the Goal-Seeking formulation that were introduced

in Section 2.7 of Chapter 2 are described in more detail.

The overall approach in the Goal-Seeking approach was illustrated in Fig-

ure 4.1. The decision-maker operates in discrete steps, each step, t, is for a certain

duration (days). Unlike the MDP and POMDP approaches, there is no attempt to

represent the underlying state of the system. Thus, this approach is useful in scenar-

ios like wellness management where the design of a state representation that accounts

52

for the various factors that affect behavior is difficult. For example, the actions rec-

ommended to a participant must also consider metabolic constraints, physical limita-

tions, motivation and disease conditions. Even after an action is recommended, the

implementation of the action may be affected by several constraints such as financial,

time, trust in the recommendation and motivation to adhere to the nutrition and

exercise actions recommended. For these reasons, the goal-seeking approach relies on

a set of feasible alternate actions that are provided by the participant. Such feasible

actions can also be generated by any algorithmic procedure that creates empirical

models for how the different factors must be weighed to select the feasible actions.

The goal-seeking approach critically relies on the participant being a rational agent

who is interested to improve their own wellness [10].

4.1.1 Alternate Actions, Π

Alternate actions represent the choices that are available to the decision-maker. For

the Goal-Seeking formulation, we select the same action space that was illustrated

in Figure 3.1. Recall, that the nutrition intake was quantified in terms of caloric

intake in the range [500, . . . , 5000] in steps of 250 calories, and exercise activity was

quantified in terms of the physical activity level in the range [1, . . . , 3] in steps of

0.1. These discrete choices resulted in about 360 actions, i.e., nutrition-exercise pairs

that could be recommended to the participant. At time epoch t, the decision-maker

would select an action (nt, lt) from a set of feasible actions that are suggested by the

individual.

4.1.2 Consequences, Ψ

A consequence ψ ∈ Ψ is the result of a specific action that is implemented by

the individual. In this investigation, the weight of the participant has been used

as a measurable attribute. Hence, the set of consequences was designed to be the

53

quantized set of weights of the participant; as in the preceding formulations, the

range [75, . . . , 140], in steps of 1 kg, was used.

Two kinds of consequences were considered. The anticipated consequence of

a recommended action, hatψ, is the target weight estimated using the Reflection

Mapper that is described in Section 4.2; at time t, the anticipated consequence for

time t+ 1 is computed and used to select the action (nt, lt). The actual consequence

of action (nt, lt) is the weight of the participant that is measured at time t+ 1.

4.1.3 Uncertainties, ∆

Wellness actions must be implemented by humans in the interval of time between

two successive time epochs. If one or more uncertainties come to pass, the actual

consequence of the recommended action may be different from the anticipated con-

sequence. In the Goal-Seeking formulation presented here, only the motivation was

explicitly modeled as an uncertainty. In the future, this uncertainty model can be

extended to incorporate many other aspects.

The motivation of the participant was assumed to be distributed over the

range [0, . . . , 100]. The Evaluation Mapper that is described in Section 4.1.6 deter-

mines whether or not the actual consequence differs from the anticipated consequence,

and whether this difference is acceptable based on the tolerance value (described in

Section 4.1.7 for the participant and the stage of the wellness intervention. The de-

cision of the Evaluation Mapper was considered to be analogous to an observation

emitted by the POMDP. The motivation distribution was updated based on this

observation.

The uncertainty in implementing a recommended action was also modeled.

The motivation levels was quantized into levels that correspond roughly to the stages

of change in the theory of planned behavior. Thus, a person with motivation level

0-20 was considered to be in the pre-contemplation stage, a person in the range 20-40

54

in the contemplation stage, etc. These stages were associated with different levels

of variance as indicated in Table 4.1. These variances were used as the range over

which the daily implementation of the recommended action. For example, when the

motivation corresponds to s2, the participant may vary the caloric intake by 750

calories each day and the physical activity level may vary by 0.75. Similarly, when

the participant is highly motivated, e.g., s4, the variance in the nutritional intake is

100 calories and the variance in the physical activity level is 0.1.

Stage Nutrition(Cal/day) Variance PAL Variances1 1000 1.0s2 750 0.75s3 500 0.5s4 250 0.25s5 100 0.1

Table 4.1: The motivation of the participant affects the variance of the nutrition andexercise actions as shown below.

4.1.4 Reflection, Ξ

The Reflection function, Ξ : Π ⊗ ∆ → Ψ, represents the decision-maker’s view of

the environment in which the participant must operate over the next time step. The

anticipated consequence, ψ, is computed based by the decision-maker by taking into

account the uncertainties that can come to pass. We used the human weight dynamics

model that is described in more detail in Section 4.2. This model accepted the

participant body metrics, daily nutrition calories and physical activity level and time

in days and estimates the weight of the participant after the action is implemented

subject to implementation uncertainties that were modeled as variances corresponding

to the motivation levels as illustrated in Table 4.1.

55

4.1.5 Evaluation Set, Λ

The Evaluation Set (or performance scale) is a metric used to compare outcomes of

selected actions. This scale helps the decision-maker to determine which alternate

action is preferable over other choices. We used the closed interval [0, 100] on the

real line as the scale - with 0 representing an undesirable choice and 100 representing

the most desirable choice.

4.1.6 Evaluation Mapping, Ω

The actual consequence ψ is not necessarily the same as the anticipated consequence

ψ. Unlike the anticipated consequence that was estimated, the actual consequence

can be measured by weighing the participant. The Evaluation Mapping is a function

that maps a selected action and its actual consequence to a value, λi ∈ Λ, on the

performance scale i.e.,Ω : Π⊗Ψ→ Λ.

There are many facets for a comprehensive evaluation mapper that can be

used for wellness management. In this thesis, the evaluation mapper compared the

actual consequence to the anticipated consequence and determined whether or not

there is a difference. If the difference is acceptable as specified in the tolerance

function described Section 4.1.7. The objective for the evaluation mapper was to

make a determination whether the participant adhered to the recommended action

or not. This determination was used to update the motivation distribution to guide

the selection of recommended actions in the next time epoch. In the future, the

evaluation mapper can be extended to account for the performance by considering

the trajectory of actions recommended to the participant. The evaluation mapper

may also be able to tradeoff the adherence to nutrition vs. the adherence to exercise

activity in making its determination.

Another facet of the evaluation mapper is the use of devices to evaluate

adherence. For example, the devices that have emerged to support the quantified self

56

movement [117] can be integrated into the evaluation mapper. Chapter 6 presents an

approach in which the Microsoft Kinect camera is used to evaluate the adherence to

an exercise recommendation.

4.1.7 Tolerance Function, Γ

The tolerance function, Γ : Π⊗Ψ→ Λ, is a bound on the amount of variation that

can be tolerated in performance before a solution is considered as unsatisfactory. In

our simulations at the end of each time slice, we required the weight change of the

participant to be at least 1 KG.

SELECTACTIONS

EstablishLimits

MATCHEDANTICIPATED

CONSEQUENCE ?

MeasuredPerformance

YESNoChange

BehavioralAnalysis

Questionnaire

NO

CURRENTUNCERTANTIES

Update

Γ =Π X ψ→ Δ

0 5 10 15 20 .. .. .. .. 75 80 85 90 95 99

Ξ : ΠX Δ→ψ Ω: Π X Δ→ψ

MEAN$

Multi-scaleActors

EvaluationMapping

ReflectionMapping

Figure 4.1: Detailed View of Goal-Seeking Interactions

57

4.2 Reflection Mapper Design

The reflection mapper used in this investigation was designed using the three com-

partment energy balance model which was introduced in Section 2.3.

The three-compartment model posits that the human body stores energy as

fat mass (FM), lean mass (LM) and extra-cellular fluid (ECF ) [56, 72, 58, 59]. The

change in each of these stores is represented as

d FM(t)

dt=

(1− p(t))EB(t)

ρFM(4.1)

d LM(t)

dt=

p(t)EB(t)

ρLM(4.2)

d ECF

dt=

1

[Na]

(∆Nadiet − ξNa(ECF − ECFinit)− ξCI(1−

CI

CIb)

)(4.3)

where ρFM = 9400 kcal/kg, ρLM = 1800 kcal/kg.

For the extra-cellular fluid volume (in ml), ∆Nadiet is the change on sodium

in mg/d, CIb is the baseline carbohydrate intake, [Na] = 3.22 mg/ml, ξNa = 3 mg/ml,

ξCI = 4000 mg/d. In this model, energy expenditure EE(t), is modeled as

EE(t) = βEI(t)+δBM+K+γLM LM+γFM FM+ηFMd FM

dt+ηLM

d LM

dt, (4.4)

where β = 0.24 is the coefficient for the thermic effect of feeding TEF, δ is the physical

activity coefficient, γLM = 22 kcal/kg/day, γFM = 3.2 kcal/kg/day, ηFM = 180 kcal/kg

and ηLM = 230 kcal/kg and K are coefficients and constants for the calculation of

RMR. The constant K accounts for the initial energy balance conditions and is

determined by solving Equation 2.5 at initial steady state conditions, i.e.,

EI − EE = 0, (4.5)

58

Figure 4.2: The Energy Balance Model

d FM

dt=d LM

dt= 0, (4.6)

and

K = −γLM LM − γFM FM − δ BM + EI (1− β). (4.7)

Finally, the body mass at time t is obtained as

BM(t) = FM(t) + LM(t) + ECF (t). (4.8)

In summary, this three-compartment energy balance model accepts as input

EI(t) comprising CI(t), FI(t) and PI(t), change in sodium intake ∆diet, and the

physical activity level δ. The model produces FM(t), LM(t) and ECF (t) as outputs

from which we can compute the body mass, BM(t), of the individual as illustrated

in figure 4.2.

The implementation of the actions was designed to account for uncertainties

as illustrated in Table 4.1.

The reflection mapper selects the best action from among a set of feasible

actions so as to get the best consequence even when uncertainties come to pass. For

this purpose, a cost-mapper is required to map the anticipated consequence to the

59

performance scale. This function ranks the consequences on a common scale so that

the action with the best consequence is selected. The reflection mapper estimates the

consequences of each action in the action set, and the cost mapper assigns a cost (or

negative reward). In this thesis, a simple cost mapper is selected that operates on

the estimated weight lost by the participant using the reflection mapper discussed in

Section 4.2. One such function is shown in Figure 4.3.

Figure 4.3: The Reward function used in the goal seeking framework to assign cost(negative rewards) to consequences of each action that is estimated through reflectionmapper.

4.3 Bayesian Update of Human Motivation Distribution

In POMDP modeling, the human motivation distribution was discretized into 5 states,

according to the definition of Social Cognitive Theory. POMDP modeling restricts

60

the problem size as the algorithms that are used to solve the problem, does not scale

well as the number of states increase. However, in practice, the human motivation

level can be quantized anywhere between 0 - 100%, where 0% can be classified as

amotivated individual and 100% can be classified as highly motivated individual. In

real world, the participant may be somewhere in between that cannot be observed

directly. Here we describe the Bayesian update technique that is employed in the

goal-seeking to update the motivation distributionDM , that is not restricted to the

number of states in the distribution.

DM is the random variable, that represents the motivation level of the partici-

pant. Hypothesis : The hypothesis H that is being tested, i.e. whether the participant

is motivated or amotivated. O : the observation data set O=Adhered, Not-Adhered

is the new information, that is inputted to the model. This information is generated

by the decision-maker upon evaluating the participant at each time step, by analyzing

the performance of the participant with respect to the action recommended in the pre-

vious time-step. Prior-Distribution: The prior motivation distribution p(DM = m),

such that|DM |∑m=1

p(DM = m) = 1

Likelihood : The likelihood function ie p(O|H) i.e. the probability of the data being

generated assuming that the hypothesis is true. For example, p(O = ”adhered”|H =

”50%motivated”) is the probability that the participant will adhere to the recom-

mendation given that the participant is 50% motivated. After gathering the new

information, the Bayes theorem is used to compute the compute the probability of

each hypothesis given the data Posterior Distribution. The Bayes theorem is given

61

by the equation

P (DM |O) = P (O|DM )P (DM )P (O)

(4.9)

= likelihood∗priorNormalization

where, normalization is calculated as follows

P (O) =

|DM |∑m=1

P (O|DM = m)P (DM = m)

and the likelihood function that is used is given as follows

P (O|DM = m) =

m/100, if O=”Adhered”

1−m/100, if O=”Not-Adhered”

(4.10)

4.4 Simulation Results

This section presents the simulation results for the goal seeking architecture described

in the previous section. To be consistent with the results obtained using the Com-

pletely Observable MDP and Partially Observable MDP, the same user scenarios are

used in the simulations i.e. a Male participant initial weight of 120 kg aspiring a tar-

get weight of 80 kg. His height is 1.7 meters and age is 29 years. The current physical

activity level is 1.2. In order to simulate the uncertainty in the action execution, an

adherence metric for nutrition and exercise is used. An adherence level of 1 represents

that the participant adheres to the recommended action in each time step, while an

adherence level of 0 represents the participant does not adhere to the recommended

action. Figure 4.5 shows how the motivation distribution DM at each various time

steps along the simulation, gets updated for a nutrition and exercise adherence level

62

of 0.5 each. Observe that the motivation distribution at the end of 12 time steps,

converged to approx 45− 55% meaning that the participant is neither motivated nor

a-motivated. Figure 4.4 shows a similar scenario however for a nutrition and exercise

adherence level of 08 and 0.4 respectively. Observe that after few iterations, the nu-

trition motivation distribution updates shifts towards 100% motivated and exercise

motivation distribution shifts towards 0% motivated; meaning that the participant is

inclined towards going on a calorie restricted diet rather than doing exercise activities.

Also observe that the action recommended at each iteration follows the motivation

level at each stage. For example, in Figure 4.4 the action recommended in the first

stage is (2750, 1.3). Since in this stage no information about the motivation level is

known to the goal seeking agent, it starts out by selecting a low intensity action. As

more observations are received and motivation distribution is updated, the selected

action intensity level reflects that of the motivation distribution. Thus the action

(1500, 1.8) is selected to reflect high motivation towards nutrition and low motivation

towards exercise activities.

Figures 4.6, 4.7 and 4.8 simulate the expected weight trajectories of the par-

ticipant for different levels of adherence levels of exercise and nutrition. As from the

above descriptions, as the participant moves towards high levels of motivation, high

intensity actions are recommended at each time step. If the adherence level of the

participant is also high, then the time taken for for the participant to reach the target

weight gets smaller.

63





64





65

0 100 200 300 400 500 600

120

130

140

150

Time(Days)

Weight(KG

)

Figure 4.6: Expected weight trajectory and its error bars for a participant with Nu-trition Adherence Level : 0.1 and Exercise Adherence Level: 0.1. Since the adherencelevel is too small, the participant does not adhere to the recommended actions andtherefore is expected to increase weight and never reach the target weight.

0 100 200 300 400 500 600

80

100

120

Time(Days)

Weight(KG

)

Figure 4.7: Expected weight trajectory and its error bars for a participant with Nutri-tion Adherence Level : 0.5 and Exercise Adherence Level: 0.5. Since the participantadheres to the recommended action 50% of the time, the participant neither increasesweight nor decreases weight as is evident by the large error bars.

66

0 100 200 300 400 500 60070

80

90

100

110

120

Time(Days)

Weight(KG

)

Figure 4.8: Expected weight trajectory and its error bars for a participant withNutrition Adherence Level : 1.0 and Exercise Adherence Level: 1.0. In this scenario,the participant adheres to each recommended action and therefore after few timesteps, the goal seeking framework recommends high intensity actions. Since theadherence levels are high, the error bars in estimating the wight trajectory to therecommended action is small.

67

CHAPTER V

SYSTEMS ARCHITECTURE AND SOFTWARE DESIGN

This chapter presents the system architecture and design that was implemented to

operationalize the personalized wellness management system. In its current form, this

chapter is only a preliminary description that is intended to provide an overview of

the software system. A more detailed design will be specified in the final dissertation.

5.1 Overview of the PWM Software Architecture

The Personal Wellness Management system architecture along with the user & expert

interaction is depicted in Figure 5.1. The System consists of the following modules.

5.1.1 Participant Dialog Module(PDM)

PWM is a personalized action recommendation system and, consequently, interaction

with participants is important for the proper functioning of the system. The objective

of the Participant Dialog Module is to gather user information, present potential

recommendations to the participant, gather participant preferences and present the

final recommendations. The user interacts with the software using this module that

starts with a login window. The user is first required to register, where in they are

required to enter the anthropometric data that is securely saved in the database.

Once the registration process is finished, the user can use this login information to

use the PWM services anytime later. For a returning user, after logging into the

software, a performance visualizer window is presented wherein the user can enter his

68

User Details Collector

Action Set Generator

USER

Action Recommender

<Update>

<Accept>

Reflection Mapper

Weight Dynamics Model

Estimator

Reward Mapper

GSF

GUI

DB

Expert

Act

ion

Se

lect

ion

A

gen

t

Performance Analyzer

Perf

orm

ance

M

app

er

User Measurements

<Participate>

Human Behavioral Model

< Profile>

Performance Visualizer

MEAN$ Estimation

<Feedb

ack>

Dat

abas

e C

on

nec

tio

n

Action Manager

<Statistics>

Motivational Questionnaire

Figure 5.1: The Personal Wellness Management System Architecture Design

actual nutrition and exercise activities on a per day basis. The data flow model for

the user login screen is presented in Figure 5.2

The participant initially is presented with a login screen, upon requesting to

use the PWM software services. Since no assumptions can be made on the platform,

or technical capabilities of the participant, the software design should be platform

independent and easy to be used. The decision-maker and the participant interacts

with the software using the GUI, which is responsible to show the requested infor-

mation in an ambiguous way and also to securely store the information inputed by

the user into the database. Any service that is requested the user, will then have to

operate on this data stored in the database.

5.1.2 Performance Analyzer

This module can be used by the user to estimate how their weight change trajectories

looks like and accordingly set performance objectives with the help of decision-maker.

69

Figure 5.2: Activity Flow in the Participant Dialog Module

Through this module, the user can retrieve historic data and other body metric

measurements that are taken by the decision-maker during each visit. The current

implementation of human weight dyanmics is the three compartment model of the

human weight dynamics by the NIDDK as discussed in Section 4.2. The weight

dynamic models in its current form, do not take into account the motivation of the

participant or any other human behavior models. Moreover, based on investigations

reported in [73] by Mahamadi, there is ambiguity in the accuracy of these models.

Therefore to allow for a future integration of these models into the weight dynamics

models and also allow a scalable deployment through RESTful APIs and to seamlessly

integrate the model into the PWM software architecture, the decision was made

70

to implement the three compartment model from scratch in Java using the Object

Oriented software design principles shown in the Figure 5.4.

The corresponding class diagram is shown in Figure 5.3. The three compart-

ment model is derived from its parent class, Human Weight Dynamic Model. This

class, makes use of the Action object to retrieve the daily nutrition input and exercise

expenditure calories. The anthropometric information retrieved from the database is

then used to integrate the differential equations using the Rangekutta method.

Figure 5.3: The class diagram of the implementation of Three compartment modelof the human body weight dynamics.

5.1.3 Goal Seeking Framework

The module implements all the artifacts of the Goal-Seeking Paradigm as described

in Section 4. A list of these artifacts along with the description of all implemented

Java Classes is presented in Table 5.1.

71

Figure 5.4: Implementation of the three compartment human weight dynamic modelsin JAVA. Simulating a hypothetical Participant aged 27 years, 100KG and 1.7 mheight, sedentary life style, eating 2000 Calories/day and doing moderate intensityexercises will weigh approximately 88.5 Kgs in 180 Days.

5.1.4 Database Architecture

The following section presents the detailed design of the database architecture, as

it plays a central role in the operationalization of the PWM software system. The

PWM Store was designed to provide access to participant data in an Object Oriented

framework. This store was designed as a layer above a traditional relational database.

To accommodate the different update rates for different data in the system, the data

tables were classified as three groups:

1. Meta Data: These data defined the parameters that were used to specify

exercises and participant profiles. These data remain constant after system

setup. There are no additions or deletions throughout the lifetime of the system.

72

2. Static Data: These data do not change after being entered into the PWM Store.

New records can be added to these data as new exercises and participants are

incorporated.

3. Dynamic Data: These data change frequently, perhaps, with each new inter-

action with the participants. The data reflect participant inputs and the results

of computations in the system.

A hierarchical design of the database based on the above considerations is

shown in Figure 5.5.

5.2 Versatile and Scalable Deployment Architecture

The objective of the personal wellness management software suite, is to provide var-

ious web services to the participant such as nutrition and exercise data logging, nu-

trition and exercise preferences, estimate weight trajectory for a certain nutrition

73

Meta Data

exercise types exercise levels exercise goals

participant occupations

Static Data

exercises

participants

participant exercise health

Dynamic Data

participant exercise preferences

participant general health

administrators

participant activity history

participant exercise schedules

exercise categories

activity levels

Figure 5.5: Multitier Structure of Database Storage in PWM.

and exercising behaviors. This software suite is designed to be used by exercise and

nutrition experts to monitor each participant progress and also to interact with the

decision support tool in recommending actions to them. Such services are typically

computationally expensive and require Giga Bytes of memory to operate on. Further-

more, the software suite should allow multiple clients and exercise professionals to

login at the same time. To improve the end user satisfaction, the software tool should

be able to run on different platforms and operating systems. Therefore a versatile

and scalable deployment architecture is to be designed. In this thesis, a RESTful web

service architecture is employed.

74

REST is an architecture style for designing networked applications. The idea

is that, rather than using complex mechanisms such as CORBA, RPC or SOAP to

connect between machines, simple HTTP is used to make calls between machines.

RESTful applications use HTTP requests to post data (create and/or update), read

data (e.g., make queries), and delete data. This architecture allows to decouple server

and client software and makes the implementation independent of each other. This

makes it easy to add or upgrade server web services with out changing the client

implementation. Figure 5.6 illustrates the design of deployment architecture pursued

in this investigation.

The Goal Seeking framework, along with its artifacts such as Reflection Map-

per (RM), Cost Mapper (CM) and Evaluation Mapper(EM) are all deployed in a

remote server that hosts several other web services such as recommender systems1.

Each web services has its own Uniform Resource Identifier (URI) that the client

software uses to explicitly mention to the server which service it wants to use. For

example, if the end user participant would like to get a weight trajectory for a specific

nutrition and exercise behavior, the client software would send a HTTP request to

the following URI

POST http://cesl.uakron.edu/pwm/reflectionMapper/weightTrajectory

This design makes no assumptions as to which device or operating system

that the participant or expert uses to use these services. Any data transfer that

happens between the client and the server is using HTTP requests and responses.

The data that is exchanged between client and server during these communications

is represented using standard JSON format. JSON is a lightweight data-interchange

format that makes it easy for for machines to parse and generate. It is based on a

subset of the JavaScript Programming Language, Standard ECMA-262 3rd Edition.

1Not discussed in this thesis

75

JSON is a text format that is completely language independent but uses conventions

that are familiar to programmers of the C-family of languages, including C, C++,

C#, Java, JavaScript, Perl, Python, and many others. These properties make JSON

an ideal data-interchange language.

Figure 5.7 illustrates an example of how RESTful API operates. When the

participant wants to use the service offered from Reflection mapper, the client software

initiates a HTTP request with all the data required for the service to operate on, in a

JSON object. This object is placed in the HTTP POST request and sent to the server.

The server upon serving the request, creates another JSON object and populates it

with the result of the computation requested by the client. This object is placed

inside the HTTP response sent by the server to the client. Through this method of

communication, the sever implementation of the web service can be upgraded with

out changing the client’s implementation.

76

Goal Seeking Artifact Class Name DescriptionAction PWMAction The Nutrition and Exercise tuple.

ActionOutcome An enumerator class to keep trackof adherence levels.(Adhered orNot-Adhered)

Consequence The estimated consequence ofeach action. In the current im-plementation, this class estimatesthe amount of weight change bypursuing a given action.

IntensityLevel The intensity level of an exerciseand nutrition activity to be se-lected by the participant at eachtime epoch.

Cost Mapper CostMapper The parent class for the CostMapper artifact of Goal Seeking.

UtilityCurve The utility curve parameters thatcaptures the decision-maker pref-erences of the consequences ofeach action.

Reflection Mapper ReflectionMapper The parent class that estimatesthe consequences of each action inthe action set

BodyWeightDynamicsModel The implementation of the threecompartment model

Evaluation Mapper EvaluationMapper The parent class for the evalua-tion mapping artifact for the goalseeking paradigm

PerformanceEvaluator The child class that evaluateswhether the action recommendedin the previous time step has beenadhered or not.

Tolerance Level ToleranceLevel A class that captures the toler-ance level for the action recom-mended. In the current imple-mentation, a tolerance level of1KG is used.

Table 5.1: The Goal Seeking Artifacts and the corresponding classes implemented

77

JSONInterfaceforParticipantDialogModule(PDM)

Internet<HTTP>

PWMStorage

MobileAppsWebApps

PDM

ExerciseRecommenderSystemsBackendLibrary

IM DKM EM

RESTfulWebService

GoalSeekingFrameworkBackendLibrary

RM CM EM

Figure 5.6: RESTful API for Scalable Deployment of PWM. The web services pro-vided for Goal Seeking resides in the server along with other services in the PWMwellness software suite. The client and server interact with each other using the JSONdata format.

78

79

Figure 5.7: An example of client software initiating with a HTTP request to use the reflection mapper web service hosted on the server module through RESTful api. The data required for the server is formatted into a JSON object and is inserted in post request. The server after processing the request, formats its response into another JSON object and places it into the HTTP Response that is then sent to the client.

CHAPTER VI

EVALUATING AND GUIDING EXERCISE BEHAVIORS

Recognizing and analyzing the activity when the participants perform exercises is an

important problem for improving Personalized Wellness Management (PWM). Such

recognition and analysis of human motion is important in many applications including

wellness management systems, sports training, rehabilitation therapy, surveillance

and assistive technologies [118] to improve quality of life [119, 120, 121, 122].

This chapter presents two applications of the Kinect Camera to improve ex-

ercise performance and adherence. The first application focuses on detecting errors

in exercises that could result in injuries and, hence, a lack of adherence. The second

application focuses on estimating the ground reaction forces while performing exer-

cises by using only the Kinect camera. These forces were later used in estimating

calories burnt during the physical activity in real-time [21]

6.1 Supporting Exercise Performance using Kinect Camera

In the recent times, it is not uncommon that participant’s perform basic exercises at

home in the absence of physical trainer. In these home settings, wearable sensors are

not the performers first choice as it hinders participant’s motion and performance.

In such scenarios, technology can play an important role in monitoring the exercise

performance and provide real time feedback to the participant to minimize injury

risk and improve home exercise adherence. Such technologies should be reliable in

monitoring the activity and fast enough to provide real time feedback. The sensors

80

that provide data to these software should be cost efficient and flexible to mount such

that they do not hinder the participant’s range of motion [123].

Toward this end, the scope of the analysis on time-series data obtained from

a non-invasive sensor, such as the Microsoft Kinect 2.0, was explored. Specifically, we

selected a set of 4 body weight exercises [124] as shown in Figure 6.1, and used the

data captured by the Kinect camera to identify the errors during exercise performance

of the participant1.

The data was collected from 46 participants. After completion of the IRB

process, each participant performed all the four exercises. Hence, a total of 184

datasets were collected. Each dataset contained 75 time series, three for each joint,

that was captured from the Kinect 2.0 camera at 30 frames per second for the duration

of the exercise. Participants were required to perform 10 repetitions of each exercise

with a short break between exercises. This data was separated into a training set

with 64 data sets and a testing set with the remaining 120 data sets. The training

set was used to train the SVM classifier. The testing set was used to evaluate the

goodness of the classifier and the results are presented in the confusion matrices.

6.1.1 Sagittal, Frontal and Transverse Plane Projections

The first feature vector considered was derived by projecting the time-series data on

the Sagittal, Frontal and Transverse planes. These three planes are important as

exercise science experts and medical professionals evaluate the exercises using these

three planes. Recall that the data obtained from the Kinect camera is the 3D, i.e.,

(x, y, z), coordinates for each joint, where the z coordinate corresponds to the depth

1The results reported in this chapter were achieved in collaboration with Master’s students [20,21, 23] in the Complex Engineered Systems lab, in the Department of Electrical and ComputerEngineering, University of Akron. The overall design and strategy was developed as a part of thisinvestigation and the Master’s students completed the detailed design, implementation and collecteddata to validate the approach. There are additional results reported in this chapter that were not apart of the Master’s thesis efforts.

81

(a) Jumping Jacks (b) lunges

(c) squat (d) high knees

Figure 6.1: Four Exercises that were selected from the HICT Suite for this study.

or distance from the camera. The XY -pane, Y Z-plane and ZX-plane corresponds

to Frontal plane, Sagittal plane and Tranverse plane respectively. Figure 6.2 presents

sample data captured from the Kinect camera. This figure also shows the trajectories

of the (x, y) positions of the data as an illustration.

Figure 6.3 shows the projections of the time-series data of the joints in the

Y Z-plane. These were used to compute the feature vectors. For each joint, using the

average initial position, the average final position, the extent of movement (`yz) in

82

Figure 6.2: Sample time-series data captured from the Kinect camera for the x-coordinate of a few joints is as shown on the left. The figure on the right shows thetrajectories of the (x, y) coordinates for each joint when the Jumping Jacks exerciseis performed.

83

Figure 6.3: Projection of all the joints on the Y Z-Plane for the Jumping Jacksexercise. This was obtained by drawing a line between the average starting positionand the average finish position of each joint illustrated in Figure 6.2.

the plane was measured. The slope (θyz) of each line with respect to the horizontal,

i.e., the Y -axis was also computed (In the Y Z-plane, Y -axis is the horizontal).

The following six scalar values θxy, θyz, θxz, `xy, `yz and `xz were computed

for each of the 25 joints tracked by the Kinect camera. In Table 6.1, these values are

illustrated for the first few joints. By concatenating these values for all the joints, a

150-dimensional vector was obtained. Each vector had a unique signature for each of

the exercises and hence this 150-dimensional vector was used as the feature vector.

Figure 6.4: Confusion Matrix achieved using the projections-based feature vector

84

Table 6.1: An illustration of the first 18 values of the 150-dimension feature vectorfor the Jumping Jacks exercise.

Joint θxy θyz θzx `xy `yz `zx

0 73.18 64.49 8.21 382.42 850.00 775.07

1 66.29 65.31 11.42 131.93 289.16 268.02

2 11.99 72.92 55.35 187.72 132.79 223.22

The feature vector described above were used to train a SVM and recognize

a new instance of an exercise. The accuracy of the SVM can be characterized by a

confusion matrix as shown in Fig.6.4. Each row of the matrix indicates the input

and each column indicates the output. Hence, the accuracy of prediction is along the

diagonals. It can be noticed that the accuracy of the SVM using projections-based

feature vector in recognizing the jumping jack exercise among 3 other exercises is

94.4%.

To illustrate exercising error detection using the Kinect camera, horizontal

shoulder adduction exercise is used as an example. The standing horizontal shoulder

adduction exercise requires the participant to maintain their posture as well as control

of the resistance band during the concentric and eccentric phases of the exercise.

This exercise primarily targets the chest and anterior deltoid muscles and requires

movement of the shoulder girdle with minimal movement of any other body joints.

Furthermore, the motion of the wrists should be smooth as the wrists approach the

end of the concentric phase. If the participant cannot handle tension in the resistance

band at the end of the concentric phase, the wrists tend to vibrate, which is classified

as an error.

85

Figure 6.5: Comparison of the Standing shoulder adduction exercise performed by anexpert participant(on the left) and a beginner participant(on the right). Observe lotof variations in the wrist position of the beginner participant. This shows that thebeginner participant is not in control of the exercise and is prone to injury.

Figure 6.5, plots the 3-D motion of 9 out of 25 joints as tracked by the

Kinect sensor. The figure on the left depicts the advanced exerciser while on the

right corresponds to the beginner exerciser. It can be seen that the motion of the

wrist joints of the advanced exerciser is uniform along the y axis while the motion

of other joints are minimal during full execution of the exercise. This indicates that

the participant is in full control of the motion and is using only the targeted muscles

while performing the exercise. Also observe in the figure, minimal variation of the

head, hip joints of the advanced exerciser compared to the beginner. This indicates

that the advanced exerciser, is in complete control of the exercise and maintains a

good body posture.

86

6.2 Multiple Kinect Cameras

The Kinect camera and the skeletal tracking algorithm provide 3D coordinates for the

location of twenty five joints. These coordinates are with respect to the location of the

camera as the origin. As already noted, when an exercise involves a complex sequence

of motions, not all the joints can be tracked by a single camera because of occlusions.

Multiple Kinect cameras are threfore used to accurately track the locations of all the

joints at all times. However, if a joint is occluded on one Kinect, the data from the

other Kinect is required to transform into one frame of reference. This problem is

solved using the technique Single Value Decomposition discussed in [23] and is out of

scope of the thesis.

To illustrate how the above transformation technique worked, the multiple

Kinect camera setup as shown in Figure 6.6 was laid out. A participant was asked to

perform Lunges exercise, while a front facing (Kinect Camera 1) and a side (Kinect

camera 2) facing cameras recorded the exercise activity. This exercise was selected

because during the exercise activity there would be brief moments during which the

back leg ankle joint is occluded in the Kinect camera 1. This behavior is shown in

Figure 6.7. It can be observed from the figure that when the participant back knee

bends down and nearly touches the ground, the knee joint occludes the ankle joint,

hence the data from the Kinect camera 1 goes from tracking(1) to un-tracked(0)

periodically. However, observe that the Kinect camera 2 has no problem tracking

this joint as evident in the bottom plot. The above discussed SVD technique is then

applied to Kinect camera 2 ankle joint data to transform into the Kinect camera 1

frame of reference as shown in Figure 6.8. Observe how the transformed data (green

data points) from Kinect camera 2 makes up for the untracked data frames from

Kinect camera 1(blue data points).

87

∆∆

Kinect Camera 1

Figure 6.6: The 3D Joint coordinates from each camera are provided by consideringthe location of the camera as the origin. When data for the same exercise are collectedusing multiple cameras, it is necessary to translate and/or rotate the coordinates fromone camera to the frame of reference of the other camera.

88

Figure 6.7: Demonstrating the need of multiple Kinect camera. While a participantperforming Lunges exercise, the Kinect camera 1 has un-tracked frames of ankle jointduring brief moments when the participant bent forward. During this period, Kinectcamera 2 has no problem tracking the joint as its positioned with an angle to theparticipant. Therefore Kinect camera 2 frames can be used in place of Kinect camera1, however they need to be transformed to Kinect camera 1 frame of reference.

89

Figure 6.8: Superimposed frames from Kinect camera 1 frames(blue data points)and transformed Kinect camera 2(green data points) data frames for the ankle jointduring a Lunges exercise. Observe the green data points cluster highlighted in thecircle appear in the absence of blue data points. The combined data can now be usedto analyze if the participant made an error during the exercise activity.

90

Tab

le6.

2:P

oten

tial

Err

ors

inJum

pin

gJac

ks

HIC

TE

xer

cise

s

Poss

ible

Pro

ble

mA

reas

of

HIC

TE

xerc

ises

NP

osi

tion

KeyJoin

ts

Not

landin

gon

the

bal

lof

the

foot

duri

ng

jum

p1

Fro

nta

lK

nee

[13,17] ,

Ankle

[14,18]

and

Foot

[15,19]

Not

keep

ing

hea

dst

raig

ht

orey

esfo

rwar

dw

hen

jum

pin

g1

Fro

nta

lH

ead[3] ,

Nec

k[2]

and

Spin

esh

ould

er[20]

Knee

sar

enot

flex

edat

the

tim

eof

landin

g1

Fro

nta

lK

nee

[13,17]

Arm

san

dle

gsar

enot

coor

din

ated

(not

synch

roniz

ed)

1F

ronta

lK

nee

[13,17]

and

Elb

ow[5,9]

Dis

tance

bet

wee

nth

ele

gsto

om

uch

orto

olitt

le1

Fro

nta

lF

oot

[15,19]

Jer

km

ovem

ent

ofth

eju

mpin

gja

cks

1F

ronta

lK

nee

[13,17] ,

Ankle

[14,18]

and

Elb

ow[5,9]

91

Tab

le6.

4:P

oten

tial

Err

ors

inW

all

Sit

HIC

TE

xer

cise

s

Poss

ible

Pro

ble

mA

reas

of

HIC

TE

xerc

ises

NP

osi

tion

KeyJoin

ts

Quad

rice

ps(

thig

h)

and

low

erle

gar

enot

90deg

rees

2Sag

itta

l[L

,R]

Hip

[12,16] ,

Knee

[13,17]

and

Ankle

[14,18]

Bac

kis

not

flat

agai

nst

the

wal

l1

Fro

nta

lSpin

ebas

e[0] ,

Spin

em

id[1]

and

Spin

esh

ould

er[20]

Bac

kis

arch

ed1

Fro

nta

lSpin

ebas

e[0] ,

Spin

em

id[1]

and

Spin

esh

ould

er[20]

Hea

d,

shou

lder

san

dhip

touch

the

wal

lflat

1F

ronta

lH

ead[3] ,

Shou

lder

[4,8]

and

Hip

[12,16]

Hee

lsnot

kept

ongr

ound

1F

ronta

lK

nee

[13,17] ,

Ankle

[14,18]

and

Foot

[15,19]

92

Tab

le6.

6:P

oten

tial

Err

ors

inP

ush

Up

HIC

TE

xer

cise

s

Poss

ible

Pro

ble

mA

reas

of

HIC

TE

xerc

ises

NP

osi

tion

KeyJoin

ts

Sta

rtin

gp

osit

ion

the

body

isnot

stra

ight

from

hee

lto

hea

d2

Sag

itta

l[L

,R]

Hea

d[3] ,

Nec

k[2] ,

Spin

ebas

e[0] ,

Hip

[12,16] ,

Ankle

[14,18] ,

and

Knee

[13,17]

Butt

ishig

hor

low

atst

arti

ng

pos

itio

n2

Sag

itta

l[L

,R]

Ankle

[14,18] ,

Hip

[12,16]

and

Spin

ebas

e[0]

Han

dp

osit

ion

isto

ow

ide

ornar

row

(must

be

shou

lder

wid

thap

art)

1F

ronta

lShou

lder

[4,8]

and

Han

d[7,11]

Dow

nan

dup

body

mot

ion

isnot

smoot

h(m

ust

godow

nan

dup

ason

eunit

)1

Sag

itta

lShou

lder

[4,8]

Elb

ows

stic

kou

tw

hen

the

body

goes

dow

n(e

lbow

sm

ust

be

clos

eto

body

duri

ng

dow

nan

dup

mov

emen

tof

the

body)

3F

ronta

lan

dSag

itta

l[L

,R]

Elb

ow[5,9]

and

Shou

lder

[4,8]

Hea

dis

hyp

erex

tended

duri

ng

upw

ard

mov

emen

tor

chin

isflex

eddow

nw

ard

1Sag

itta

lH

ead[3]

and

Nec

k[2]

93

Tab

le6.

7:P

oten

tial

Err

ors

inA

bdom

inal

Cru

nch

HIC

TE

xer

cise

s

Poss

ible

Pro

ble

mA

reas

of

HIC

TE

xerc

ises

NP

osi

tion

KeyJoin

ts

Bac

kis

not

flat

onth

egr

ound

duri

ng

crunch

1T

ransv

erse

Spin

ebas

e[0] ,

Spin

em

id[1]

and

Spin

esh

ould

er[20]

Knee

flex

ion

isto

om

uch

orto

olitt

leduri

ng

the

crunch

pos

itio

n1

Tra

nsv

erse

Knee

[13,17]

Hea

dis

flex

edto

om

uch

tow

ards

the

ches

t(m

ust

be

stra

ight)

duri

ng

the

crunch

(upw

ard

mot

ion)

1Sag

itta

lH

ead[3] ,

Nec

k[2]

and

Spin

eShou

lder

[20]

Fee

tar

enot

stat

ionar

yon

the

floor

when

crunch

acti

onis

don

e(m

ust

be

onth

egr

ound)

2Sag

itta

l[L

,R]

Knee

[13,17] ,

Ankle

[14,18]

and

Foot

[15,19]

Shou

lder

isro

unded

duri

ng

the

crunch

acti

on(m

ust

be

stra

ight)

2Sag

itta

l[L

,R]

Shou

lder

[4,8]

Han

dar

enot

stra

ight

duri

ng

the

crunch

acti

on2

Sag

itta

l[L

,R]

Shou

lder

[4,8] ,E

lbow

[5,9]

and

Han

d[7,11]

94

Tab

le6.

8:P

oten

tial

Err

ors

inSte

p-u

pon

toC

hai

rH

ICT

Exer

cise

s

Poss

ible

Pro

ble

mA

reas

of

HIC

TE

xerc

ises

NP

osi

tion

KeyJoin

ts

The

step

pin

gle

gon

the

chai

ris

not

90deg

rees

atth

ehip

,knee

and

ankle

2Sag

itta

l[L

,R]

Hip

[12,16] ,

Knee

[13,17]

and

Ankle

[14,18]

Duri

ng

clim

bth

eupp

erb

ody

isnot

stra

ight

1F

ronta

lSpin

ebas

e[0] ,

Spin

em

id[1]

and

Spin

esh

ould

er[20]

Body

isnot

stra

ight

afte

rth

eco

mple

tion

ofth

ecl

imb

1F

ronta

lH

ead[3] ,

Nec

k[2] ,

Spin

em

id[1] ,

Spin

ebas

e[0] ,

Hip

[12,16] ,

Knee

[13,17]

and

Ankle

[14,18]

The

step

pin

gle

g(dow

n)

isnot

flex

edat

conta

ct1

Fro

nta

lK

nee

[13,17] ,

Ankle

[14,18]

and

Foot

[15,19]

Aft

erth

est

eppin

gle

g(d

own)

conta

cts

the

grou

nd

the

chai

rle

gm

ust

be

90deg

rees

athip

,knee

and

ankle

1F

ronta

lH

ip[12,16] ,

Knee

[13,17] ,

and

Ankle

[14,18]

The

whol

eac

tion

isnot

exec

ute

din

”one

smoot

hm

ovem

ent”

from

star

tto

finis

h2

Sag

itta

l[L

,R]

all[1−25]

Chai

rhei

ght

isto

hig

hw

hic

hcr

eate

sex

acer

bat

ion

2Sag

itta

l[L

,R]

Hip

[12,16] ,

Knee

[13,17]

and

Ankle

[14,18]

95

Tab

le6.

9:P

oten

tial

Err

ors

inSquat

HIC

TE

xer

cise

s

Poss

ible

Pro

ble

mA

reas

of

HIC

TE

xerc

ises

NP

osi

tion

KeyJoin

ts

Sta

rtin

gfe

etp

osit

ion

isnot

shou

lder

wid

thap

art(

too

nar

row

orto

ow

ide)

1F

ronta

lF

oot

[15,19]

and

Shou

lder

[4,8]

At

Squat

pos

itio

nth

ean

gle

atth

eknee

and

ankle

isnot

90deg

rees

2Sag

itta

l[L

,R]

Hip

[12,16] ,

Knee

[13,17] ,

Ankle

[14,18]

and

Foot

[15,19]

Tru

nk

pos

itio

nis

not

stra

ight

and

ben

tat

hip

sfo

rwar

d1

Fro

nta

lSpin

ebas

e[0] ,

Spin

em

id[1]

and

Spin

esh

ould

er[20]

Knee

sar

enot

dir

ectl

yov

erto

es1

Fro

nta

lK

nee

[13,17]

and

Foot

[15,19]

Han

ds

are

not

stra

ight

and

par

alle

lto

the

grou

nd

2Sag

itta

l[L

,R]

Shou

lder

[4,8] ,

Elb

ow[5,9]

and

Wri

st[6,10]

Knee

sb

owin

war

dor

outw

ard

1F

ronta

lK

nee

[13,17]

96

Tab

le6.

10:

Pot

enti

alE

rror

sin

Tri

ceps

dip

onC

hai

rH

ICT

Exer

cise

s

Poss

ible

Pro

ble

mA

reas

of

HIC

TE

xerc

ises

NP

osi

tion

KeyJoin

ts

Sta

ring

pos

itio

nof

the

body

onth

ech

air

isnot

stra

ight(

hee

lto

hea

d)

3F

ronta

lan

dSag

itta

l[L

,R]

Hea

d[3] ,

Nec

k[2] ,

Spin

em

id[1] ,

Spin

ebas

e[0] ,

Hip

[12,16] ,

Knee

[13,17]

and

Ankle

[14,18]

Han

ds

tow

ide

apar

t1

Fro

nta

lH

and

and

Shou

lder

[4,8]

Han

ds

supp

orti

ng

the

chai

ris

not

stra

ight

atel

bow

s2

Sag

itta

l[L

,R]

Shou

lder

[4,8] ,

Elb

ow[5,9]

and

Han

d[7,11]

At

dip

the

arm

do

not

com

ple

teth

efu

llra

nge

ofm

otio

nin

flex

edp

osit

ion

3F

ronta

lan

dSag

itta

l[L

,R]

Shou

lder

[4,8] ,

Elb

ow[5,9]

and

Han

d[7,11]

Duri

ng

the

dip

the

body

isnot

goin

gst

raig

ht

dow

nw

ith

legs

stra

ight

3F

ronta

lan

dSag

itta

l[L

,R]

Knee

[13,17] ,

Hip

[12,16]

and

Spin

ebas

e[0]

Elb

ows

are

not

at90

deg

rees

atfu

lldip

3F

ronta

lan

dSag

itta

l[L

,R]

Shou

lder

[4,8] ,

Elb

ow[5,9]

and

Han

d[7,11]

97

Tab

le6.

11:

Pot

enti

alE

rror

sin

Pla

nk

HIC

TE

xer

cise

s

Poss

ible

Pro

ble

mA

reas

of

HIC

TE

xerc

ises

NP

osi

tion

KeyJoin

ts

The

conta

ctar

eaw

ith

grou

nd

isnot

flat

wit

hth

efo

rear

ms

3F

ronta

lan

dSag

itta

l[L

,R]

Han

d[7,11] ,

Elb

ow[5,9]

and

Shou

lder

[4,8]

For

earm

sar

enot

shou

lder

wid

thap

art(

too

nar

row

orto

ow

ide)

3F

ronta

lan

dSag

itta

l[L

,R]

Elb

ow[5,9]

and

Shou

lder

[4,8]

The

whol

eb

ody

isnot

stra

ight

from

hea

dto

hee

l(b

utt

inline

wit

hth

ere

stof

the

body)

3F

ronta

lan

dSag

itta

l[L

,R]

Hea

d[3] ,

Nec

k[2] ,

Spin

em

id[1] ,

Spin

ebas

e[0] ,

and

Knee

[13,17]

Angl

esat

ankle

,sh

ould

eran

del

bow

isnot

90deg

rees

3F

ronta

lan

dSag

itta

l[L

,R]

Han

d[7,11] ,

Elb

ow[5,9] ,

Shou

lder

[4,8] ,

Hip

Knee

[13,17] ,

Ankle

[14,18]

and

Foot

[15,19]

98

Tab

le6.

12:

Pot

enti

alE

rror

sin

Hig

hK

nee

s/R

unnin

gin

Pla

ceH

ICT

Exer

cise

s

Poss

ible

Pro

ble

mA

reas

of

HIC

TE

xerc

ises

NP

osi

tion

KeyJoin

ts

Upp

erto

rso

isnot

stra

ight(

upri

ght)

duri

ng

the

leg

dri

ves

1F

ronta

lSpin

ebas

e[0] ,

Spin

em

id[1]

and

Spin

esh

ould

er[20]

99

Tab

le6.

13:

Pot

enti

alE

rror

sin

Lunge

HIC

TE

xer

cise

s

Poss

ible

Pro

ble

mA

reas

of

HIC

TE

xerc

ises

NP

osi

tion

KeyJoin

ts

At

lunge

the

upp

erb

ody

isnot

stra

ight

1F

ronta

lSpin

ebas

e[0] ,

Spin

em

id[1]

and

Spin

esh

ould

er[20]

The

step

pin

gle

gis

not

90deg

rees

atan

kle

,knee

and

hip

2Sag

itta

l[L

,R]

Hip

[12,16] ,

Ankle

[14,18] ,

Knee

[13,17]

and

Foot

[15,19]

Knee

isnot

dir

ectl

yov

erth

eto

es1

Fro

nta

lK

nee

[13,17]

and

Foot

[15,19]

Rea

rle

gis

not

90deg

rees

atth

eK

nee

2Sag

itta

l[L

,R]

Hip

[12,16] ,

Knee

[13,17]

and

Ankle

[14,18]

Dip

duri

ng

the

lunge

isnot

stra

ight

dow

n1

Fro

nta

lSpin

ebas

e[0]

Not

step

pin

gat

the

consi

sten

tsp

otduri

ng

the

lunge

1F

ronta

lF

oot

[15,19]

100

Tab

le6.

14:

Pot

enti

alE

rror

sin

Push

-Up

HIC

TE

xer

cise

s

Poss

ible

Pro

ble

mA

reas

of

HIC

TE

xerc

ises

NP

osi

tion

KeyJoin

ts

Duri

ng

rota

tion

the

arm

s(b

oth)

do

not

mak

e90

deg

atth

eto

rso

2Sag

itta

l[L

,R]

Spin

em

id[1] ,

Spin

esh

ould

er[20] ,

Elb

ow[5,9]

Arm

saf

ter

rota

tion

are

not

inst

raig

ht

line

2Sag

itta

l[L

,R]

Han

d[7,11] ,

Elb

ow[5,9]

and

Shou

lder

[4,8]

Supp

orti

ng

arm

isnot

stra

ight

alon

gw

ith

the

shif

ting

arm

2Sag

itta

l[L

,R]

Han

d[7,11] ,

Elb

ow[5,9] ,

Shou

lder

[4,8] ,

Spin

esh

ould

er[20]

101

Tab

le6.

15:

Pot

enti

alE

rror

sin

Sid

eP

lank

HIC

TE

xer

cise

s

Poss

ible

Pro

ble

mA

reas

of

HIC

TE

xerc

ises

NP

osi

tion

KeyJoin

ts

The

conta

ctin

gfo

rear

mis

not

at90

deg

atth

eel

bow

and

shou

lder

1Sag

itta

lShou

lder

[4,8] ,

Elb

ow[5,9]

and

Han

d[7,11]

The

tota

lb

ody(h

ead

tosi

de

ofth

efo

ot)

isnot

ina

stra

ight

line

duri

ng

the

pla

nk

1Sag

itta

lH

ead[3] ,

Nec

k[2] ,

Spin

esh

ould

er[20] ,

Hip

[12,16] ,

Knee

[13,17]

and

Ankle

[14,18]

The

hip

sar

enot

inline

wit

hth

ere

stof

the

body(t

oo

low

orto

ohig

h)

1Sag

itta

lSpin

ebas

e[0] ,

Hip

[12,16]

and

Knee

[13,17]

Hea

dan

dnec

kar

enot

inline(

stra

ight)

wit

hth

ere

stof

the

body(t

oo

low

orto

ohig

h)

1Sag

itta

lH

ead[3] ,

Nec

k[2] ,

Spin

esh

ould

er[20]

and

Spin

em

id[1]

Res

ting

han

dis

not

stra

ight

atth

esi

de

ofth

eb

ody

duri

ng

the

pla

nk

1Sag

itta

lSpin

ebas

e[0] ,

Spin

em

id[1] ,

Spin

esh

ould

er[20] ,

Shou

lder

[4,8] ,

Elb

ow[5,9]

and

Han

d[7,11]

102

6.3 Estimating Ground Reaction Forces during a Vertical Jump

Figure 6.9 illustrates the kinematics of a single jump. This figure plots the spine base

joint position from the data collected using the kinect camera. Point A is the initial

start position from rest. During the time interval between B to C, the participant

squats down in preparation for jump. The interval from C to D is the take-off time

during which the participant exerts force on to the force plate. The point D is the

exact time instance at which the maximum force is applied on the force plate and the

participant takes-off into the air. The interval from D to E is the time for which the

participant is in the air. Time E is the exact instance at which the participant touches

down on to the Force plate. During E to F the participant is in the landing phase

and starts to exert force again on to the force plate. The interval from F to G is the

time it takes for the participant to recoil back to the rest position. To calculate the

ground reaction forces during the landing and takeoff phases, the impulse-momentum

method as described in section 6.3.1is used. In order to compare the accuracy of the

results from the data collected using Kinect 2.0, a traditional force place was used

that directly reported the ground reaction forces.

6.3.1 Impulse-Momentum Approach

Since the forces between the participant feet and ground are equal in magnitude and

opposite in direction, and since the times for which these forces act are equal in

magnitude, it follows that the impulses experienced by the participant and ground2

are also equal in magnitude and opposite in direction. As an equation during the

take-off and landing phases, this can be stated as

2The participant was required to stand on a force plate during the jump

103

Figure 6.9: Plot of spine base over a single jump.

∫ D

B

FGRF dt = m · vto +

∫ D

B

m · g dt Take-off phase (6.1)∫ F

E

FGRF dt = m · vtl +

∫ F

E

m · g dt Landing phase (6.2)

where, FGRF is the ground reaction forces captured using the force plate, vto is the

instantaneous velocity during the Take-off phase calculated from the data collected

from Kinect or Vicon, vtl is the instantaneous velocity during the Landing phase cal-

culated from the data collected from Kinect or Vicon, m is the mass of the participant,

and g is the gravitational acceleration (9.8 m/sec2).

Figure 6.10 verifies that our data is in accordance with what is expected from

equation 6.1. It can be observed from the figure that the impulse calculated at the

104

Figure 6.10: Impulse calculated using force plate, vicon and kinect data. The graphsshows that the impulse calculated using force plate data is approximately equal toimpulse calculated using kinect and vicon data. This shows that the change in mo-mentum is conserved during jumping phase.

force plate is approximately equal to the impulse calculated using the data collected

from Kinect and Vicon.

105

CHAPTER VII

DISCUSSION

In this thesis, the problem of Personalized Wellness Management was formulated as

a decision making under uncertainty problem, where an individual makes a series

of decisions in each time step, regarding the amount of nutrition calorie intake and

exercise calorie expenditure to indulge in each day. The outcome of such decisions

depends on various factors such as individual body metabolism, the ability to adhere

to the action decided, the motivation level of the individual, various metabolic changes

to the human body, etc. Under an appropriate set of assumptions, this problem was

mathematically formulated under Completely Observable Markov decision processes

and Partially-Observable Markov decision processes. Chapter 3 presented the system

model under each of these frameworks and the use of various commercially available

solvers to solve them.

To formulate the problem as a Markov decision processes, it was necessary for

the MDP State to be completely observable, and hence the weight of the participant

was chosen as it can be measured with out ambiguity. This required the transition

model between States depend on the individual body metrics and had to be derived

from carefully conducted and validated experimental studies. Since such data did

not exist, mathematical models of human weight dynamics discussed in Section 4.1.4

was used. These models however inherently possessed computation errors and did

not account for various other human factors such as ethnicity, disease condition etc.

Therefore an uncertainty model as described in Section 3.1.3 was included in the

model. Such a formulation allowed us to integrate domain expert knowledge and

106

user preferences into the framework through the use of reward functions. To demon-

strate this, three different reward functions were used namely 1) quick weight loss, in

which participant desired to reach target weight in the least amount of time 2) safe

weight loss, in which participant desired to reach target weight in a safely manner

recommended by experts and 3) personalized weight loss, in which the participant

stated his preferences in selecting exercise activity level. The outcomes of each strat-

egy was executed on an hypothetical participant and the results were described in

Section 3.3.1. Since the transition matrix and reward functions were stationary and

did not change with respect to time, the policy computed was also stationary. This

meant that if for some unknown reason the participant had trouble changing his state,

the action mapped by the policy did not change. This method also did not account

for the motivation level of the participant to adhere to the recommended action.

To account for the limitations of the Completely Observable Markov Decision

process, Partially Observable Markov decision process was explored. Since motivation

level is not directly measured, the adherence level of the participant to the recom-

mended action is used as the observation to maintain a probability distribution of

motivation level of the participant. However, this method required extensive data to

model the PWM problem through state transition matrices, observation transition

matrices. Such data was difficult to obtain and also required continuous monitoring

of the participant for long periods of time. Hence these matrices were hand con-

structed and solved using a widely accepted open source solver called pomdp-solve.

In POMDP modeling, the motivation distribution is used to map the intensity level

of the action. This intensity level is used as a filter to eliminate actions in the action

set of MDP. This filtered action set is used by the MDP to select an exercise and

nutrition action. Both the MDP and POMDP frameworks are computed for an op-

timal policy and therefore require large amounts of experimental data to formulate

107

the system dynamics. However, in the human wellness domain, we assume that the

participant is a rational agent and therefore good enough solutions suffice. The Goal-

Seeking paradigm that is presented in this thesis is one such paradigm that does not

optimize for actions, but instead selects the best action among the available choices.

The Goal-Seeking artifacts identified in section 2.7 formulated the system

model for the Personal Wellness Management domain. The three compartment model

of human weight dynamics was adapted as the reflection mapper artifact. A reward

function that captured safe weight loss dynamics was used to assign rewards to conse-

quences of each action served as the cost mapper artifact. Each action is run through

the reflection mapper to estimate the amount of weight change for the next time step

and was assigned a cost using the cost mapper. The design of the Evaluation Mapper

had multiple facets to it. In this investigation, we assumed that the decision-maker

can make the observation of whether the participant is adhered or not, by examin-

ing user reported data and physical examination. The design of more robust tools

that help the participant recognize exercising errors in real-time are currently under

investigation. The software design chapter 5 described the database, user interface

designs that were implemented for robust interaction of the participant, goal-seeking

software and the back-end database. To allow the participant to use goal-seeking

software services using either a smart phone or a PC, RESTful API’s were designed

that allowed for a flexible and scalable deployment. Several more aspects of this

system including human behavior models and tools for quantifying motivation along

with a robust tool to track nutrition behavior need to be developed. These tools must

be validated in field trials and revised before they can be commercialized.

108

CHAPTER VIII

CONCLUSIONS

This dissertation described the goal-seeking paradigm and presented a formulation

of a decision-maker for personal wellness management. The decision-maker estab-

lishes performance limits and measures the actual performance to adjust the wellness

prescription. Ultimately, the decision-maker seeks to transform individuals to be

intrinsically motivated to improve their personal wellness by recommending actions

that correlate with the adherence level of the participant. The comprehensive man-

ner in which the decision-maker incorporates knowledge from disparate domains, the

manner in which it integrates humans in the decision-making loop, and the focus on

good-enough solutions (instead of optimal solutions) make this a viable framework to

empower personal wellness management. In the future, this framework can be made

more robust by incorporating more complicated human weight dynamic models and

utility theory based action selection.

109

BIBLIOGRAPHY

[1] S.P. Keehan, G.A. Cuckler, A.M. Sisko, A.J. Madison, S.D. Smith, D.A. Stone,J.A. Poisal, C.J. Wolfe, and J.M. Lizonitz. National health expenditure pro-jections, 2014-24: Spending growth faster than recent trends. Health Affairs,2015.

[2] C.L. Ogden, M.D. Carroll, and B.K. Kit et. al. Prevalence of childhood andadult obesity in the united states, 2011-2012. The Journal of American MedicalAssociation, 311(8):806–814, 2014.

[3] L. Chapman. Meta-evaluation of worsite health promotion economic returnstudies: 2012 update. American Journal of Health Promotion, 26(4):TAHP1–TAHP12, 2012.

[4] P. Coorevits, M. Sundgren, G. O. Klein, A. Bahr, B. Claerhout, C. Daniel,M. Dugas, D. Dupont, A. Schmidt, P. Singleton, G. De Moor, and D. Kalra.Electronic health records: New opportunities for clinical research. Journal ofInternal Medicine, 274(6):547–560, 2013.

[5] D. Georges, S. Mats, and K. Dipak e.t. al. Using electronic health recordsfor clinical research: The case of the ehr4cr project. Journal of BiomedicalInformatics, 53:162–173, 2015.

[6] H. Tao, L. Liang, F. Xuexian, A. Peng, M. Junxia, and W. Fudi. Promisesand challenges of big data computing in health sciences. Big Data Research,2(1):2–11, 2015.

[7] F. Keith, D. Darcy, and C. Nitesh. Scaling and contextualizing personalizedhealthcare: A case study of disease prediction algorithm integration. Journalof Biomedical Informatics, 57:377–385, 2015.

[8] N. Kenney, A. Ghoting, S.R. Steinhubl, W.F. Stewartand B. Malin, and J. Sun.PARAMO: A PARAllel predictive Modeling platform for healthcare analyticresearch using electronic health records. Journal of Biomedical Informatics,48:160–170, 2014.

[9] S. Jimeng and C.K. Reddy. Big Data Analytics for Healthcare. In InternationalConference on Knowledge Discovery and Data, page 1525, 2013.

110

[10] H.A.Simon. Models of Man: Social and Rational- Mathematical Essays onRational Human Behavior in a Social Setting. John Willey & Sons, 1957.

[11] P.B. Shull, W. Jirattigalachote, M. A. Hunt, M. R. Cutkosky, and S.L. Delp.Quantified self and human movement: A review on the clinical impact of wear-able sensing and feedback for gait analysis and intervention. Gait and Posture,40(1):11–19, May 2014.

[12] C.C. Chow and K.D. Hall. The dynamics of human body weight change. PLoSComputational Biology, 4(3):1–13, 2008.

[13] M Chippa and S. Sastry. Battery systems management using a goal-seekingframework. In 9th Annual IEEE international Systems Conference (SysCon),pages 335–339, 2015.

[14] M. Chippa, S.M. Walen, S. Sastry, and F. Douglas. Goal-seeking formulationfor empowering personalized wellness management. In Medical Cyber PhysicalSystems, 2013.

[15] M.D. Mesarovic and Y. Takahara. Mathematical Theory of General Systems.Academic Press, 1974.

[16] M.D. Mesarovic, D.L. McGuiness, and D.A. West. Cybernetics of global change:Human dimension and managing of complexity, most policy paper. UnitedNations Educational Scientific and Cultural Organization, 1996.

[17] S. Sastry. Smartspace for automation. Assembly Automation, 24(2):201–209,2004.

[18] Y. Takahara and M.D. Mesarovic. Organization Structure - Cybernetic SystemsFoundation. Kluwer Academic, 2003.

[19] S.Sastry, O. Gimdogmus, T.T. Hartley, and R. J. Veillette. Coordinated dis-charge of a collection of batteries. Journal of Power Sources, 166(1):284–296,2007.

[20] H. Pidaparthy, S. Narayan, and S. Sastry. Automated exercise feedback systemusing a non-invasive sensor. In Workshop on Applications of Computer Vision,Graphics and Image Processing, dec 2014.

[21] S.P.R. Gaddam, M.K. Chippa, S. Sastry, A. Ange, V. Berki, and B.L. Davis.Estimating forces during exercise activity using non-invasive kinect camera. InInternational Conference on Computational Science and Computational Intel-ligence, pages 825–828, Nov 2015.

[22] P. Fernando. Hybrid recommender system for exercises. Master’s thesis, TheUniversity of Akron, Computer Engineering, 2016.

111

[23] S. Vankamamidi. Fusing joint information from multiple kinect cameras todetect errors in exercises. Master’s thesis, The University of Akron, ComputerEngineering, 2016.

[24] U.S Food and Drug Administration. Paving the way for personalized medicinefda’s role in a new era of medical product development, 2013.

[25] U. S. Office of the Actuary. National health expenditures projection. Technicalreport, Centers for Medicare and Medicaid Services, 2010.

[26] M. Barkhordari and M. Niamanesh. ScaDiPaSi: An Effective Scalable andDistributable MapReduce-Based Method to Find Patient Similarity on HugeHealthcare Networks. Big Data Research, 2(1):19–27, 2015.

[27] United States Department of Agriculture. www.choosemyplate.gov.

[28] S.R/ Colberg, R. J. Sigal, and B. Fernhall et.al. Exercise and type 2 diabetes.Diabetes Care, 33(12):e147–e167, Dec 2010.

[29] NIH Publication No. 98-4083. National institutes of health. clinical guidelines onthe identification, evaluation, and treatment of overweight and obesity in adults:The evidence report. National Heart, Lung, and Blood Institute;, September1998.

[30] R.H. Eckel, S.E. Kahn, and E. Ferrannini et.al. Obesity and type 2 diabetes:What can be unified and what needs to be individualized? The Journal ofClinical Endocrinology & Metabolism, 96(6):1654–1663, Jun 2011.

[31] V. Kotsis, S. Stabouli, and S. Papakatsika et.al. Mechanisms of obesity-inducedhypertension. Hypertension Research, 33(5):386–93, May 2010.

[32] C.J. Lavie, R. V. Milani, and H.O. Ventura. Obesity and cardiovascular disease:Risk factor, paradox, and impact of weight loss. Journal of the American Collegeof Cardiology, 53(21):1925–1932, May 2009.

[33] K. D Hall, G. Sacks, D. Chandramohan, C. C Chow, Y C. Wang, S.L. Gort-maker, and B. A Swinburn. Quantification of the effect of energy imbalance onbodyweight. Lancet, 2011.

[34] B.A. Swinburn, G. Sacks, and S.K. Lo et al. Estimating the changes in en-ergy flux that characterize the rise in obesity prevalence. American Journal ofClinical Nutrition, 89:1723–1728, 2009.

[35] J.O. Hill, H.R. Wyatt, G.W. Reed, and J.C. Peters. Obesity and the environ-ment: where do we go from here? Science, 299:853–55, 2003.

112

[36] B. Swinburn, G. Sacks, and E.Ravussin. Increased food energy supply is morethan sufficient to explain the us epidemic of obesity. American Journal ofClinical Nutrition, 90:1453–1456, 2009.

[37] K.D. Hall, J. Guo, M. Dore, and C.C. Chow. The progressive increase of foodwaste in america and its environmental impact. Plos ONE, 4(11):e7940, 2009.

[38] D. T. Levy, P. L. Mabry, Y.C.Wang, S. Gortmaker, T. T.K. Huang, T. Marsh,M. Moodie, and B. Swinburn. Simulation models of obesity: a review of theliterature and implications for research and policy. Obesity Reviews, 12:378–394,2010.

[39] A.R. Behnke, B.G. Feen, and W.C. Welham. The specific gravity of healthymen. Journal of the American Medical Association, 118(7):495–498, 1942.

[40] G.B. Forbes. Weight loss during fasting: Implications for the obese. TheAmerican Journal of Clinical Nutrition, 23(9):1212–1219, 1970.

[41] V. W. Antonetti. The equations governing weight change in human beings. TheAmerican Journal of Clinical Nutrition, 26(1):64–71, 1973.

[42] N.L. Keim, C.A. Blanton, and M.J. Kretsch. America’s obesity epidemic: Mea-suring physical activity to promote an active lifestyle. Journal of the AmericanDietetic Association, 104(9):1398–1409, 2004.

[43] P. Trumbo, S. Schlicker, AA. Yates, and M. Poos. Dietary reference intakesfor energy, carbohydrate, fiber, fat, fatty acids, cholesterol, protein and aminoacids. Journal of the American Dietetic Association, 102(11):1621–1630, 2002.

[44] B.E. Ainsworth, W.L. Haskell, S.D. Herrmann, N. Meckes, D.R. Bassett Jr,C. Tudor-Locke, J.L.Greer, J. Vezina, M.C. Whitt-Glover, and A.S. Leon.Compendium of physical activities: A second update of codes and met val-ues. Medicine & Science in Sports & Exercise, 43(8):1575–1581, 2011.

[45] K.R. Westerterp, J.H.Donkers, E.W. Fredrix, and P. Boekhoudt. Energy intake,physical activity and body weight: a simulation model. The British Journal ofNutrition, 73(3):337–347, 1995.

[46] N.J. Fuller, S.A. Jebb, M.A. Laskey, W.A. Coward, and M. Elia. Four-componentmodel for the assessment of body composition in humans: comparison with al-ternative methods, and evaluation of the density and hydration of fat-free mass.Clinical Science, 82(6):687–693, 1992.

[47] J.S. Garrow. Energy balance in man- an overview. American Journal of ClinicalNutrition, 45:1114–1119, 1987.

113

[48] S.A. Jebb, A.M. Prentice, G.R. Goldberg, P.R. Murgatroyd, A.E. Black, andW.A. Coward. Changes in macronutrient balance during over- and underfeedingassessed by 12-d continuous whole-body calorimetry. The American Journal ofClinical Nutrition, 64(3):259–266, 1996.

[49] G.B. Forbes. Body fat content influences the body composition response tonutrition and exercise. Annals of the New York Academy of Sciences, 904:359–365, 2000.

[50] J.P. Flatt. Carbohydrate-fat interactions and obesity examined by a two-compartment computer model. Obesity Research, 12(12):2013–2022, 2004.

[51] H. Rahmandad. Human growth and body weight dynamics: An integrativesystems model. PLoS ONE, 9(12):e114609., 2014.

[52] D.M. Thomas, C.K. Martin, S. Heymsfield, L.M. Redmanand A. Dale, andJ.A. Levine. A Simple Model Predicting Individual Weight Change in Humans.Journal of Biological Dynamics, 5(6):579–599, 2011.

[53] A. E.Dugdale and P. R. Payne. Pattern of lean and fat deposition in adults.Nature, 266:249–351, 1977.

[54] K.D. Hall. Body fat and fat-free mass inter-relationships: Forbes’s theory re-visited. The British Journal of Nutrition, 97(6):1059–1063, 2007.

[55] G.B. Forbes. Lean body mass-body fat interrelationships in humans. NutritionReviews, 45(8):225–231, 1987.

[56] K.D. Hall. Computational model of in vivo human energy metabolism duringsemistarvation and refeeding. American Journal of Physiology, Endocrinologyand Metabolism, 291(1):E23–E37, 2006.

[57] A. Keys, J. Brozek, A. Henschel, O. Mickelsen, and H.L. Taylor. The biology ofhuman starvation volume 1,2. Technical report, University of Minnesota, 1950.

[58] KD Hall. Mechanisms of metabolic fuel selection: modeling human metabolismand body-weight change. IEEE Engineering in Medicine and Biology Magazine,29(1):36–41, 2010.

[59] K.D. Hall. Predicting metabolic adaptation, body weight change, and en-ergy intake in humans. American Journal of Physiology, Endocrinology andMetabolism, 298(3):E449–466, 2010.

114

[60] W. H. M. Saris, S. N. Blair, M. A. Van Baak, S. B. Eaton, P. S. W. Davies, L. DiPietro, M. Fogelholm, A. Rissanen, D. Schoeller, B. Swinburn, A. Tremblay,K. R. Westerterp, and H. Wyatt. How much physical activity is enough toprevent unhealthy weight gain? outcome of the IASO 1st stock conference andconsensus statement. Obesity Reviews, 4(2):101–114, 2003.

[61] J. A.Harris and F. G. Benedict. A biometric study of human basal metabolism.In Proceedings of the National Academy of Sciences of the United States ofAmerica, 1919.

[62] N. Barrientos, J.E.D.E. Rivera, and L.M. Collins. A dynamical systems modelfor understanding behavioral interventions for weight loss. International Con-ference on Social Computing Behavioral Modeling, and Prediction, March 2010.

[63] I. Ajzen and T.J. Madden. Prediction of goal-directed behavior: Attitudes,intentions, and perceived behavioral control. Journal of Experimental SocialPsychology, 22(5):453–474, 1986.

[64] R.M. Ryan and E.L. Deci. Self-determination theory and the facilitation ofintrinsic motivation, social development and well-being. American Psychologist,55(1):68–78, 2000.

[65] J.O. Prochaska. Decision making in the transtheoretical model of behaviorchange. Medical Decision Making, 28(6):845–849, Nov 2008.

[66] T.A. Shimp and A. Kavas. The theory of reasoned action applied to couponusage. Journal of Consumer Research, 11(3):795–809, Feb 1984.

[67] N.K. Janz and M.H. Becker. The health belief model: A decade later. HealthEducation and Behavior, 22(1):1–47, March 1984.

[68] J.E. Navarro and D.E. Rivera. A dynamical systems model for weight changebehavioral interventions. Technical report, Control Systems Engineering Lab-oratory, Arizona State University, 2010.

[69] N. Barrientos, J.E.D.E Rivera, and L.M. Collins. A dynamical model for de-scribing behavioural interventions for weight loss and body composition change.Mathematical and Computer Modelling of Dynamical Systems,, 17(2):183–203,Apr 2011.

[70] N. Nandola and D.E.Rivera. Model-on-demand predictive control for nonlinearhybrid systems, with application to adaptive behavioral interventions. In 49thIEEE Conference on Decision and Control (CDC), Dec 2010.

115

[71] A. Zafra-Cabeza, D.E. Rivera, L.M. Collins, M.A. Ridao, and E.F. Camacho.A risk-based model predictive control approach to adaptive interventions inbehavioral health. In Proceedings of the 45th IEEE Conference on Decisionand Control, volume 19, pages 891–901, 2011.

[72] N. Nandola and D.E. Rivera. A novel model predictive control formulationfor hybrid systems with application to adaptive behavioral interventions. InProceedings of the 2010 American Control Conference, pages 6286–6292, 2010.

[73] A. Mahamadi and S.Sastry. Bond graph models for human behavior. In IEEEInternational Conference for Basic Science and Engineering, 2016.

[74] D. S. Alexiadis, P. Kelly, P. Daras, N. E. O’Connor, T. Boubekeur, and M. B.Moussa. Evaluating a dancer’s performance using kinect-based skeleton track-ing. In Proceedings of the 19th ACM International Conference on Multimedia,pages 659–662. ACM, 2011.

[75] E. Davaasambuu, C. Chiang, J. Y. Chiang, Y. Chen, and S. Bilgee. A microsoftkinect based virtual rehabilitation system. In The 5th International ConferenceFITAT, pages 44–50, 2012.

[76] T. Y. Lin, C. H. Hsieh, and J. D. Lee. A kinect-based system for physicalrehabilitation: Utilizing tai chi exercises to improve movement disorders inpatients with balance ability. In Modelling Symposium, pages 149–153, 2013.

[77] S. Obdrzalek, G. Kurillo, F. Ofli, R. Bajcsy, E. Seto, H. Jimison, and M. Pavel.Accuracy and robustness of kinect pose estimation in the context of coachingof elderly. In IEEE Annual International Conference of the Engineering inMedicine and Biology Society, pages 1188–1193. IEEE, 2012.

[78] D. Gonzalez-Ortega, F. J. Dıaz-Pernas, M. Martınez-Zarzuela, and M. Anton-Rodrıguez. A kinect-based system for cognitive rehabilitation exercises moni-toring. Computer Methods and Programs in Biomedicine, 113(2):620–631, 2014.

[79] D. Anton, A. Goni, A. Illarramendi, J. J. Torres-Unda, and J. Seco. Kires:A kinect-based telerehabilitation system. In Proceedings of 15th InternationalConference on e-Health Networking, Applications Services, pages 444–448. IEEE,2013.

[80] A. Ghose, K. Chakravarty, A. K. Agrawal, and N. Ahmed. Unobtrusive indoorsurveillance of patients at home using multiple kinect sensors. In Proceedingsof the 11th ACM Conference on Embedded Networked Sensor Systems, pages40–42. ACM, 2013.

116

[81] M. Robinson and M. B. Parkinson. Estimating anthropometry with microsoftkinect. In Proceedings of the 2nd International Digital Human Modeling Sym-posium, 2013.

[82] J. Han, L. Shao, D. Xu, and J. Shotton. Enhanced computer vision with mi-crosoft kinect sensor: A review. IEEE Transactions on Cybernetics, 43(5):1318–1334, 2013.

[83] J. Shotton, A. Fitzgibbon, M. Cook, T. Sharp, M. Finocchio, R. Moore, A. Kip-man, and A. Blake. Real-time human pose recognition in parts from singledepth images. In IEEE Conference on Computer Vision and Pattern Recogni-tion, pages 1297–1304, 2011.

[84] Z. Zhang. Microsoft kinect sensor and its effect. IEEE MultiMedia, 19(2):4–10,2012.

[85] B. BonnechAre, B. Jansen, P. Salvia, H. Bouzahouene, L. Omelina, F. Moiseev,V. Sholukha, J. Cornelis, M. Rooze, and S. V. S. Jan. Validity and reliability ofthe kinect within functional assessment activities: Comparison with standardstereophotogrammetry. Gait and Posture, 39(1):593–598, 2014.

[86] A. Fern’ndez-Baena, A. Susin, and X. Lligadas. Biomechanical validation ofupper-body and lower-body joint movements of kinect motion capture datafor rehabilitation treatments. In 4th International Conference on IntelligentNetworking and Collaborative Systems (INCoS), pages 656–661. IEEE, 2012.

[87] C. Chen, Y. Zhuang, F. Nie, Y. Yang, F. Wu, and J. Xiao. Learning a 3d humanpose distance metric from geometric pose descriptor. IEEE Transactions onVisualization and Computer Graphics, 17(11):1676–1689, 2011.

[88] M. Hussein, M. Torki, M. Gowayyed, and M. El-Saban. Human action recogni-tion using a temporal hierarchy of covariance descriptors on 3d joint locations.In Proceedings of the International Joint Conference on Artificial Intelligence,pages 2466–2472. AAAI Press, 2013.

[89] R. Chaudhry, F. Ofli, G. Kurillo, R. Bajcsy, and R. Vidal. Bio-inspired dynamic3d discriminative skeletal features for human action recognition. In IEEE Con-ference on Computer Vision and Pattern Recognition Workshops, pages 471–478. IEEE, 2013.

[90] E. Ohn-Bar and M. M. Trivedi. Joint angles similarities and HOG2 for ac-tion recognition. In 2013 IEEE Conference on Computer Vision and PatternRecognition Workshops (CVPRW), pages 465–470. IEEE, 2013.

[91] I. Jolliffe. Principal Component Analysis. Wiley Online Library, 2002.

117

[92] J. Wang, Z. Liu, Y. Wu, and J. Yuan. Mining actionlet ensemble for actionrecognition with depth cameras. In IEEE Conference on Computer Vision andPattern Recognition (CVPR), pages 1290–1297. IEEE, 2012.

[93] M. Muller. Information Retrieval for Music and Motion. Springer-Verlag NewYork, 2007.

[94] N. Kumar and R. V. Babu. Human gait recognition using depth camera: Acovariance based approach. In Proceedings of the Eighth Indian Conference onComputer Vision, Graphics and Image Processing, pages 21–26. ACM, 2012.

[95] R. Bellman. A markovian decision process. Journal of Mathematics and Me-chanics, 6, 1957.

[96] R.E. Bellman. Dynamic Programming. Princeton Univeristy Press, Princeton,NJ, 1957.

[97] M. K. Chippa and S. Sastry. Markov decision process formulation of managinghuman weight loss. In IEEE Systems Confererence, 2016.

[98] l.P. Kaelbing, M.L. Littman, and A.R. Cassandra. Planning and acting in par-tially observable stochastic domains. Artificial Intelligence, 101:99–134, 1998.

[99] M.A. Alsheikh, T.H. Dinh, and N. Dusit et.al. Markov decision processes withapplications in wireless sensor networks: A survey. IEEE CommunicationsSurveys & Tutorials, 17(3):37, 2014.

[100] R. Gillani and A. Nasir. Incorporating Artificial Intelligence in Shopping As-sistance Robot using Markov Decision Process. International Conference onIntelligent Systems Engineering, pages 94–99, 2016.

[101] S.Feyzabadi and S.Carpin. Risk-aware path planning using hierarchical con-strained markov decision processes. In IEEE International Conference on Au-tomation Science and Engineering (CASE), pages 297–303, 2014.

[102] A.N. Burnetas and M.N. Katehakis. Optimal adaptive policies for markovdecision processes. Mathematics of Operations Research, 22(1), 1995.

[103] M. Sandri, P. Berchialla, I. Baldi, and D. Gregori et. al. Dynamic BayesianNetworks to predict sequences of organ failures in patients admitted to ICU.Journal of Biomedical Informatics, 48:106–113, 2014.

[104] K.J. Van Arendonk, K.H. Chow, and N.T. James et. al. Choosing the order ofdeceased donor and living donor kidney transplantation in pediatric recipients:a Markov decision process model. Transplantation, 99(2):360–6, 2015.

118

[105] T. Ben-Zvi, T. Chernonog, and T. Avinadav. A two-state partially observableMarkov decision process with three actions. European Journal of OperationalResearch, 254(3):957–967, 2016.

[106] T. M. Hansen, K.P. Edwin, S. Suryanarayanan, and A. A. Maciejewski. APartially Observable Markov Decision Process Approach to Residential HomeEnergy Management. IEEE Transactions on Smart Grid, 3053(July):1–8, 2016.

[107] T. Ayer, O. Alagoz, and N. K. Stout. A POMDP Approach to Personalize Mam-mography Screening Decisions. Operations Research, 60(5):1019–1034, 2012.

[108] S. Temizer and M.J. Kochenderfer. Unmanned aircraft collision avoidance usingpartially observable markov decision processes, 2009.

[109] P. Tseng. H-horizon, stationary markov decision problems in time proportionalto log(h). Operations Research, 9(5):287–297, 1990.

[110] A.M. Farahmand, R. Munos, and C. Szepesvari. Error propagation for approx-imate policy and value iteration. Advances in Neural Information ProcessingSystems, 2010.

[111] M. L. Littman, T. L. Dean, and L. P. Kaelbling. On the complexity of solvingmarkov decision problems. In In Proceedings of the Eleventh Annual Conferenceon Uncertainty in Artificial Intelligence, Montreal, Quebec, Canada,, 1995.

[112] J. MacGlashan. The brown-umbc reinforcement learning and planning (burlap).

[113] J.H. Ahn and J.C. Hornberger. Involving patients in the cadaveric kidneytransplant allocation process: A decision-theoretic perspective. ManagementScience, 42(5):629–641, 1996.

[114] M.F. Drummond, B. OBrien, G.L. Stoddart, and G.W. Torrance. Methods forthe Economic Evaluation of Health Care Programmes. Oxford University Press,1997.

[115] G.W. Torrance. Social preferences for health states: an empirical evaluation ofthree measurement techniques. Socio-Economic Planning Sciences, 10(3):129–136, 1976.

[116] G.W. Torrance, D.H. Feeny, W.J. Furlong, R.D. Barr, Y. Zhang, and Q. Wang.Multiattribute utility function for a comprehensive health status classificationsystem: Health utilities index mark 2. Medical Care, 34(7):702–722, 1996.

[117] P.B. Shull., W. Jirattigalachote, and M.A. Hunt et. al. Quantified self andhuman movement: A review on the clinical impact of wearable sensing andfeedback for gait analysis and intervention. Gait and Posture, 40(1):11–19,May 2014.

119

[118] V.E.D. Pinheiro and H.A. Simon. An operational model of motor skill diagnosis.Journal of Teaching in Physical Education, 11:288–302, 1992.

[119] J. K. Aggarwal and M. S. Ryoo. Human activity analysis: A review. ACMComputing Surveys (CSUR), 43(3):16–58, 2011.

[120] J. K. Aggarwal and L. Xia. Human activity recognition from 3d data: A review.Pattern Recognition Letters, 48:70–80, 2014.

[121] C.H. Chen, J. Favre, G. Kurillo, T.P. Andriachhi, R. Bajcsky, and R. Chel-lappa. Camera networks for healthcare, teleimmersion, and surveillance. IEEEComputer, 47:26–36, 2014.

[122] L. Wenmin, D. Wanchun, Z. Zuojian, and L. Chang. A cloud-based frameworkfor Home-diagnosis service over big medical data. Journal of Systems andSoftware, 102:192–206, 2015.

[123] J. H. M. Bergmann, V. Chandaria, and A. McGregor. Wearable and implantablesensors: The patient’s perspective. Sensors, December 2012.

[124] B. Klika and C. Jordan. High-intensity circuit training using body weight:Maximum results with minimal investment. ACSM Health and Fitness, 17(3):8–13, 2013.

120

GOAL-SEEKING DECISION SUPPORT SYSTEM TO EMPOWER PERSONAL WELLNESS

Documents

Transcript of GOAL-SEEKING DECISION SUPPORT SYSTEM TO EMPOWER PERSONAL WELLNESS