+

Automated Approaches to Health Assessment Christa Simon and Chris Cain

+Aging

n  Estimated number of individuals over the age of 85 expected to triple by 2050 (Vincent & Velkoff, 2010).

n  1 in 3 households is anticipated to have at least 1 family member with cognitive decline within the next decade

n  Significant concern is the projected drop in the labor pool of health workers and caregivers (Comeau, 2005).

n  The resulting prevalence of chronic illnesses and costs to health care are challenges that our society must address.

n  It is becoming increasingly critical to develop innovative and preventive health care methods and to consider additional roles for neuropsychologists in health promotion and assistance.

+Health Assessment

n  Purpose: to establish where on the health continuum the individual is because this guides how to approach and treat the individual

n  Plan of care that identifies the specific needs of the client and how those needs will be addressed by the healthcare system

+ Health Assessment in Older Adults n  Why health assessment of older adults is important

n  Health assessments can help older adults identify several aspects of their health and well-being that influences their daily lives.

n  “Quality of Life” n  Assessing physical functions

n  Functional status (ADLs) n  Nutrition n  Vision n  Hearing

n  Assessing mental functions n  Memory n  Executive functioning

n  Assessing psychological functions n  Depression

n  Social assessment

+Automated Health Assessment in Older Adults

n  Smart Home Monitoring- Cook, 2012 n  Ideal environment for performing automated health assessment

n  Arm Motor Ability Test (AMAT)- Allin and Ecker n  Computer vision techniques to correlate motor statistic of stroke survivor’s

motion obtained from multiple digital cameras with expert functional scores.

n  Smart Phones- Palmes et al., 2010 n  Equipped with accelerometers and gyroscopes to recognize movements and

gestures patterns.

n  RFID tags and shimmer sensors- Kwapisz et al., 2010 n  Tagging objects (medication dispenser, phone etc.) and using the data for

activity recognition.

n  Fall Detection- n  Active: user triggered n  Passive: environment senses fall and alerts others (i.e., wall sensors, pressure

sensitive floors, intelligent video surveillance)

+Smart Home Monitoring

n  Smart Home can be viewed as an environment in which computing and communications technologies employ artificial intelligence techniques to reason about and control our physical home setting.

n  Sensor events are generated while residents perform their normal daily routines.

n  Detecting cognitive impairment in everyday task performance

n  Differences can be automatically detected between cognitively healthy individuals and those with dementia and mild cognitive impairment (MCI)

+Scenario- n  “For example, suppose you are working with an elderly woman

who is living independently. She and her children are beginning to question whether she can continue to live alone. Her children wonder whether their mother is taking her medications regularly and getting enough exercise. Because the children do not live nearby, they only get snapshots of how their mother is doing when they visit or talk with her on the phone. If she lived in a smart environment where her activities were being tracked continuously, this type of information would be readily available. By tracking her normal daily routine, the time it takes her to complete her usual activities, and other everyday patterns, the smart home could assess for possible changes in her functional performance. In addition, if she failed to get out of bed one morning and prepare her breakfast per her normal routine, an alert could be sent indicating that she should be checked on. Using activity-aware prompting, the smart home could also cue her to engage in health-related activities such as exercise or socialization, potentially helping to reduce falls and increase overall well-being.”

-Maureen Schmitter-Edgecombe, Adriana Seelye, and Diane J. Cook “Technologies for Health Assessment, Promotion, and Assistance: Focus on Gerontechnology”

+Everyday Functioning in Older Adults

n  Gain insight about difficulties that effect quality of life

n  Assist individuals in completing daily activities

n  Cooking, managing finances, driving, and other activities of daily living that individuals must complete to live competently and independently

n  Deficits and changes in everyday functioning are considered as precursors to more serious cognitive problems such as dementia and MCI

+Assessment Measures for Everyday Functioning

n  Self-report questionnaires of activities of daily living

n  Informant-report questionnaires

n  Performance-based simulation

n  Problems: n  Self-report and informant-report are subject to reporter bias

n  Performance-based measures collected via simulation measures in a lab or clinical setting may not capture subtle details of activity performance that occur in a home setting

+

n  Primary Goals:

1.  Perform automated assessment of tasks performed in a smart home

2.  Propose a machine learning methodology to automatically quantify the QUALITY of the performance of an activity with respect to how other individuals perform the same activity.

3.  Implement the approach to activity assessment in the Smart Home testbed and correlate automated scores with measurements derived from direct observation of participant performances

4.  Analyze correlation between activity quality and health diagnosis of individuals

+ Experimental Setup n  Participants (N=179)

n  Cognitively healthy participants (N=145)

n  Mild cognitive impairment (N= 32)

n  Dementia (N= 2)

n  Female= 141, Male= 38

n  YoungYoung under 45 YO (N= 37)

n  MiddleAge 45-59 (N= 27)

n  YoungOld 60-74 YO(N= 84)

n  OldOld 75+ YO (N= 31)

n  Feature Extraction- DOT Feature Set

Feature Set Feature Type

DOT features Duration, sensor counts, sensor events, activity completeness

Interruption features Number of activity interruptions

Sequencing features Sequence vector

Parallelism feature Pindex

n  Testbed n  Motion sensors n  Sensor events

n  Day Out Task n  8 Subtasks

1.  Magazine 2.  Heating pad 3.  Medication 4.  Bus map 5.  Change 6.  Recipe 7.  Picnic basket 8.  Exit

+Evaluation n  To evaluate the ability to automate assessment of cognitive

health, comparing diagnoses generated from machine learning-based Smart Home algorithms with diagnoses based on clinical sets

n  Performed 4 experiments to evaluate Smart Home-based task quality assessment algorithms n  Measured the correlation between subsets of Smart Home sensor

features and direct observation scores n  Measured the correlation between the entire set of sensor

features and direct observation scores n  Assessed how well a support vector machine learning algorithm

correctly classifies task quality, using the direct observation scores as ground truth labels

n  Determined how well the unsupervised learning-derived score correlates with direct observation scores

+n  Correlations between most of the feature subsets and direct

observation (accuracy & sequencing scores) are statistically significant n  Accuracy score – higher correlation with feature subsets (i.e.,

DOT features, interruption features, sequencing features, & parallelism feature)

n  Sequencing score

n  This is because the accuracy score takes into account the mistakes that an individual makes in a subtask while the sequencing score only considers how a participant initiated the activity.

n  Correlation between sensor based feature sets and direct observation scores were high, indicating that the sensor based data provides useful information to assess task quality.

n  Therefore, sensor data can provide us with similar information as human trained clinicians can.

Results

+Observations/Conclusions

n  Demonstrated that machine-learning algorithms can find meaning in sensor based data in order to perform automated assessment of task quality.

n  Preliminary results indicate that Smart Homes and ubiquitous computing technologies can be useful for monitoring complex everyday functions and to automate assessment of daily activities.

n  Valuable for monitoring the well-being of individuals in their own environments

+ Future Work n  In the future, smart home technologies could provide neuropsychologists and

other health care professionals opportunities to see variability and trends or trajectories in functional performance and identify how daily activities impact traditional measures. n  Ex: Smart Home technologies could capture acute health care changes that

require immediate intervention. n  Data collected through these technologies, combined with symptoms reported

by patients in the clinician’s office, could increase health and well-being through early detection and treatment.

n  Consider using a person as their own performance baseline and automate longitudinal assessment of well being and change in functional independence by analyzing sensor data collected in everyday environments over time. n  Horizon House

n  Alternative ways of detecting meaningful change for prevention studies n  Jeff Kaye – OHSU: added a key portal to assessing the activity and cognitive

health of seniors and their ability to use and interact with a personal computer n  Basic Level (manual motor activity) n  Higher Order (play games i.e., FreeCell) n  Psychometric tests

+

assessments of both functional and cognitive changes in thesubject’s natural environment, their home. Home-based assess-ment affords the opportunity to not only observe change in theperson’s usual environment but also to more frequently, and insome cases continuously, monitor a subject for salient change.A number of converging computer and sensor technologydevelopments provide the opportunity to assess both functionaland� cognitive� status� in� an� automated� manner� [18,19].� Forexample, more accurate assessments of medication manage-ment are facilitated by using instrumented pill containers thatrecord�when�medication� is� removed� [20–22].�More� frequentcognitive assessment can be performed with computer-basedtests, and recent advances in automated speech recognition(ASR)� technology�[23]�allow�for�accurate�automated�scoringof oral responses to constrained verbal tests, removing the needfor keyboarding, which can challenge some older subjects.Finally, indirect measures such as activity pattern monitoringor computer use that might be conducted “in the background”might afford sensitive early measures of decline even withoutdirect�measurement� [24–27].�The� integration�of� these�home-based assessment technologies originated in the concept of“smart�homes”�[28]�and�telemedicine�more�than�20�years�ago.

They have now converged and matured to be capable of widerscale deployment and integration into common clinical trialpractice.

An example of a scalable home-based assessment plat-form and assessment protocol currently developed and de-ployed by the Oregon Center for Aging and Technology atOregon� Health� &� Science� University� [18,24]� is� shown� inFigure� 2.� In� this� system,� continuous� activity� data� are� col-lected unobtrusively in the home by using passive infraredpyroelectric motion sensors strategically placed in everyroom at locations oriented to pick up the participant’s move-ments restricted to that room. In addition, magnetic contactsensors placed on each door provide a means to track theflow of visitors and absences from the home. An estimate ofwalking speed is obtained by using motion sensors with arestricted field of view installed along a hallway, thus firingonly when a person passes directly in front of them. Todetermine who is moving in the home, subjects wear radio-frequency identification tags whose relative signal strengthshelp infer location.

In addition to this simple home sensor net, a key portal toassessing the activity and cognitive health of seniors is their

SecureInternet

A

B

C D E

Fig. 2. In-home assessment system for conducting continuous monitoring in clinical trials. The system consists of a network of wireless pyroelectric X10motion and contact sensors (A and D), which send radiofrequency signals to a transceiver attached to the serial port of a research computer. Strategicallyplaced restricted field sensors (B) assess walking speed. A home computer (C) provides additional assessment input (typing speed, mouse movements, higherorder cognitive assessments). This forms a portal for assessing further motor activity. Medication-taking ability (an instrumental ADL) is assessed by usingthe MedTraker pillbox device (E), instrumented to automatically record the time of day each box door is opened.

S63J. Kaye / Alzheimer’s & Dementia 4 (2008) S60–S66

Home-based�technologies:�A�new�paradigm�for�conducting�dementiaprevention�trials

Jeffrey�Kaye*NIA�–�Layton�Aging�&�Alzheimer’s�Disease�Center�and�ORCATECH,�the�Oregon�Center�for�Aging�&�Technology,�Oregon�Health�&�Science�University,

Portland,�OR,�USA

Abstract� The� approach� toward� treatment� of� Alzheimer’s� disease� has� evolved� out� of� a� large� body� ofresearch,�resulting�in�a�common�trial�paradigm�directed�toward�establishing�efficacy�and�safety�ofputative� treatments�for�dementia.�To�achieve� the�ultimate�goal�of�preventing�dementia,� the�futurewill� build� on� this� model� but� will� need� to� incorporate� new� paradigms� to� more� efficiently� detectmeaningful� change� over� time.� Among� new� innovations� needed� is� the� capability� for� assessingtrajectories�of�change�over�many�years.�Intrinsic�to�this�success�is�the�ability�to�capture�increasingvariability�as�well�as�subject�heterogeneity�inherent�in�longitudinal�aging�cohorts.�New�methods�ofubiquitous�embedded�home�sensing�and�computing�afford�the�opportunity�to�assess�volunteers�notonly� during� long� periods� of� time� but� with� greater� frequency� and� convenience.� Clinical� studiesidentifying�the�optimal�ways�of�using�these�methods�are�needed.©�2008�The�Alzheimer’s�Association.�All�rights�reserved.

Keywords:� Clinical� trials;� Dementia;� Mild� cognitive� impairment� (MCI);� Technology;� Sensors;� Cognitive� assessment;Activity�assessment

1.�Current�paradigms�for�Alzheimer’s�diseasetreatment�leading�to�preventions

Effective�treatments�are�universally�perceived�as�the�pri-mary,�bottom-line�goal�of�the�international�effort�engaged�inAlzheimer’s�disease�(AD)�research.�Beyond�arresting�man-ifest� dementia� in� its� tracks� is� the� hope� that� one� mightactually�prevent�dementia� from�occurring�at�all.�Many�en-couraging�developments�have�emerged�during�the�past�de-cade� suggesting� that� these� goals� might� be� rationally� ap-proached� and� achieved.� In� the� process� of� striving� for� thisvision,� especially� during� the� last� decade,� remarkable� ad-vances� have� been� made.� Along� the� way� much� of� this� re-search� progress� has� been� based� on� several� fundamentalassumptions.� These� are� widely� accepted� by� many� thoughtleaders� in� the� field� [1].� These� principles� include� that� (1)there�is�a�standard�characteristic�AD�clinical�phenotype;�(2)underlying� that� phenotype� is� a� common� neuropathologicsubstrate� consisting� in� large� measure� of� the� deposition� of

amyloid-forming�proteins�and�the�formation�of�neurofibril-lary� (or� tau� bearing)� tangles,� which� lead� to� progressiveregionally� specific� neuron� loss;� (3)� the� substrate� developsslowly�many�years�before�clinical� symptoms�develop;�and(4)�the�emergence�of�that�phenotype�is�dependent�on�a�suiteof� susceptibility� genes� that� are� modified� by� the� environ-ments�and�lifestyles�one�is�exposed�to�during�a�lifetime.

The�remarkable�progress�that�has�been�made�in�closingin�on�achieving� the�goal�of�effective� therapies�and�evenpreventatives� has� relied� on� this� chain� of� knowledge.� Ithas�been� the� foundation� for� the�primary�guideposts� thathave� played� off� one� another� during� the� past� decade� inmoving� many� drugs� into� human� clinical� trials,� namelytaking�observations�on�various�exposures�or�clinical�fea-tures� experienced� by� people� or� observed� in� populationspresumed to have AD or escaped AD and then applyingthose substances in the laboratory to cellular and trans-genic rodent models of the presumptive human substrate.If along with safety, a signal of efficacy is detected, thisprovides a rationale for moving these treatments into thelong pipeline of progressively larger human safety andefficacy trials.

*Corresponding author. Tel.: 503-494-6695; fax: 503-494-7499.E-mail�address:�kaye@ohsu.edu

Alzheimer’s & Dementia 4 (2008) S60–S66

1552-5260/08/$ – see front matter © 2008 The Alzheimer’s Association. All rights reserved.doi:10.1016/j.jalz.2007.10.003

+Limitations

n  Assessment technique relies on participants completing scripted activities in a single smart home setting

n  Currently use direct observation scores and clinician-based cognitive diagnosis as ground truth label to train our learning models

n  Some of the features that are derived rely on human annotation of sensor data, which is time-consuming and can introduce errors

+

n  Interactive Activity Recognition and Prompting

n  Important for mental health: n  Adhering to schedule n  Completing ADLs as independently and frequently as possible n  Minimize intrusions into older adults' lives

n  Basic idea: Automatically detect user and world state through sensor readings.  n  Prompt user to begin, resume, or end tasks.  n  Query user if algorithm unsure.  n  Adapt activity recognition to user feedback.

n  Context-aware prompting avoids unnecessary prompting, minimizes cost of interruptions.

Journal of Ambient Intelligence and Smart Environments 0 (0) 1 1IOS Press

Interactive Activity Recognition andPrompting to Assist People with CognitiveDisabilitiesYi Chu a, Young Chol Song a, Richard Levinson b and Henry Kautz a

a Department of Computer Science, University of Rochester, Rochester, NY 14627, USb Attention Control Systems, 650 Castro Street, PMB 120-197, Mountain View, CA 94041, US,www.brainaid.com

Abstract. This paper presents a model of interactive activity recognition and prompting for use in an assistive system for personswith cognitive disabilities. The system can determine the user’s state by interpreting sensor data and/or by explicitly queryingthe user, and can prompt the user to begin, resume, or end tasks. The objective of the system is to help the user maintain adaily schedule of activities while minimizing interruptions from questions or prompts. The model is built upon an option-basedhierarchical POMDP. Options can be programmed and customized to specify complex routines for prompting or questioning.

The paper proposes a heuristic approach to solving the POMDP based on a dual control algorithm using selective-inquirythat can appeal for help from the user explicitly when the sensor data is ambiguous. The dual control algorithm is workingeffectively in the unified control model which features the adaptive option and robust state estimation. Simulation results showthat the unified dual control model achieves the best performance and efficiency comparing with various alternatives. To furtherdemonstrate the system’s performance, lab experiments have been carried out with volunteer actors performing a series ofcarefully designed scenarios with different kinds of interruption cases. The results show that the system is able to successfullyguide the agent through the sample schedule by delivering correct prompts while efficiently dealing with ambiguous situations.

Keywords: assistive system, context-aware prompting, activity recognition, MDP, POMDP, ambient intelligence

1. Introduction

The fast growing population of the ageing soci-ety will result in a dramtic increase in the number ofpeople diagnosed with cognitive disabilities (such asAlzheimer’s disease or other forms of dementia). Peo-ple with cognitive disabilities suffer from memory lossand executive function impairment [19,12,33] that pre-vent them from organizing, managing or carrying outeveryday tasks independently. Commonly, they are ex-periencing failures such as failing to initiate a task,forgetting an unfinished task after interruptions, per-forming the tasks incorrectly and so on. To compensatefor deficits in cognitive function, human assistance isneeded to provide regular prompts that help the patientthrough the activities of daily living. The constant de-pendence and pressure on the caregivers have a nega-

tive impact on both the patient and caregiver that couldlead to diminished quality of life, increased level ofanxiety, poor self-esteem, and social isolation [2].

To support independent living and reduce the costof heath care, researchers have developed various tech-nologies and computing systems that can automate theprompting behavior and alleviate the burden on care-givers. This could be electronic devices that providetimely prompts and reminders to support schedule ad-herence and time management [13,24,3]. While to-day’s commercially-available prompting tools are es-sentially enhanced calendars, researchers have recog-nized the need for context-aware systems that infer theuser’s state and provide the right kind of help at theright time [22,25,15,19,17]. For instance, prompting tostart a task that the person has already started could beconfusing. Similarly, a prompt for taking medication

1876-1364/0-1900/$17.00 c� 0 – IOS Press and the authors. All rights reserved

+Example

n  Prior data: user remembers to start breakfast often.  Thus system waits to generate prompt.

n  Upon sensing start of breakfast, system cancels all prompts related to starting breakfast

n  Incoming phone call distracts user

n  After call, user forgets breakfast and begins a new activity

n  System identifies suspension of task and generates a prompt due to distraction, scheduled end time for breakfast occurs.  System unsure of status and queries user.  User responds "finished eating, cleaned up". System terminates end breakfast prompting to avoid confusion.

+Temporal Planning

n  Solves set of temporal constraints over the time windows of all necessary tasks for the day (Bellman Ford algorithm)

n  Entered by caregiver daily

n  Defined by start and end times

n  Start and end points bound by earliest and latest times the task can start/end.

n  Can define constraints such as "task A starts after task B finishes" and "task B lasts less than 15 minutes”

n  Can also handle user preferences via slack variables

+ Options n  Each task has own Markov Decision Process (MDP) n  Each task has a set of options, each option runs over state space of

associated task n  Task status determines which option(s) is/are enabled n  Possible statuses:

n  Inactive = Initial state.  Options = pass deadline, begin time arrives, start.

n  Ready = Start window has begun, task not initiated yet.  Options = start, pass deadline.

n  Underway = Task initiated.  A task is active when either ready or underway.  Options = pass deadline, execute incorrectly, finish.

n  Suspended = User has stopped task in middle.  Options = pass deadline, resume.

n  Error = User not completing task correctly.  Options = recover, pass deadline

n  Failed = Impossible for task to be completed within time window.  Final state.

n  Completed = Task successfully completed.  Final state. n  Q Learning is done for each MDP individually n  Utility of an option reflects the effectiveness of that prompting strategy

+Adaptive Options

n  Motivation: Optimal prompting strategy does not prompt user before they have had chance to self-initiate.  Not too late, risking task failure.  Learning to choose between fixed options with different timings requires too much training time in practice.  Bad prompts from exploring can be annoying to user.

n  Adaptive options: Allow algorithm to adapt to user

n  Initiative: How soon user will initiate task with no prompt

n  Responsiveness: How long user will take to respond to a prompt

n  Model uses these to avoid unnecessary prompts and to ensure task occurs on time

n  Note: Important to consider prompts, as user did not self initiate before that time

+Uncertainty n  Current activity determined by Hidden Markov Model

n  States are possible activities associated with tasks + "other activity" state.

n  Observations are sensor streams.  Computes probability distribution over the set of activities.

n  Solving full Markov Decision Process computationally infeasible.  Need a simpler control mechanism.

n  When uncertainty small, state with highest probability is the true state.  When uncertainty large, query user.

n  Take into account the value of an inquiry.  If all possible states and uncertainty lead to the same action, no need to inquire.  Inquiry only needed when possible states result in selection of different actions.

+Results

n  Tested 2 students with no prior knowledge of the system.  n  Had them start breakfast, finish breakfast, start medicine, stop

medicine as well as start breakfast, start medicine, stop medicine, stop breakfast.

n  System not perfect, had one instance of thinking breakfast had been suspended when it had actually been finished.  Only the one incorrect prompt.  There were other errors, but system recovered before prompts generated.

+Limitations

n  Still early in development

n  Not thoroughly tested in practice, especially among older adults.  Will older adults comply with prompts and resume activities?

n  Only been tested with 2 activities, and suspending one to do another.  Real life can be much more complicated.

n  Tests in practice have classification mistakes, creating the possibility of incorrect prompts, which can lead to elevated stress levels.

+Further Directions

n  Longitudinal or case studies with this system could look at the number of prompts generated over time, which could measure cognitive decline and possibly differentiate normal aging from dementia.  This would be especially helpful if combined with a measure for how well they are performing ADLs.

n  Apply different prompting techniques (i.e visual, reward-based) to increase compliance rates among certain individuals.  Higher compliance means higher ADL completion rate, which increases mental and possibly physical health.

n  Not responding to prompts, spending too long in "other activity" state, or specific sensor readings (i.e motion sensors detecting a fall) could indicate something is wrong with user, should alert caregiver.  

+Future Direction of Automated Health Assessments in Older Adults

n  By combining the automated health assessment in the Smart Home environment with prompting techniques, a more in depth picture of what the individual is experiencing can be better detected. n  Smart Home is a way to MONITOR the adults, but there isn’t an

automated way to get the individual back on task and also no way to intermix tasks

n  Smart Home technology could then alert their caregiver to go and assist the individual

n  But with the addition of prompting technology, the individual could be alerted, possibly allowing them to correct their own actions or schedule, easing the burden on caregivers, and promoting independence.

n  Compare activity recognition techniques between HMM and machine learning.  Does a combination of the two lead to increased classification accuracy?

+Thank You.

n Questions?