The Aware Home: A living laboratory for technologies for successful aging

Gregory D. Abowd, Aaron F. Bobick, Irfan A. Essa, Elizabeth D. Mynatt, Wendy A. Rogers

GVU Center Georgia Institute of TechnologyAtlanta, GA 30332-0280, USA

Abstract

We describe our ongoing research in the area of de-veloping and testing technologies for successful ag-ing. Sensing and perception technologies can enablea home environment to be aware of the whereaboutsand activities of its occupants. Motivated by the de-sire to use such an awareness to help maintain inde-pendence and quality of life for an aging population,we describe the technological, design and engineeringresearch challenges inherent in this problem domain.Our work is situated in the Georgia Tech BroadbandInstitute’s Residential Laboratory, a unique living lab-oratory for this exploration of ubiquitous computing ina domestic setting.

IntroductionHow can your house help you and your family if it ismade aware of your whereabouts and activities? Toanswer this question, and other questions related tothe implications of ubiquitous computing technologiesin the home, Georgia Tech opened the doors on May1, 2000 a new research facility, the Broadband In-stitute’s Residential Laboratory (see Figure 1), This5,040 square foot laboratory — a two story house con-taining two identical, two bedroom apartments, base-ment and attic — has been specially designed and con-structed to facilitate advanced research and develop-ment as well as extended human subjects experimenta-tion.

We have initiated a long-term research program, theAware Home Research Initiaive (AHRI), to push on thespecific multidisciplinary challenges of building tech-nology and appropriate services that leverage aware-ness of human activity in the domestic setting. Inthis paper, we will address the motivation and researchchallenges of this research initiative. We have cre-ated a unique living laboratory for investigating future

Copyright c� 2002, American Association for Artificial In-telligence (www.aaai.org). All rights reserved.

ubiquitous computing technologies in a domestic set-ting. There are significant technological and engineer-ing challenges to overcome in order to create an en-vironment that can perceive the activities of its occu-pants. Equally significant is the challenge to invent asatisfactory human experience amidst the awareness.We will explain our human-centered motivation forpromoting awareness in a domestic setting and thenaddress some of the design, technology and softwareengineering challenges we are pursuing.

Motivation: Maintaining quality of life for anaging populationAmerica’s population is aging. The U.S. Census Bu-reau reported nearly 44 million adults over age 60 in1996, and the projected number for 2025 is approxi-mately 82 million (over 20% of the total population).Baby boomers approaching late middle age ask, ”Howdoes one care for a population that lives many yearslonger than any of the preceding generations?” Fur-thermore, the costs of traditional assistive care willcontinue to be out of reach for the majority of Amer-icans, costing on average $60,000 per year, and of-ten requiring a substantial entry fee (Greenwald 1999).These costs are more prohibitive as the demographicsin the U.S. continue to change. Specifically, the ratioof wage earners to senior adults will shift from 4-1 to2-1 in the next 25 years (Vanderheiden 2000).

For economic reasons alone, increasing the lengthof time that individuals can defer assistive care is valu-able. For reasons beyond economic ones, a primarygoal of many older individuals is to maintain an in-dependent lifestyle. Thus, many older adults live inprivate homes, typically either alone or with family.Our own ethnographic studies of local Atlanta assis-tive care centers focused on understanding why peoplechose to leave their homes for an institutional care set-ting. We were impressed by the significant and some-times dominant role of adult children and their need forday-to-day assurance of their parent’s well being. Weobserved the growing emphasis on cognitive declines.

From: AAAI Technical Report WS-02-02. Compilation copyright © 2002, AAAI (www.aaai.org). All rights reserved.

We also noted the negative effects of institutional life,for example, decreased motivation for self-care, andincreased isolation and depression.

Based upon these observations, we believe thatthe development and careful placement of appropri-ate technological support can empower older adults tocontinue living in their own homes longer. We specif-ically focus on the effects of declining cognitive abil-ities on independent living, and the role informationtechnology can play in augmenting those capabilities.We have identified two areas that are key to sustainingquality of life for an independent senior adult:

Maintaining a Familial Vigilance A critical func-tion of an assistive care setting is to provide a safe,healthy environment. This clearly requires detectingcrises or emergencies. A more demanding aspect ofproviding such an environment, however, is the moregeneral monitoring of the occupants’ situation and thenoting of significant changes in the manner in whichthey behave or interact. We refer to such extendedmonitoring over time as trending. Examples includenoticing reduced mobility or far less time engaged incognitively stimulating activities. Furthermore, the in-formation about an older individual should be sharedin a privacy respecting manner, sharing appropriatelyfiltered information with concerned family membersand trusted caregivers, thus promoting a peace of mindthat comes with mutual awareness and day-to-day as-surance.

Supporting Daily Routines: Declines in cognitiveand sensory capabilities manifest themselves as diffi-culties in performing many daily tasks, such as tak-ing medication, preparing food, and operating house-hold appliances. An aware environment should helpoffset typical cognitive impairments, such as problemswith retrospective memory — the episodic remember-ing of what the occupant was doing recently, say, be-fore being interrupted. Providing relevant informationat the appropriate time (e.g., which medications shouldbe taken now that the occupant has awakened and isheaded to the bathroom), reduces the cognitive burdenof doing the everyday activities essential to indepen-dent living.

Outlining of the research questions

Though human needs motivate the research in theAware Home, two principal technical intellectual chal-lenges are being explored simultaneously:

Human-Computer Interaction The design of de-vices, services and interfaces to facilitate novelhuman-human and human-environment interactions

(a) (b)

Figure 1: The Georgia Tech Broadband Institute Res-idential Laboratory, home for the Aware Home Re-search Initiative. (a) The floor plan for one of two iden-tical living floors. (b) The exterior view of the property.

in light of increased awareness, with particular fo-cus on interactions for an aging population and theirdistributed families.

Computational Perception The development oftechniques with a variety of sensing devices thatlead to an environmental awareness of humanactivity in the home.

Designing novel interactions in the AwareHome

Research in Human-Computer Interaction encom-passes examining the usefulness and usability of fu-ture technologies. Both of these goals are particularlychallenging in home environments. In the context ofthis proposal, usefulness focuses on supporting inde-pendence and quality of life. Much of our researchis guided by understanding typical cognitive declinesthat accompany aging and their implication for inde-pendent life. Moreover communication technologiescan improve the quality of life by increasing social net-works, daily interaction and feelings of connectedness.To support independence, usability focuses on provid-ing support while not undermining the person’s abilityto care for themselves. For seniors, it is important todesign interfaces that take into account many of thetypical declines in perception and cognition.

Supporting familial vigilance andcommunicationIn addition to supporting the needs of older adults, wehave discovered that it is critical to support communi-cation and awareness between extended family mem-bers (Mynatt, Essa, & Rogers 2000). An aging adult’sdesire to remain in the familiar setting of the familyhome frequently must be balanced with their extendedfamily’s desire to keep them safe. Clearly this balancebecomes more precarious as age increases. An agingcouple can support one another, but if one becomes in-capacitated, can the other support himself? No longer

(a) (b) (c)

Figure 2: The Digital Family Portrait. (a) illustrates 28 days of activity information about the woman pictured. (b)shows detailed information about one day including the weather, indoor and outdoor temperature, and the number ofroom-to-room transitions in 15 minute increments. (c) A surrogate memory display. Snapshots of six recent actionsare shown in order (upper left to lower right) for a cooking activity.

can family members have the peace of mind that de-rives from knowing that one aging parent can supportthe other. Additionally geographic distance betweenextended family members exacerbates the problem bydenying the casual daily contact that naturally occurswhen families are co- located. Providing support forlight-weight communication and remaining aware of adistant family members’ day-to- day activities, whatcould be called surrogate social support, can promotethe peace of mind necessary for those senior familymembers to age in place.

Designing a Digital Family Portrait: Our digitalfamily portrait addresses the “peace of mind” deficitthat often compels adult children to move their agingparent to an institutional care setting (Mynatt et al.2001). Our goal is to support awareness of the long-term health, activity, and social well-being of senioradults living by themselves, answering questions suchas, “Has she been eating enough?” and, “Is he active orsedentary?” Like a traditional portrait, it is designed tobe hung on the wall or propped on a mantle, blendingwith household decorations. In our interface, we rep-resent the well-being of family members with iconicimagery, based on sensing data from their home. Fromgeneral measurements of activity, to indications of theweather, the portrait attempts to capture the observa-tions that would naturally occur to someone living inthe same home or next door. Most awareness inter-faces only provide a snapshot of the present. Sincemany questions about an elderly parent refer to trendsover time, such as, “Is she becoming more socially iso-lated?,” we provide representations of the past, as wellas the present.

Our current design of the digital family portrait is

shown in Figure 2b. Each icon represents a day; thecurrent day is white with time moving clockwise. Inthis portrait the size of the butterfly indicates how ac-tive the person has been in his home. By touching thebutterfly, the viewer gets information about that day. Inthis example, activity is based on room to room move-ments in 15 minute increments. A graph at the bottomof the screen indicates room transitions throughout theday. Information about current weather and tempera-ture conditions at the remote location are also included.

Our initial research indicates the feasibility of cre-ating a visual representation of a senior adult’s well-being and sharing that information with select familymembers in their home (Mynatt et al. 2001).

Providing serendipitous information in support ofdaily activities: An oft-cited frustration of daily lifeis not having the information you need when you needit. However, as people age, this missing information isless likely to be a current stock price, and more oftenone of the details in a common routine. For example, asretrospective memory capabilities decline, a commonmemory lapse is failing to recall recent events in themidst of an interrupted task (Rogers et al. 1998). Dif-ficulties in successfully completing household tasks,such as preparing a meal, can undermine a person’sself-confidence, and hence their ability to live inde-pendently. Likewise, both cognitive and perceptual de-clines may lead to difficulties in recognizing changesin the physical environment, precipitating a fall or theinability to recover a moved object (Smith 1990). Inthis research, we investigate the design of interfacesthat provide a constant, but unobtrusive, flow of infor-mation that may be used serendipitously in the midstof common daily activities.

Surrogate Memory Systems: In our What Was ICooking? interface, we are investigating how sens-ing and capture capabilities can support recovery fromcommon memory lapses that occur when a person isdistracted while cooking. As cooks perform actions inour kitchen, sensors register these events and triggerthe grabbing of snapshots from a continuously runningvideo stream. These snapshots are currently arrangedusing a cartoon panel metaphor; six to eight imagesin two rows (see Figure 2a). Questions that our inter-face may help people answer are, “Did I add the bakingpowder?” and, “When did I put the roast in the oven?”Our goal is for users, by glancing at our visual dis-plays, to ascertain the answers to those questions withminimal effort. In order to further design and evaluatethis interface we are conduct a series of Wizard of Ozempirical studies.

One common memory lapse is referred to as “lossof activation.” Specifically, a person will initiate a taskin a certain context (e.g. glancing at a calendar andremembering to pay a bill), and then move to a newcontext and forget the remainder of the task (e.g. leftstanding in the study with no memory of why he wentthere). In these memory lapses, the “interruption” isself-initiated as the person shifts context. A recoverytechnique is for a person to retrace his footsteps un-til he recovers the original context and catalyst for thetask. Similar to the What Was I Cooking? interface, aroom-by-room record can aid in bringing that contextwith the person as they change rooms.

Making the environment aware of humanactivity

The novel interfaces just described demand a range ofperceptual capabilities to uncover the context of every-day life in the home. The computational perceptionresearch focuses both on providing base-line trackingperformance (who is where and when) and on develop-ing fundamental new methods for determining qualita-tive behavior (what).

Location and orientation: A significant number ofpotential applications require knowledge of the loca-tion of the occupant or occupants at all times. One ofthe major categories in the ADL description of routinesis that of “ambulation” — a measure of a person’s abil-ity to get around in their environment. A rudimentaryassessment of a persons mobility can be established bymerely knowing the location of that individual at all (ormost) times.

As an initial attempt to provide location informationquickly in the Aware Home, we have custom-designedan RF ID system that uses antennas in the form offloor mats distributed throughout the house. Individ-uals wearing passive RF ID tags below the knee can

now be tracked by simply recording the traversal overthe strategically placed mats (see Figure 3). This sys-tem works now and can provide room-level occupancyinformation of known individuals in the house. Whatwe will describe now are more sophisticated methodsto obtain location and orientation information through-out the home.

We have installed an unobtrusive camera grid on thefirst floor in the Aware Home. These optical sensorsare installed in the ceiling tiles at a height of 3 meters,with a FOV of approximately 120 degrees; the cam-era and a sample image are seen in Figure 4. Thesecameras are calibrated with respect to the location inthe house. We have also developed a client/server“blob” management system where sensor clients per-form background subtraction on a per camera basis andreport detailed information of each significant fore-ground connected component to the server process.The server process maintains a spatially and tempo-rally addressable database of these blobs that consumerclients can access. We have already implemented somesimple gesture recognition clients to act as a triggersfrom a person to the environment to initiate some fur-ther interaction, such as voice recognition.

We are implementing a “track-stitching” algorithm(similar to our real time closed-world tracking ap-proach in (Intille, Davis, & Bobick 1997)) that willgive temporally-extended tracking information per per-son. One of the benefits of conducting this research inthe situated context of the home is that the interface ap-plications establish performance criteria. For example,if there are two people in the house and they sit to-gether on a couch for a period of time, the system mayhave trouble determining who is who, especially froman overhead view. However, as long as once the twopeople separate the system can regain identity, then theconfusion during the couch interval is unlikely to af-fect performance in tasks such as assessing mobility.Furthermore, since assessment does not require causalprocessing, it is possible to use future information todisambiguate uncertainty.

Even given accurate tracking information there iswork necessary to correlate properties computablefrom time-stamped tracking data with ADL assess-ments. We expect that the tracking capability willprovide the earliest technology-based collaboration be-tween technology developers and those researchers fo-cused on assessing the quality of life of an aging pop-ulation. Once tracking is in place, we will perform ini-tial experiments in which independent observers pro-vide Ambulation ADL labelings about subjects that aretracked by the vision system. These labelings will al-low us to apply standard pattern recognition techniquesto the properties of the trajectories from which we needto generate such assessments. Furthermore, any such

(a) (b) (c)

Figure 3: The RF ID room location system in the Aware Home. (a) The RF ID antenna are hidden under doormats,as shown, or under carpets within the home. (b) A view of the hand-crafted RF ID antenna. (c) A floorplan indicatingpositioning of antenna/mats throughout one floor of the Aware Home

automatic labeling, regardless of its correlation withhuman assessment, will be able to provide trend analy-sis of a given occupant over extended periods of time.

There is also fundamental work to be done in de-termining the orientation and/or pose of a person.Such a description is necessary for, say, the context-establishing capture required by the “room-to-room”reminder system discussed above. To collect an imagethat will serve to remind an occupant what they wereintending to do when they entered a particular room,we will need to capture an image in the previous roomthat shows either what they were doing or at least towhat they were attending. While the design researchdetermines the most appropriate image to display, theperception research must be able to determine the rel-evant parameters.

We are pursuing two separate approaches topose/orientation estimation. The first method is strictlyappearance-based. The fundamental idea is to add tem-poral continuity to a silhouette-driven, body-part label-ing method. Our second approach is to take advantageof the multiple oblique views available in each roomof the house. We make use of an image-based visualhull (Matusik et al. 2000) to construct the approxi-mate three-dimensional volume of the person from aset of calibrated cameras or a calibrated environment.For any such volume there is a limited range of bodyposes consistent with the approximate model. We arealso pursuing an approach to detect features on the vi-sual hull that can aid in tracking of limbs over time torefine the skeletal shape, with its inherent articulatoryconstraints.

Local behavior recognition: Though much contex-tually relevant information can be gleaned from the ge-ometric information of location and pose, it is clearthat deeper inferences can be drawn if some informa-

tion is provided about which activity is taking place.Consider a simple example — the location and track-ing system can determine the occupant is sitting onthe couch. But, there is a significant difference froma cognitive engagement perspective between sitting ona couch watching TV (typically cognitively passive) orreading a newspaper (more likely to be cognitively en-gaging). Determining in which activity the resident isengaged increases the sophistication and efficacy of thefamilial vigilance.

The reading scenario is an example of simple localbehavior recognition. By local we mean that the be-havior itself can be defined and therefore recognizedin terms of elements that are local in terms of time andspace. For example, the presence of the newspaper,the location on the couch, and a repeated low duty-cycle lateral motion is probably sufficient to recognizenewspaper reading. There is no extended interactionbetween the occupant and the multiple objects.

Most previous work on human activity recognition,especially at the level of say gesture or simple humanmotions, has focused on this type of problem. A com-mon element of these approaches is that are probabilis-tic, assigning a likelihood that an observed sequenceof features is generated by a particular behavior. In ourwork here we need to include contextual information inthe local behavior recognition methods. For example,simple attributes such as location or time of day canbe exploited to yield much more robust and effectiverecognition procedures.

A more subtle form of contextual influence is used todecide between competing behaviors. Again, supposethe person is sitting on the couch. The system maybeuncertain whether the newspaper is present, and fur-thermore the motion observed could be page turning,or channel surfing with a remote control. The simplecontextual fact that the television is not on is enough to

Fixed Ceiling Cameras

PTZ Camera

(a) (b)

Figure 4: (a) Living room in the house, showing several small cameras installed in the ceiling tiles. Also shown is apan-tilt-zoom (PTZ) camera. (b) View from the ceiling cameras in the kitchen, with segmented person.

reduce the prior probability of channel surfing to zero.Thus, the context about the television affects the sys-tem’s willingness to believe that the occupant is read-ing. This is an example of the conditional dependen-cies that arise in probabilistic reasoning.

Extended behavior recognition: The final and mostambitious effort in the computational perception areais the recognition of extended behavior. By extendedwe mean either in time or in time and space. For ex-ample, the act of preparing a meal is not restricted toa narrow location, and is best determined by the inter-action between the occupant and various other objectsand locations in the scene. To recognize such activ-ities requires a representation that makes explicit thetemporal and spatial relationships between componentelements. For example, the moving of objects fromrefrigerator to counter, counter to table, table to dish-washer, with an indeterminate number of stops back tothe refrigerator is highly indicative of someone prepar-ing a meal and then eating it.

To perform the recognition of these types of ac-tivities, we apply two related yet structurally distinctmethods. The first of these is a structural analysisapproach that uses a stochastic context-free grammar(SCFG) to analyze incoming video frames. The attrac-tion of such a method is the opportunity to provide apriori grammatical definitions of an activity. For exam-ple, one can define meal preparation as requiring activ-ity near a pantry or refrigerator before sitting down atthe table with a plate of food. The probabilistic natureof SCFGs permits robust performance in the presenceof uncertain low-level feature detectors. It is an openquestion as to whether we can design low-level detec-tors of sufficient power to support the type of grammat-ical reasoning required. Determining what the appro-priate features or primitives are is where much of ourinitial effort would be focused.

Handling extremely noisy detectors is the goal of analternative approach. This method employs belief net-

works to probabilistically weigh perceptual evidence.The insight here is that extended behaviors can be rec-ognized by observing relatively well structured localevents and then combining the observation of thoseevents to determine larger scale activity. We employexactly the types of detectors designed for the gram-matical parsing approach above, but apply them in themore forgiving belief network integration method.

ConclusionsWe have described the long-term multidisciplinary re-search agenda of the Aware Home Research Initiative.Situated in a unique living laboratory, we are in a verygood position to explore the technical and human im-plications of ubicomp in a domestic setting. From thehuman perspective, our research is driven currently bythe main theme of aging in place. As researchers, wesee significant intellectual challenges in the domains ofComputational Perception, Human–Computer Interac-tion and Software Engineering and we have used thispaper to review our progress and future goals in thesedistinct areas.

Any significant research problem covers many morechallenges than we have been able to address in thisbrief overview. We realize that much of our workin designing applications for an aware environmentcan be generalized to cover more universal design is-sues. Indeed, several applications projects not re-ported here focus on other demographics in the home,such as families with small children (Stevens, Vollmer,& Abowd 2002). Other research efforts are rightlyfocusing on more general health issues (see Uni-versity of Rochester’s Center for Future Health athttp://www.futurehealth.rochester.edu) or issues of de-signing the physical space to accommodate new tech-nologies (see MIT’s Changing Places/House n projectat http://architecture.mit.edu/house n).

Though what we presented in this overview does notdirectly address social implications of our work, we areexplicitly examining socio-legal implications of sens-

ing in a home environment through a structured explo-ration of privacy litigation. This work is leading towarda defined process to help technologists and designersdiscover how the specific details of sensing and the useof context will be perceived in the wider social and le-gal contexts of modern life.

ReferencesGreenwald, J. 1999. Elder care: Making the rightchoice. Time Magazine 52–56.Intille, S.; Davis, J.; and Bobick, A. 1997. Real-time closed-world tracking. In Proceedings of theIEEE Computer Society Conference on Computer Vi-sion and Pattern Recognition, 697–703. IEEE Com-puter Society Press.Matusik, W.; Buehler, C.; Raskar, R.; Gortler, S. J.;and McMillan, L. 2000. Image-based visual hulls. InAkeley, K., ed., Siggraph 2000, Computer GraphicsProceedings, Annual Conference Series, 369–374.ACM Press / ACM SIGGRAPH / Addison WesleyLongman.Mynatt, E. D.; Rowan, J.; Jacobs, A.; and Craighill,S. 2001. Digital family portraits: Supporting peace ofmind for extended family members. In Proceedings ofthe ACM SIGCHI 2001 Conference, 333–340. ACMPress.Mynatt, E. D.; Essa, I.; and Rogers, W. A. 2000.Increasing the opportunities for aging in place. InProceedings of the 2000 Conference on Universal Us-ability (CUU 2000).Rogers, W. A.; Meyer, B.; Walker, N.; and Fisk, A.1998. Functional limitations to daily living tasks inthe aged: A cocus group analysis. Human Factors40:111–125.Smith, D. 1990. Human factors and aging: Anoverview of research needs and application opportu-nities. Human Factors 32:509–526.Stevens, M.; Vollmer, F.; and Abowd, G. D. 2002.The living memory box: Function, form and user-centered design. In Companion Proceedings of the2002 conference on Human Factors in ComputingSystems — CHI 2002. Interactive Poster. To appear.Vanderheiden, G. 2000. Fundamental principles andpriority setting for universal usability. In Proceedingsof the 2000 Conference on Universal Usability (CUU00).