A Projection-based Augmented Realityfor Elderly People with Dementia

Hyocheol RoMedia System LabYonsei UniversitySeoul, Republic of Koreahyocheol.ro@msl.yonsei.ac.kr

Yoon Jung ParkMedia System LabYonsei UniversitySeoul, Republic of Koreayoonjung.park@msl.yonsei.ac.kr

Tack-Don HanMedia System LabYonsei UniversitySeoul, Republic of Koreahantack@msl.yonsei.ac.kr

Paste the appropriate copyright statement here. ACM now supports three differentcopyright statements:

• ACM copyright: ACM holds the copyright on the work. This is the historicalapproach.

• License: The author(s) retain copyright, but ACM receives an exclusivepublication license.

• Open Access: The author(s) wish to pay for the work to be open access. Theadditional fee must be paid to ACM.

This text field is large enough to hold the appropriate release statement assuming it issingle spaced in a sans-serif 7 point font.Every submission will be assigned their own unique DOI string to be included here.

AbstractAs aging societies grow, researchers are actively studyingcare systems concerning the life and diseases of the el-derly. Among these diseases, dementia makes it difficult tomaintain daily life due to the degradation of cognitive func-tioning, memory, and reasoning, as well as the ability toperform actions. Moreover, dementia does not have a per-fect cure, though therapy and care can slow its onset andprovide patients with physical and mental support. In thispaper, we developed a projection-based augmented realitysystem robot that can cover 360 degrees of space. We alsopropose an application that supports continuous monitor-ing of dementia patients to address the difficulties they facein daily life. The system is also designed to provide ther-apy applications, such as entertainment and spatial art, toprovide mental care aids for the patients.

Author KeywordsProjection-based Augmented Reality; Dementia; ElderlyPeople; Long-term Memory; Mental Therapy; Health Care

CCS Concepts•Human-centered computing → Mixed / augmented re-ality; Human computer interaction (HCI); Interactive sys-tems and tools;

arX

iv:1

908.

0604

6v1

[cs

.HC

] 1

6 A

ug 2

019

IntroductionThe aging society continues to grow as quality of life andmedical technology improve. Recently, dementia was foundto be the third leading cause of death in the elderly, afterheart disease and cancer. Dementia makes it difficult forthe elderly to maintain daily life due to the degradation ofcognitive functioning, memory, and reasoning, as well asthe ability to perform actions. Causes of the illness varywith changes occurring in the brain. Elderly dementia pa-tients are expected to reach 46.8 million worldwide by 2015and double every 20 years [5]. However, there is currentlystill no complete cure for dementia. Current systems slowthe onset of the disease or assist people suffering fromit. To reduce memory loss and support various therapies,studies are being conducted using augmented reality (AR)and virtual reality (VR) that can access a wide range ofmedia and content. Research is also being conducted toaddress neurodevelopmental disorders and improve atten-tion by providing visual experiences such as 360 degreepanoramic video to the elderly through VR headsets. How-ever, VR headsets can cause dizziness (motion sicknessand digital point-of-view strain) and have interfaces difficultfor the elderly to use [3].

Figure 1: Our Projection ARSystem. (top) Portable System.(bottom left) Movable System.(bottom right) Mobile RobotSystem.

Various studies have also used AR headsets to address theproblems of VR headsets; the former improves provides vi-sual information to the real world while improving tangibilityand inducing little dizziness, nausea, and the like. Recentstudies have been conducted on therapeutic systems tostimulate short-term memory and spatial memory usingHoloLens [1]. However, HoloLens has a narrow viewingangle and requires the device to be worn continuously, in-conveniencing the user. Projection-based AR systems thatcan address these problems and simultaneously transmitsinformation to many people and provides virtual informa-tion directly to the real world through projection to deliver

high realism. Research is being conducted on therapeuticcare using interactive artwork that supports user interactionwith projection AR systems. This paper proposes a systemthat can construct an AR environment for all the spaces inwhich the user exists. This will provide the role of life sup-port, care, and therapy to the elderly and patients livingalone or in nursing homes to slow the onset of diseasesand provide convenience to their daily life. Accordingly, theproposed system supports mental care aids such as spa-tial art therapy and memory photo reproduction, in additionto physical support and memory assistance activities. Thissystem can therefore slow the progression of the disease ofdementia patients and improve their quality of life throughtherapy-based mental care.

System ConfigurationOur projection AR system is implemented in various formsas shown in Fig.1. The common point of each system isthat they can analyze a space and track the users and pro-vide the same service even if they are put into an unfamiliarenvironment. The existing researches that provided theimmersive media room form through the projection wasnot possible until the whole system was built in the spe-cific space. On the other hand, no separate installation isrequired to use our system. Therefore, our system can beput into any environment immediately at any time. 3D mapreconstruction using SLAM algorithm was implemented todetect the plane to provide the projection. To do this, we in-stalled an RGB-D camera on a pan-tilt system capable ofrotating 360 degrees, and the system automatically rotatesand collects peripheral space information. RANdom SAm-ple Consensus (RANSAC)-based plane detection works toobtain candidate planes that can be projected. The appli-cations all support speech recognition and (depth) touchinterfaces, and the feedback to them is also made up ofvoice and visual projection.

ApplicationsMonitoringAfter the entire environment is constructed, the proposedsystem continues to track the user’s position through thetracking camera as shown in Fig.2 . The user’s position isexpressed in 3D coordinates in the space through the 3Dcamera, and is used to augment the information throughprojection when the user approaches the plane on whichthe content is displayed. The system is also used to mon-itor abnormal situations of the elderly as well as falls andemergencies. For elderly users living alone, emergency sit-uations often worsen because they cannot request medicalsupport. Accordingly, the system rotates the pan tilt system,continuously monitors the position and status of the user,and provides information to the connected family membersto prepare for an emergency situation.

Figure 2: Monitoring Application.

Figure 3: Internet of ThingsApplication.

Figure 4: Personal AssistantApplication.

Internet of Things / User IdentificationElderly people who suffer from long-term memory prob-lems brought on by diseases such as dementia can lose theability to recognize faces [2]. This phenomenon is partic-ularly pronounced in the ability to identify others in a shorttime. In other words, even if the patient does not forget theother person due to dementia symptoms, they must oftenspend long periods of time remembering who the person is.The proposed system can provide a solution to this prob-lem: an IoT application that is able to identify the user asshown in Fig.3. Assuming that a guest visiting the home ofthe elderly user presses the doorbell, the user is displayeda screen of the IoT door on a certain plane through the pro-jection AR [6]. If the user on the screen is registered in ad-vance, then that person’s name and brief information areprovided to the elderly user. Meanwhile, if the user is notregistered, the result "Unknown" is provided to alert the el-derly user.

Personal AssistantNext, we introduce a personal assistant application thatnotifies the user of the time to take their medicine [6]. In ad-dition to mental illnesses such as dementia, elderly peoplealso suffer from chronic diseases such as hypertension anddiabetes. The prognosis of these chronic diseases variesgreatly with how they are managed after diagnosis. How-ever, medication adherence (taking medication properly ac-cording to instructions from a specialist) for elderly patientsis often reduced as they may easily forget their dosage orwhen to take the medicine. For this purpose, various mobileapplications that notify the time to take the medicine havebeen introduced. However, because the elderly are gener-ally not familiar with using mobile devices such as smart-phones, these methods are also limited. In contrast, ourproposed dosing notification system provides dosing timeintuitively and effectively through auditory and visual effectsas shown in Fig.4. Depending on the situation, a projectionis displayed near the elderly user or in the direction they arelooking, and after taking the medicine, the alarm is stopped.

Media ServicesAccording to a statistical survey [4], media literacy is signif-icantly reduced for the elderly living alone who have littlecontact with younger people. This indicates that the in-formation gap (digital divide) between the elderly and theyounger generation is becoming more severe in aged so-cieties. Particularly for younger generations, the proportionof media utilization (photo and video viewing, etc.) throughmobile devices increases daily, while elderly cannot use thesame services even if they have access to the same device[6]. Therefore, rather than teaching the user how to operatethe mobile device, we implemented a method that directlylinks the system with media services to provide them tothe elderly. Fig.5 is an example of an application in whichthe screen is linked to memories of iOS. The media ser-

vice automatically edits the photos and videos taken by theuser and provides it to them in video form [6]; this serviceis linked to help improve the media utilization of the elderlyuser. Among methods to address the symptoms of demen-tia, one involves using a photograph that can remind thepatient of certain people or places. Such media servicescan be provided to alleviate the symptoms of dementia.

Figure 5: Media Service(Memories) Application.

Figure 6: Painting Spatial ArtApplication.

Figure 7: Color Mixing Spatial ArtApplication.

Spatial ArtA scenario that supports the mental therapy of users suffer-ing from dementia, Spatial Touch is a spatial art applicationin which the user interacts with the interface to combinevarious colors and paint or draw as shown in Fig.6 and 7.Through AR spatial art based on visual projection, patientscan improve their freedom of expression and imagination,while the use of therapy also assists with their mental care.By interacting through a paintbrush, a metaphor for thesame action in the real world, the improvement of the user’sphysical ability is supported. In real world settings with noseparate touch sensors, this method can support intuitiveand natural interactions through spatial touch interactions.The user is presented an example picture through the pro-jection, and the user can trace the picture or paint based onit using the brush, the interaction metaphor.

ConclusionIn this paper, we propose a variety of applications for careof patients with dementia using the projection AR system.It provides the functions to support mental and physicaldifficulties. It also supports media services and therapiesthat can reduce emotional lability. Our AR platform providesproper information and intuitive interactions. Therefore, itcan provide various services to dementia patients effec-tively. In future works, we will explore the future possibilitiesof projection AR and dementia care by demonstrating theseapplications directly to patients and receiving feedback.

References[1] Beatrice Aruanno, Franca Garzotto, and

Mario Covarrubias Rodriguez. 2017. HoloLens-basedMixed Reality Experiences for Subjects withAlzheimer’s Disease. In Proceedings of the 12thBiannual Conference on Italian SIGCHI Chapter. ACM,15.

[2] Toshikazu Kawagoe, Masateru Matsushita, MamoruHashimoto, Manabu Ikeda, and Kaoru Sekiyama.2017. Face-specific memory deficits and changes ineye scanning patterns among patients with amnesticmild cognitive impairment. Scientific reports 7, 1(2017), 14344.

[3] Seizo Ohyama, Suetaka Nishiike, Hiroshi Watanabe,Katsunori Matsuoka, Hironori Akizuki, Noriaki Takeda,and Tamotsu Harada. 2007. Autonomic responsesduring motion sickness induced by virtual reality. AurisNasus Larynx 34, 3 (2007), 303–306.

[4] Natalie Pang, Samantha Vu, Xue Zhang, and SchubertFoo. 2015. Older adults and the appropriation anddisappropriation of smartphones. In InternationalConference on Human Aspects of IT for the AgedPopulation. Springer, 484–495.

[5] Martin Prince, Anders Wimo, Maelenn Guerchet, GCAli, YT Wu, and M Prina. 2015. The global impact ofdementia: an analysis of prevalence, incidence, costand trends. World Alzheimer Report 2015 (2015).

[6] Yoonsik Yang, Yoon Jung Park, Hyocheol Ro, SeunghoChae, and Tack-Don Han. 2018. cARe-bot: Portableprojection-based AR robot for elderly. In Companion ofthe 2018 ACM/IEEE International Conference onHuman-Robot Interaction. ACM, 384–384.