Post on 15-Apr-2017
Eldercare ResearchCognitive Computing for Aging Society
Hiro TakagiSTSM, IBM Research - Tokyo
Aging Strategic Initiative lead, IBM ResearchHuman Computer Interaction area lead, IBM ResearchAcademy of Technology member
62 87
28%
27%
Dementia 15% + MCI 13%estimation by Japanese Government, 2012
of people over the age of 65 are at risk of dementia.
People over the age of 65, 2017.
70%of people want to work over the age of 70 (and forever).
Life expectancy of women in 1950 and 2015.
* numbers in Japan
2017
We need a new model for active life-long society.
©2016IBMCorporation
2003“Elderly Vision Simulator”
2007Eclipse.orgOpen source
2011Official tool for JIS (Japanese Industrial Standard)
2011Japanese Government Science Grant“IT for Active Aging Society”
2009Open Collaborative ResearchUniversity of Tokyo“Elderly Accessibility”
2013Japanese Government Pilot Grant“Remote community collaboration”
2015Sekisui House JDA
2017Japan Post Official Service “Conversation as Sensors”
GTO 2016 Demographic Shift
2015Japan Post Announcement
2015 – 2016Japan Post Pilot “Voice QA”
2005 2010 2015 2016 2017
Tokyo Research Aging Research History
Research Aging Strategic Initiative2014
KDDI “Senior Smartphone”
Government and University projects Customer projects Internal activities
Power of IT – Five Long-term Focus Areas
Social
Intelligent assist
Sensing
Accessibility
Brain
Social
Intelligent assist
Sensing
Accessibility
Brain
Use case barrier
Psychological barrier
User interfacebarrier
Price Barrier
�Small fonts, small buttons�Don’t know how to operate
�Fear, uncertainty
�Cannot imagine use cases
Barriers for Senior Citizens to Start Using Information Technologies
Target Actual(average)
Fingertip angle + Parallax
Landed on
Took off
(px)
(px)
Touch duration + Tremor
Unintentional Motion
Accessibility: Touching a target location
Gesture Analysis for Novice Elderly Assist (2014)
Pressrelease&images (Japaneseonly)IBMJapanPressRelease:http://bit.ly/1gavLuNKDDIR&DLabsPressRelease:http://bit.ly/1hK1B2KKDDICorporationPressRelease:http://bit.ly/1f5zbTx
Providedappropriateguidancetousersbasedonuser'slevelofsmartphoneoperationproficiency(skillandoperationalmisstep)byanalyzingsmartphoneoperationlogswithmathematicalanalysistechnology
Improved confidenceandsubjective operation speed
33 0 3 -- 7 0 -6 067 ,3
-- 7
6 067 ,3
2 7 - 2 1
172 17
You entered“one-hundred thousand” yen
Accessibility: Voice-over Effectiveness
Social
Intelligent assist
Sensing
Accessibility
Brain
Elderly Skill Matching System – J-scouter
Senior profiledatasets
(Free-form text)
Job / ActivityDescriptions
(Free-form text)
Find bestpersons
Find bestjobs/activities
A Prime minister initiative“Work-style transformation.”“Senior workforce” is one of focuses.
A senior worker with rich experience in legal compliance is working as an advisor in a start up company
Supported by Japan Science and Technology Agency
Senior HR company10,000 seniors registered
from TV BS11By Kyodo News
“ReimaginingLife”
“Target 5 million users.” http://www-03.ibm.com/press/us/en/pressrelease/46740.wss
2015
Japan Post Project• Pilot with 1000 elderlies completed (Oct. 2015 – Oct. 2016)• Medicine reminders, video phones, shopping assist, photo sharing, etc.
Video
Global Apple+IBM Apps for Elder Services
SupportWorker
Elderat
Home
Family /LovedOnes
Social
Intelligent assist
Sensing
Accessibility
Brain
)( )
(
-- (-
(
( ))
-
( -
-
-(
Understandambiguousquestions
Pick upappropriate
answers
Learnuser
feedback
- -
Video
Social
Intelligent assist
Sensing
Accessibility
Brain
IBM Accessibility Research
IoT, Health and Aging
Ecosystem Partners / Enriched Data
Watson IoT & Health Platform
Sensors & Devices
Client’s Data
Personality Insights & Social Interaction Data
Other Data Sources
ApplicationsData &
Platforms
0101100010001001
Cognitive Services for Elders
Empowered Living
Empowered Social
Empowered Care
KnowledgeReactor
“Infrastructure Providers”
• Voice-base natural user interface for daily watchover• Analyze life patterns, feelings, interests and issues from daily conversation (e.g. cognitive decline detection)• Share information with families and care givers• Provide support that can keep the elderly self-sufficient
Conversation as Sensors
©2014IBMCorporation
Video
IntentofEachQuestionnaire
Questionnaire Intent“What did you have last night?” Dementia (Neuro Cognitive Disease) Risk
Assessment
“Did you go out today?” Activity checking“You mentioned about gardening in diary. Are you interested in community gardening?” Recommendation of activities
”Which city do you want to travel?” Marketing“The shop, you mentioned last week, is now on sale!” Promotion“We delivered a package, but you were not there. Please contact post office" Notification (business related)
25
Emotion and Physical Condition Recognition
Voice data Features
988 acoustic features for emotion recognition.
IntensityLoudness12 MFCCPitch (F0)
Probability of voicing, F0 envelope,
8 LSF, Zero-Crossing Rate
���
Labeled voice
dataset
ML-basedRegression
Estimation
Feeling good or bad
40
0
100
Physical condition
Ex: Good night
*Each circle represents one greeting.*One participant data
Emotion
NegativePositive
Confusion FrustrationDelight Flow
Physical Condition
EmotionYorktown – Speech team
Tokyo Research
Happy
Sad
Dashboard for Family and Care Givers
27
Condition
Emotion (voice sound analysis)
Personality analysis (Watson personality insights)
Daily activities
Eating habits
Dementia questionnaire scores
���
Dashboard Visualization Examples
28
“what did you do today?” = Variety of daily activities.
practice, songs, poems, musical instruments, sessions, friends, music hall, …. House (largest), cleaning, washing, room,
study…..
Smaller variety and words are not so active. Wider variety, and words are active.
“what did you have today?” = Variety of nutritionWider variety, and healthy meals
Vegetables, salad, radish, juice, natto, miso, eggs, ….
Rice (largest), bento, sandwich, ….
Smaller variety and ready-made meals
Feature extraction of gait
• Gait speed & its variability• Step frequency• Stride time variability• Step-length & its variability• Foot swing velocity• Stance and stride time
Detection• Motor function
• Fall risks• Cognitive decline
• Episodic memory• Executive function
• Diseases• Parkinson disease• MCI• Alzheimer’s disease
Gait and Cognitive Decline
Video
Social
Intelligent assist
Sensing
Accessibility
Brain
Brain Simulation: Understanding Cognitive Deficit (2016)
Video
Power of IT – Five Long-term Focus Areas
Social
Intelligent assist
Sensing
Accessibility
Brain