A PROBABILISTIC MODEL FOR

COGNITIVE-AFFECTIVE USER STATE

AWARENESS

ARTEMIS JOINT UNDERTAKING

GRANT AGREEMENT: 269334

PROPOSAL ACRONYM: ASTUTE

Astute project 2

INTRODUCTION

Higher data flow

Complex services

Embedded Systems are becoming more and more complex

Different applications

Data

Information

Advice

Actions

Support

Confirm

Error

WHY?

Astute project 3

INTRODUCTION

ASTUTE project aims to improve usability of embedded systems by using user state and context situation capture to provide pro-active decision

support via multi-modal interfaces.

Pro-active decision support system based on human centered design able to support user intentions while keeping him in control.

CONTEXT CAPTURE

CONTEXT MODELING

ENGINE

PROACTIVE DECISION ENGINE

MULTI-MODAL HMI

Astute project 4

USER AWARENESS

PHYSICAL ENVIRONMENT

CONDITIONS

PHYSIOLOGIC MEASURES

PROCESSING DATA INPUTS USER

STATE

•Information Priority

•Decision Support

•Data delivery

•Keep in control

Astute project 5

USER STATE

USER STATE

PHYSICAL ENVIRONMENT

CONDITIONS

SENSOR NETWORKS

EXTERNAL DATABASES

VISUAL ODOMETRY

SPECIFIC MEASUREMENTS

PHYSIOLOGIC MEASURES

ECG

EEG

USER FEEDBACK

MOTION

ARTIFICIAL VISION

BIOMETRICS

Astute project 6

USER STATE

EEG:

Electroencephalography is the measurement of electrical activity resulting from ionic currents flows in brain neurons using multiple electrodes placed on the scalp.

It is commonly used in medicine for diagnostic applications, like epilepsy, encephalopathies or sleep disorders, by analyzing its spectral content.

Further applications include EEG average analysis for cognitive sciences by analyzing response to time-locked events and stimulus.

STRESS FATIGUE

FATIGUE

CONCENTRATION

RELAXATION

DISEASES

Astute project 7

USER STATE

ECG:

Electrocardiography is the transthoracic measurement of electrical activity in heart using different electrodes attached to user’s skin.

It is commonly used in medicine to measure heartbeats rate, size and position of the different heart chambers and any effect of external source on heart.

Although ECG information is limited to physiological user status, it is useful to complement EEG data to complement obtained information increasing system performance and reliability.

How this information is merged?

Astute project 8

RATIONALE & MOTIVATION

• Cognitive-affective states are relevant in the realisation of tasks that:

– Manage a large volume of information in the interface/system/process – are a cognitive challenge

– Are critical (urgency, safety, health, sports)

– Involve people (human resources, leadership, coaching, social and personal relationships)

• Availability of information on user state facilitates interface/system/process pro-activeness, which is accomplished via decision support built on top of data-/knowledge-based models, providing:

– Alarms, feedback

– Recommendations for adapting the interface/system/process

Astute project 9

RATIONALE & MOTIVATION

• Present work provides a description of research and development of a user state diagnostic system, within the broader context of the European Artemis ASTUTE project ASTUTE aims to develop advanced and innovative pro-active HMI

supported by reasoning engine system, for improving the way the human being deals with complex and huge information quantities in different operative conditions and contexts.

• A number of previous projects generally used a limited range of sensors network, mainly focused on autonomous psycho-physiological information, and used concrete context scenarios

• More technical effort may be done to incorporate measures of brain activity, and thus to delineate a full picture of brain-body reaction

• A probabilistic model has been developed, given its capability to handle uncertainty in sources information and inference, and sound mathematical framework.

Astute project 10

OVERVIEW

Sensors

User Profile Context

Web services

Cloud App Mobile

User state

Mobile Cloud App

Astute project 11

DATA CAPTURE

EEG raw data Frequency bands

α, β, θ,... Calibration & normalisation

Heart rate

User’s profile

Calibration & normalisation

To user state model

Web services connectivity

Astute project 12

OUR COGNITIVE MODEL

• Three user states that are relevant in working environments and safety-critical tasks: – stress, mental workload and fatigue

• A fourth state, namely inaptitude, is derived as a combination of the three aforementioned user-states (stress, mental workload and fatigue): – this is input to decision support, where recommendation for assistance will be

given

Astute project 13

• Input used for diagnosis of user state comes from – brain activity (EEG) and heart activity sensors

• This is complemented by selected – predictive factors from context (context complexity, task workload) and – user profile (experience, age, fitness), – extracted from each user-case

OUR COGNITIVE MODEL

Astute project 14

OUR COGNITIVE MODEL

p(Symptom2 | Disease1, Disease2) Symptom2

Disease1 Disease2 False True

False False 80 % 20 %

False True 40 % 60 %

True False 30 % 70 %

True True 20 % 80 %

As the number of parents of a node increase, conditional probability tables (CPT) become larger. Limit relations between nodes if there’s conditional independence Use canonical distributions

NoisyOR is used in our model Adapt distributions with statistical analysis

of a large amount of training data

Astute project 15

OUR COGNITIVE MODEL

• This first model can be extended with “contrasts”

– for verifying the need for assistance via the state inaptitude.

– In the end, this strategy improves the decision reliability

Astute project 16

OUR COGNITIVE MODEL

• Assistance recommendation is provided – alarms, voice, warning

messages, etc.

– assistance is based on the expected utility in order to advice the user

Astute project 17

USAGE EXAMPLE

Astute project 18

USAGE EXAMPLE

Astute project 19

SCHEMATIC ARCHITECTURE

Astute project 20

FUTURE WORK

• additional sensors – for example to complement existing information or to cope with the

unavailability of specific sensors in particular settings

• further input parameters extracted from existing sensors • improved pre-processing and fine-tuning of features can enhance

robustness • new user’s cognitive and affective states • further exploitation of user’s state contrasts and assistance • exploring a dynamic version for anticipation or prediction of the

user’s state • adaptability to users via training procedures and to improve the

prediction capacity of the cognitive model • mobility and user profiling and personalization are key to our

system and deserve significant attention

Astute project 21

CONCLUSIONS

• we’ve shown present and future work within the framework of ASTUTE project

• emphasis is on providing an integrated solution to monitor and adapt the user’s state to the task demands in complex contexts – to design and implement a probabilistic cognitive model that is

predicting the user’s state based on complex context use-cases – a set of sensors is diagnosing such user’s states based of brain

and heart rate evidence

• this solution is partially overcome by other projects, however we add value in increasing the set of sensors, range of user’s states in real and intensive scenarios

Astute project 22

THANKS FOR

YOUR ATTENTION!