Midi2013 presentation

Project of multi-modal application based on artificial intelligence software

A Dialogue With a Virtual Imaginary Interlocutor as a Form of Psychological Support for Well-being

Agata Pasikowska - University of Social Sciences and HumanitiesAbolfazl Zaraki - Reaseach Center “E.Piaggio”, University of PisaNicole Lazzeri - Reaseach Center “E.Piaggio”, University of Pisa

Warsaw, 25th June 2013(c) Pasikowska, Zaraki, Lazerri

1. Introduction

2.The Application

The Idea

Structure of The Application

User Flow

3.Technological Methodology

Power of Mobile Devices

Structure of The System

Brainwave Reader Module

Questionnaires Module

Facial Expression Recognition Module

Voice Analysis Module

Interface - chatbot

AIML

Potential technology for Avatar

4.Summary

5.Questions&Answers

(c) Pasikowska, Zaraki, Lazerri

Introduction


❖ create a virtual reality application to sustain well-being and improve quality of life

❖ develop a solution for self-discovery and broadening techniques for coping with stressful

situations

❖ build a multi-modal application based on storing and tracking over the time both verbal and

non-verbal cues but also on the explicit declarations of the users through the questionnaires

and the chat activity with a chatbot

❖ make a research tool for PhD study “Dialogue with an imaginary person and a coping with

difficult situations”

Aim of the Project


❖ The concept of interactive mind assumes different states, based on a combination of internal and

external dialogue. Internal dialogue can be as effective in generating possible solutions to the

problem situation (Baltes and Staudinger, 1996)

❖ Inner dialogue is a tool of support (Puchalska - Wasyl, Chmielecka - Kuter, Oleś, 2008)

❖ Inner dialogue with an imaginary person is a supportive coping technique in a stressful situation

(Pasikowska, 2013)

Importance of the Project

Staudinger U. M., Baltes P. P. (1996). Interactive Minds: A Facilitative Setting for Wisdom-Related Performance. Journal of Personality and Social Psychology 1996, Vol 71, No. 4, 746-762.Pasikowska, A (2013). Dialog z wirtualnym i wyobrażonym rozmówcą jako forma wsparcia i poprawy zdrowia psychicznego (w druku).Puchalska-Wasyl, M., Chmielnicka-Kuter, E. i Oleś, P. K. (2008). From internal interlocutors to psychological functions of dialogical activity. Journal of Constructivist Psychology, 21, (3), 239-269.


Importance of the Project

❖ What is the function of internal dialogue?

❖ Is the internal dialogue is particularly effective way to benefit from support in a stressful

situation?

❖ Is a meeting with a person is needed?

❖ How to build a multi-modal application to support internal dialogues benefits?


The Application


❖ born during Human Computer Confluence Summer School 2012

in Milan

❖ uses the internal dialogue as a form of support in daily challenges

and problems of life

❖ is based on artificial intelligence software that learns our internal

dialogue and delivers it to us in the form of a virtual conversation

with the imaginary interlocutor

The Idea

!


INTERFACE -CHATBOT

USER

MOOD DETECTOR

Structure of the System


User Flow


Technological Methodology


Power of Mobile Devices

Xpression Application allows users to record their mood through their speech acoustics

Aurora App is fostering socially supportive behaviors by the sharing of affect on mobile devices.

The INTERSTRESS project is a VR treatment of psychological stress.

Aurora: http://edition.cnn.com/2013/02/26/tech/mood-tracking-app-xperience/index.htmlXpression: http://idl.cornell.edu/projects/aurora/ INTERSTRESS: http://interstress.eu


http://idl.cornell.edu/projects/aurora/


http://en.wikipedia.org/wiki/Stress_%28biology%29

http://en.wikipedia.org/wiki/Stress_%28biology%29

http://edition.cnn.com/2013/02/26/tech/mood-tracking-app-xperience/index.html?iref=allsearch

http://edition.cnn.com/2013/02/26/tech/mood-tracking-app-xperience/index.html?iref=allsearch



http://interstress.eu

http://interstress.eu

INTERFACE -CHATBOT

USER

MOOD DETECTOR

Structure of the System


Brainwave Reader Module

❖ NeuroSky device to capture Brainwave signals

❖ It is not really an easy task.

❖ This part of the work is prototype and we haven't

finalized it yet. The output of the system will be sent to the

Mood Detector engine.

http://www.neurosky.com/products/mindwavemobile.aspx(c) Pasikowska, Zaraki, Lazerri

http://www.neurosky.com/products/mindwavemobile.aspx

http://www.neurosky.com/products/mindwavemobile.aspx

Questionnaires Module

❖ effectiveness of the coping techniques controlled by achieved

support.

❖ level of self-esteem

❖ anxiety

❖ emotional intelligence

❖ quality of life.



❖ Standard face detection and facial analyzer engine

❖ Strong feedback for the system

❖ Reflect emotional mood of user.

❖ Angry, happy, sad, surprise also age and gender


http://file://localhost/users/agatka/downloads/agata_2_videook.wmv









❖ Standard face detection and facial analyzer engine

❖ Strong feedback for the system

❖ Reflect emotional mood of user.

❖ Angry, happy, sad, surprise also age and gender








Voice Analysis Module

❖ Analyses the user’s voice to obtain an in-depth view of her/his range of emotions inferring

the stress and anxiety level.

❖ Voice analysis in real-time is a challenging issue. In this work we prefer to use voice signals as

a communication tools between users and system.

❖ We integrate speech recognition engine which recognizes some keyword that are trained by

user before experiments.

(c) Agata Pasikowska(c) Pasikowska, Zaraki, Lazerri

Interface - chatbot

For example:

<category>

<pattern>HELLO</pattern>

<template>Hi there!</template>

</category>!

User says "Hello"

Chatbot answers "Hi there!"=


AIML❖ The AIML pattern language is simple, consisting only of words,

spaces, and the wildcard symbols _ and *. The words may consist

of letters and numerals, but no other characters. The pattern

language is case invariant. Words are separated by a single space,

and the wildcard characters function like words.

!

User says "Yes"

Chatbot asks "Do you like movies?"

=

Another example:

<category>

<pattern>YES</pattern>

<that>DO YOU LIKE MOVIES</that>

<template>What is your favorite movie?</template>

</category>“What is your favorite movie?”


❖ Faceshift analyzes the face motions of an actor, and describes them as a mixture of basic

expressions, plus head orientation and gaze.

Potential technology for Avatar

http://www.faceshift.com

http://youtu.be/Bz4WSLUy0Wo






Summary

We will:

❖ Create an innovative e-service which aims to increase knowledge of the

human phenomenon of internal dialogues

❖ Support people at risk of stress

❖ Develop a new technology for self-discovery and improve the quality of life

❖ Build an artificial intelligence in the form of virtual speaker for storing and

tracking over the time both verbal (a voice analysis) and non-verbal

(brainwave, facial emotion recognition) cues but also on the explicit

declarations of the users through the questionnaires and the chat activity

!(c) Pasikowska, Zaraki, Lazerri

Q & AAny technical questions? Please contact:

Abolfazl Zaraki ([email protected])

Nicole Lazzeri ([email protected])


mailto:[email protected]




Thank You.Agata Pasikowska ([email protected])




Midi2013 presentation

Design

Transcript of Midi2013 presentation