V Jornadas eMadrid sobre “Educación Digital”. Jesús G. Boticario, Universidad Nacional de...

107
aDeNu-UNED 2015 © Jesus G. Boticario [email protected] aDeNu Research Group UNED

Transcript of V Jornadas eMadrid sobre “Educación Digital”. Jesús G. Boticario, Universidad Nacional de...

aDeNu-UNED 2015 ©

Jesus G. Boticario [email protected] aDeNu Research Group UNED

USER MODELING (LEARNING)

Student Centered AI-ED / ML / EDM /

LAK / ITS / CSCL / L@S… ??

Context & Motivation

Different User Agents

Varied Assistive Technologies Multiple contents format

Negotiation requirements

¿ ?

¿ ?

Context & Motivation

Observable features and AMI

RFID tracking Movements tracking

Face tracking

Context & Motivation

Standards support transferability and interoperability

Research Questions

Standards usage & extension

Adaptive interaction support (run-time affect oriented) Student’s centered learning (UCD affect oriented)

ADAPTIVE & AFFECTIVE & INCLUSIVE &

LEARNING SYSTEMS

Affect detection: Accurately detect meaningful features in LEARNING

New inputs & outputs: modality fusion

Educational Issues Pedagogical interventions

New learning opportunities

Transferability Domain independent features

Transferrable processes

2. aDeNu Background

http://adenu.ia.uned.es/

[email protected]

Accessibility Guidelines & Standards W3C Web Accessibility Initiative: WCAG (contents), ATAG (authoring tools), UAAG (user agents) ISO 24751, IMS-LIP, IMS-AfA, IMS-MD, IMS-CP, IMS-MD, IEEE-LOM, IMS-RDCEO, ISO/IEC TR 24763, IMS-QTI, SCORM, W3C CC/PP

User Modeling Individual & CSCL (needs, preferences, learning styles, CSCL features...) Monitoring dynamic information from interactions

aDeNu projects

Standards-based Modeling

<imsld:locpersproperty identifier="resourcetest4.score">

<imsld:datatype datatype="integer" />

<imsld:initialvalue>0</imsld:initialvalue>

</imsld:locpersproperty>

<imsld:learning-activity identifier="LA-task41">

<imsld:title>4.1. Impressionism Evaluation</imsld:title>

<imsld:activity-description>

<imsld:item identifierref="resource-test4" />

</imsld:activity-description>

<imsld:complete-activity>

<imsld:when-property-value-is-set>

<imsld:property-ref ref="impeval-good-enough"/>

<imsld:property-value>true</imsld:property-

value>

</imsld:when-property-value-is-set>

</imsld:complete-activity>

</imsld:learning-activity>

<imsld:conditions>

<imsld:if>

<imsld:greater-than>

<imsld:property-ref ref="resource-test4.score"/>

<imsld:property-value>5</imsld:property-value>

</imsld:greater-than>

</imsld:if>

<imsld:then>

<imsld:change-property-value>

<property-ref ref="impeval-good-enough" />

<property-value>true</property-value>

</imsld:change-property-value>

</imsld:then>

Practice

Contact Tutor

QTI re-test

Remediation

QTI test

Inductive/visual Deductive/verbal

Yes

No

Yes

No

Learning Style

Property value?

Mastered QTI test?

Mastered

Re-test?

Practice

Contact Tutor

QTI re-test

Remediation

QTI test

Inductive/visual Deductive/verbal

Yes

No

Yes

No

Learning Style

Property value?

Mastered QTI test?

Mastered

Re-test?

Standards-based Modeling

aLFanet

CSCL & OLM

CSCL: The CLFA

User profiles Participative, Insightful, Useful, Non-

collaborative, With-initiative, Communicative

Thinker-out, Unsecure, Gossip, Inspirable, Inspiring, Thorough

Forum conversations started Forum messages sent Replies to student interactions

N_thrd = ∑in(xi); x number of

threads started on day i and n a

set of days in the experience

N_msg = ∑in(xi); x number of

messages sent on day i and n a

set of days in the experience

N_r_thrd = number of

messages in the thread started

by user

M_thrd = average (N_thrd) =

(1/N)( ∑in(xi)); N number of

days in the experience

M_msg = average (N_msg) M_r_thrd = N_r_thrd / N_thrd

V_thrd = variance (N_thrd) V_msg = variance (N_msg) N_r_msg = number of replies

L_thrd = N_thrd /√V_thrd L_msg = N_msg /√V_msg M_r_msg = N_r_msg / N_msg

Accesibilidad y Adaptación

Agente Usuario

Tecnologías de Apoyo

12/06/2008 ALPE Workshop - EDeAN Congress

aLFanet, ADAPTAPlan, FAA, ALPE, EU4ALL

LMS-

EVA Apoyo

autoría

Valoración

Necesidades

Accessibility

support

Upload and tagging

Accesibilidad

Recurso

Comunicación y soporte

Descripción de

necesidades Evaluación

Adaptación

Recursos

Needs

Related

Guidance

Accessibility

Evaluation

& Guidance

Adaptation

Request &

Supervision Trans-

formation

Tagging

Supervision

Feedback Resource-Course

Feedback

Profesor

Estudiante

Estudiante

Personalised

resource

Bibliotecario Técnico de

Transformación

Resource

Feedback

Resource

Feedback

UM

MR

CP

RS

Accesibilidad

Curso

Senior

Manager

Decision

Taking

DM

Framework

component

EU4ALL

eService

Técnico Atención discapacidad

UC-AI→ Accessibility / Context??

User who requests textual captions for the

audio on videos. ISO PNP <accessForAllUser>

<content>

<adaptationPreference>

<adaptationType=”caption”/>

<originalAccessMode=”auditory/”>

<usage=”required”/>

<language=”eng”/>

</adaptationPreference>

</content>

</accessForAllUser>

UC-AI → Interoperability / scale-up??

CONTENT PERSONALIZATION

CP Service

UM

DM

MR

LMS

IMS-LIP

ISO-AfA

ISO-

DRD

CC/PP

Rules

File

SOAP

SOAP

SOAP

RESACCINFO

Service

SOAP

SOAP

• Isa

– Daisy as an alternative to the SCORM

– Transcript for video

• Leo

– Sign language

– Subtitles for video

EU4ALL architecture components CP: Content Personalization UM: User Modeling MR: Metadata Repository DM: Device Model

LMS: Learning Management System (Moodle, dotLRN, Sakai)

UC-AI: Recommendation delivery

Recommenders The SERS approach

Extend e-learning services with ANS actions (read / contribute) on

platform objects

standard-based service oriented architecture (IMS, W3C, ISO)

Recommendations model Elements: type, content,

runtime information, justification, recommendation features

Support for eliciting

educational oriented Rec-s TORMES methodology

▪ UCD based (ISO 9241-210)

TORMES methodology

Service oriented architecture

Example of Rec Recommendation 1: Read the tutorial on how to use the platform

Object: tutorial Action: read

Content (text + link): “Visit the platform tutorial of the platform”

Title: “Access to the page with the platform tutorial”

Applicability conditions:

The learner is new to the platform

The learner has interacted with the platform several times

The learner has not accessed the tutorial

The learner has not contributed in any of the platform services

Restrictions:

There is a tutorial in the platform

Category:

technical

support

Stage:

getting used to the

platform

Origin:

tutor

Relevance:

4.2

Rationale: make the learner get familiarised with the platform

Explanation: “Since you are new to the platform and you have not yet used the

services available in the platform, you can access this tutorial to get familiarised

with the platform operation”

Recommenders UGIs

Recommenders SERS Experiences

ID Description

R1 Choose a lesson to review

R2 Start the review of the concepts

R3 Review the concept estimated as less known

R4 Use the forum to share a doubt

R5 Read a thread of the forum with many posts

R6 Read the educators’ welcome message

R7 Change the avatar that represents

R8 Change the avatar that represents the learner

R9 Look at the learner conceptual model

R10 Look at the conceptual model of the class

R11 Log in to start the course for the first time

R12 Log in to keep reviewing the contents

Indicators Description p

avg_sessions participants who received recommendations spent more sessions in average 0.0362

avg_hits participants who received recommendations made more hits (visited more pages) in average

0.0241

ratio<2days_period less participants from those who received recommendations entered less than 2 days

0.0136

ratio_all_days_connected more participants from those who received recommendations had sessions every day

0.0299 0.0150

avg_connection_window the connection window (number of days between the first day and the last day) is larger in average for those who received recommendations

0.0019

avg_days_to_enter participants who received recommendations waited less days in average to enter the module

0.0206

avg_days_to_end participants who received recommendations spent more days in average 0.0169

avg_days_connected participants who received recommendations connected in average more days

0.0357

P values for Engagement

Indicators

Description R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12

ratio_all_correct_answers Percentage of learners who answered all questions correctly

+ + + + + + + +

avg_correct_answers Average of questions answered correctly by the learners

+ + + + + + + + + + +

Knowledge acquisition confidence level 95%

Modeling Affect

aDeNu-UNED 2015 ©

MAMIPEC Project (TIN2011-29221-C03-00)

Partners Funded by

TORMES elicitation cycle

Understand Context of

Use

Interviews & Questionnaires:

educators’ best practices on

affective issues

DM analysis: affective data

processing complement educators’

descriptions

Specify requirements

Scenario based approach: problem

(situations demanding affective

support) and solution (affective recs

in terms of semantic affective rec

model) scenarios

Evaluate designs against

requirements

Running prototype / Wizard of Oz:

educators’ & learners’ evaluation of

affective recs.

Produce design solutions

Focus group (educators with

affective computing experience)

validate model-formulated affective

recs

eliciting

affective

recs

validated

semantically

described

affective

recs

Understanding recommendations needs

in intelligent educational contexts that

consider multimodal approaches

for affective modelling

[@ RecSysTEL 2012

ISO 9241-210

iterative design

cycle

Motivation

Relation between emotions & cognitive

processes

Advances in affective computing

AIED 2015 Doctoral Consortium 24 June 2015, Madrid, Spain 27

Wrap up: State of the Art

→ Identify research gaps

Psychology Computer Science

Learning technologies

Parasympathetic Nervous System

Physiological signals

Emotion theories

Signal Processing Interaction indicators

Affective Computing Data Mining

Learner Modelling

Intelligent Tutoring Systems

Ubiquitous learning

Large Scale Experience (≈ 80 users / 4Tb)

Sensor belt with the following

sensors: Electrocardiogram

(ECG), Galvanic Skin Response

(GSR), Respiratory Rate (RR),

and Blood Pressure (BP)

Kinect for Windows for face

features extraction

Webcam (with integrated

microphone) and infrared-light

webcam

Keyboard and mouse (via a

keylogger and a mouse tracker)

Questionnaires: General Self-

Efficacy Scale (GSE), PANAS,

BFI, SAM

Large Scale Experience (≈ 80users / 4Tb)

Desktop / Mouse / Keboard tracking

Mouse: 17:24:29:734-154,183 [mov] [154, 183] 17:24:29:781-154,184 [mov] [154, 184] 17:24:29:812-154,185 [mov] [154, 185] 17:24:29:968-154,185 [pre] [left] [154, 185] 17:24:30:093-154,185 [rel] [left] [154, 185] 17:24:30:250-156,185 [mov] [156, 185]

Keyboard: 16:59:32:484-48 [down]->0

16:59:32:546-48 [up]->0

16:59:42:062-32 [down]->SPACE

16:59:42:156-32 [up]->SPACE

Labelling emotions: SAM

Valence / Arousal

Task-1: math problems (lack of time)

Task 2: Logic series (easy task)

Affect Reports

Me he divertido resolviendo los ejercicios

Me he sentido orgulloso/a por haber sido

capaz de resolver los ejercicios

Me he sentido enfadado/a por la dificultad de

los ejercicios

Me he puesto nervioso al enfrentarme a la

resolución de los ejercicios

Me he sentido avergonzado por no ser capaz

de resolver los ejercicios

Me he desesperado tratando de hallar la

solución de los ejercicios

Me he aburrido haciendo los ejercicios

SC:

Al realizar esta tarea me he sentido ...

Al realizar la tarea he pensado ...

Las dificultades que he encontrado para resolver la tarea han sido..

Y para superar estas dificultades he ...

Puedes usar: ABURRIDO, ACTIVO, ADMIRADO, AGOBIADO,

ALEGRE, AVERGONZADO, CABREADO, DEFRAUDADO,

DEPRIMIDO, DESANIMADO, DESESPERADO…

Minería de datos (4 TB): Diseño

Puntuación Sentiment

analysis

Interacciones de usuario

Medidas fisiológicas

Puntuación de la tarea

Cuestionarios

Algoritmos de DM

SAM del usuario

Etiquetado emocional de los reportes emocionales

Etiquetado de registros

Predicción de la puntuación de

valencia afectiva

Inclusive scenarios: Experimental design

Physiological sensors, Kinect, Mouse, Webcam

Questionnaires Sensor

placement Initial Baseline

Sensor calibration questions

Task 1

(Problem solving)

Emotional Report 1

Task 2

(Problem solving with time limit)

Emotional Report 2

Task 3

Logical series

Emotional Report 3

Final Baseline

Physiological sensors, Kinect, Mouse, Webcam

Physiological sensors, Kinect, Mouse, Webcam

Physiological sensors

SAM

SAM

SAM

SAM

Keystrokes

Keystrokes

Keystrokes

Personality traits

Ad

ap

tati

on

s fo

r in

clu

sive

ne

ss

[@ HCI 2013]

Inclusive scenarios: Physiological Signals

DESAFÍOS ANTE LA DIVERSIDAD FUNCIONAL

Adaptaciones en logística, tareas y técnicas de interacción. Ajustes en tamaño de fuentes en pantalla y papel. Ajustes en luminosidad. Reducción distancia a la pantalla (20ctms) que interfirió en el

dispositivo de Kinect. Filtrado por parte de Kinect de movimientos estereotipados

relacionados con discapacidad visual. Estudiar de manera diferencial el patrón de interacción

mediante teclado durante navegación/introducción de datos.

Cambios fisiológicos y conductuales en función de la dificultad y la limitación en la tarea

Sonidos con componente afectivos provocaron cambios fisiológicos.

Research question

How ERS can take advantage of affective computing

to improve the personalized support in educational scenarios with

emotional and affective issues?

Approach

User Device e-Learning

Platform

User Model

• Learning outcomes • Questionnaires scores bio-feedback

devices

sensor & interaction

data

Learning Interactions

Personality Traits

Data Mining

Affective information

+Learner

Affective

Model

Affective Feedback

[@ CAEPIA 2013]

Some Recommendations identified

R1: Provide course instructions for newbie learners, so they do not get lost in the course space

R2: Carry out self-assessment questionnaires to foster learners’ meta-cognitive issues

R3: Review related course concepts to help progressing in the course contents

R6: Propose strategies to cope with temporal failures of the platform, especially when deadlines are approaching

R8: Change task type to keep motivation and engagement in the tasks

Validation criteria

C1: need to deliver the recommendation C2: recommendation content suitability C3: timely delivery of the recommendation C4: benefit of the recommendation C5: suitability of the recommendation

presentation mode

Validation results

C1 C2 C3 C4 C5

Avg Std. Avg Std. Avg Std. Avg Std. Avg Std.

R1 7,83 1,17 7,83 1,17 8,33 1,21 8,17 0,75 7,00 1,90

R2 8,17 0,75 7,83 1,17 8,17 1,17 7,33 1,21 7,50 1,38

R3 8,17 0,75 7,83 1,17 8,00 1,10 8,00 1,67 7,50 1,76

R4 8,33 0,52 8,50 0,55 7,83 1,60 8,50 0,84 8,00 1,26

R5 8,00 1,10 7,67 1,51 7,67 1,86 7,67 1,86 8,50 0,55

R6 8,67 0,52 8,17 0,75 8,17 0,98 8,67 0,52 8,00 1,10

R7 8,33 0,52 8,00 1,55 8,50 0,55 8,17 1,17 7,17 3,13

R8 7,83 1,17 8,33 0,52 8,00 1,10 8,17 0,75 8,17 0,98

R9 8,67 0,52 8,50 0,55 8,33 1,21 8,50 0,84 8,33 1,03

R10 8,00 1,55 8,17 1,17 8,17 1,17 8,17 1,17 8,17 0,98

Selecting a Rec. for the pilot

R2 appropriate to provide personalized support while participants are carrying out the mathematical self-assessment tasks proposed. situations to be encounter ed: focusing attention on relevant data,

reviewing mistakes and results, analyzing the information provided in the problem wording, avoiding a lack of motivation as consequence of wrong results, etc.

Texts & Emotion-aware recommendation rules prepared by the 2 educational researchers involved in activity 2 adapted to the tasks particular context consider lessons learnt compiled in reviews regarding formative

feedback [Shute, 2008] and affective feedback [Girad et al., under review]

Different metacognitive strategies (2different moments) ▪ Reading and planning how to solve the problem ▪ Reviewing the results obtained

Texts proposed for R2

Text-A: “Some of the exercises can be a bit confusing. Thus, you should read the wording in detail”

Text-B: “Take your time and read the different alternative options in detail to solve the exercise”

Text-C: “Focusing is very important to solve mathematical tasks. If you focus on the wording and the options, you will solve it”

Text-D: “Don’t worry about the results obtained so far. The most important is to keep motivated to try to solve the next ones the best you can”

Text-E: “We learn from our mistakes. Thus, if you review in which issues you have failed, this will help you to do better next time”

Some Emotion-aware Recommenation rules (R2)

Rule 1: If there are 2 wrong responses in a row, deliver Text-A

Rule 2: If the learners’ face shows confusion when reading the wording of an exercise, deliver Text-A

Rule 3: If the learner cannot decide which option to select and moves from one to another, deliver Text-B

Rule 5: If the learner is distracted, looks away the screen, deliver Text-C

Rule 6: If the learner increase her facial and body movements’ rate, deliver Text-C

Rec. delivery in the pilot

Works on Emotions Detection & ERS

2012: Experimental design for collecting affective data at Madrid Science Week in 2012: 75 participants

2013: Machine learning techniques in an incremental way, considering a subset of the collected input sources.

2014: Detailed analysis on keyboard and mouse features

Related papers:

SWJ´14, SCP’14, SWJ´14, AIED’13, HCI’13, UMAP’13-14, EDM’13-14, SCP14, ERS&T12, IJAIT’13, ESWA11, EXSY13, IJWBC12, UMUA’11, … IJAIED’16

Springer volume on Recommender Systems for Technology Enhanced Learning (13)

49

ITS UV-UNED

aDeNu-UNED 2015 ©

Experiment

Real context

High School

14-year old students

Using the high school’s computers room

Using an ITS

Teaching the resolution of story problems in an arithmetic way

6 problems were proposed

AIED 2015 22-29 June 2015 Madrid 51

Experiment

Data collected Physiological data

Video data

Interaction data

Task Data

AIED 2015 22-29 June 2015 Madrid 52

ITS: Tipos de problemas (I)

Uno de los mayores problemas al aprender álgebra es la traducción de los problemas del mundo real a notación simbólica. Problemas que suelen aparecer:

El estudiante puede utilizar un número arbitrario de letras o símbolos para designar las cantidades del problema

El tipo de ecuaciones planteadas

El camino elegido para solucionar el problema

ITS: Tipos de problemas (II)

Algunos ejemplos de problemas: Luis, Juan y Roberto ganaron 960 € por pintar una casa. Debido a que no

trabajaron el mismo tiempo, Luis recibió 24 € menos que Juan y la tercera parte de lo que ganó Roberto. ¿Cuánto ganaron cada uno?

Una cesta tiene 60 piezas de fruta, entre manzanas y peras. En concreto tiene 10 veces más manzanas que peras. ¿Cuántas manzanas hay en la cesta?

El padre de Miguel es 3 veces mayor que Miguel. Hace 4 años era 4 veces mayor. ¿Cuál es la edad de Miguel?

GUI ITS MAMIPEC

Domain-specific knowledge representation

• Allows:

• Representing all potential solutions to the problem

• Representing the current state of the resolution process

• Determining all potentially valid user actions

Ba

se

d o

n h

yp

erg

rap

hs

With semantic annotations provided in a XML file

Mike's father is 3 times as old as Mike. 4 years ago, he was 4 times

older. How old is Mike?

Ayudas personalizadas

Mensajes predefinidos para

los errores más comunes y

frecuentes Mensajes Adhoc que usan

anotación semántica

Ayudas personalizadas (solicitadas)

Anotaciones semánticas usadas para

producir ayudas bajo demanda

Experiment

Data collected

Affective data

▪ At the beginning of the experiment ▪ Attributional Achievement Motivation Scale

▪ At the end of each exercise ▪ Self Assessment Manikin scale

▪ At the end of the problem series ▪ a descriptive self-report

▪ At the end of the experiment ▪ A recorded visualization of the experiment was made by the

student with a psychologist who had followed the experiment

AIED 2015 22-29 June 2015 Madrid 59

Facial expressions and body movement detection

AU3. Brow Lowerer

Head pose . Roll

User Task Time Duration Location Movement type Comments Affective State

act2usr1ses10d09

m11 T2 0:10:27 5 sec Brow furrow brow

reasoning about why he failed a problem

Confused

,,,

act2usr2ses10d09

m11 T3 0:10:50 8 sec Head

Tilt head to one side

rereading the problem statement

Concentrated

Ejemplo

Operaciones

Cantidades para operar

Explicación

de lo que hago

Operar

Ayuda

Enunciado del Problema a resolver

Operaciones

Cantidades para operar

Explicación

de lo que hago

Operar

Ayuda

Enunciado del Problema a resolver

Operaciones

Cantidades para operar

Explicación

de lo que hago

Operar

Ayuda

Enunciado del Problema a resolver

Resultado de la

operación

realizada

Se añade en la lista de cantidades para poder utilizarse

Ayuda

Enunciado del Problema a resolver

Valencia Valora si te ha gustado el problema

Activación Da igual si te ha gustado o no. Valora cuánta energía has puesto para resolver el problema

Objective

Improve emotion detection

Following a multimodal approach

Using non-intrusive devices

Exploring new approaches

AIED 2015 22-29 June 2015 Madrid 70

Exploring new approaches

Current methods:

▪ Try to classify the emotion directly

A two-class classifier to identify relevant time slots

▪ Adding a new layer ▪ A simpler initial problem (2 values to predict VS a list of emotions)

▪ Using that information for the generating a more balanced dataset and more detailed predictions.

AIED 2015 22-29 June 2015 Madrid 71

2-step emotion detection approach

Emotion? No

Emotion? Yes

Emotion? Yes

Boredom

Engageme

nt

First

Step

Second

Step

AIED 2015 22-29 June 2015 Madrid 72

Data preparation

Video

Synchronizing Facial and desktop videos

▪ During the experiment

▪ Reviewing the experiment with the students

Movement labelling

▪ Following an already proposed methodology

▪ Taking into account:

Type of movement Body part moved Movement duration

AIED 2015 22-29 June 2015 Madrid 73

Data preparation

Physiological signals

Features had to be generated…

▪ Which time window to use?

▪ A recursive analysis was performed with different time windows (1 minute, 30 seconds, 20 seconds)

▪ Finally, the 20 second time window was used in all the data

…According to a baseline

▪ Initial baseline was discarded

▪ Some signals were still not completely stabilized

▪ Possible reactions to the experimental situation

▪ The final baseline was used as a reference for each student

AIED 2015 22-29 June 2015 Madrid 74

Data preparation

Physiological variations according to the baseline

An ANOVA was carried out, looking for significant differences between the baseline and each of the time windows.

▪ Those temporal windows (per subject and signal) significantly different from the final baseline (p <0.001) were labeled as “activated”.

AIED 2015 22-29 June 2015 Madrid 75

Features used

Features generated for every 20 seconds time window: Physiological signal significance flag.

Number of Significant physiological signals.

Number of wrong actions.

Number hints requested.

Movement indicators: ▪ Body part moved.

▪ Type of movement.

▪ Movement duration

AIED 2015 22-29 June 2015 Madrid 76

Emotional labeling

We have the SAM scores for each problem

An expert labeled the emotions seen in the videos following a categorical approach

▪ The expert’s labeling was supported by the emotional reports and experiment review done by the student.

AIED 2015 22-29 June 2015 Madrid 77

Results

Emotion indicator 2-step approach Traditional approach

Boredom

Frustration

Boredom

Frustration

None

Less than 60%

accuracy

Kappa: 0.31

Emotion

? No

62.6% accuracy

Kappa: 0.31

Emotion? Yes

Emotion? No

Emotion? Yes

74.8% accuracy

Kappa: 0.49

Emotion?

Yes

Emotion?

Yes

AIED 2015 22-29 June 2015 Madrid 78

Recommendations textual

Responding to emotions

Open issues in detecting emotions do NOT stop research of affective interventions

Runtime access to signals collected

Rapid prototyping

Open hardware

Goal

Identify Ambient Intelligent Recommendations that provide

Interactive Context-Aware Affective Educational support

Publication:

Santos, O.C., Saneiro, M., Rodriguez-Sanchez, M.C. and Boticario,

J.G. (2015)

Towards Interactive Context-Aware Affective Educational

Recommendations in Computer Assisted Language Learning.

New Review of Hypermedia and Multimedia, in press.

Responding to emotions

Open issues in detecting emotions do NOT stop research of affective interventions

Wizard of Oz: human detects user behaviour & reacts

(simulates system response)

Eliciting AmI Recommendations

Educational scenario:

Preparing for the oral examination in a foreign language learning course

TORMES Methodology Santos & Boticario, 2015

Computers & Education, vol.

81

Recommendation:

Suggest the learner to breathe slowly to calm her down when nervious

▪ without interrupting her activity

Detecting emotions with Arduino

E-Health Platform:

• Pulse

• GSR

• T

• ECG

• Airflow

+ Adaptation &

Integration of

a piezoelectric

breathe belt

Integrating piezoelectric belt

Deciding affective response (rec)

Rule-based approach

weighted physiological states: Current vs. Baseline ▪ if HR_now > 20%HR_bl trigger Rec1

Wizard of Oz decides ▪ physiological signals

▪ participant’s facial & body movements

▪ task progress

▪ …

Open issues

1. How to deliver the recommendations

2. When to provide recommendations

3. Learners’ features

4. Social aspects (when collaboration)

Multisensorial actuation channels

First try (unary): on relax

1 blue led (lights)

1 buzzer (sounds & ‘vibrates’)

Second try: modulate breathing

Acompany learner’s desired breathing behaviour

Fixed rate:

▪ 4 breathes/minute (inhalation/exhalation = 7.5 segs)

Dynamic rate:

▪ Relative to breathing at Baseline (or other signal)

Second try: modulate breathing

Approaches considered per sensorial channel

Flashlights (-)

Array of leds

Ambient light (+)

Speaker Vibrator (+)

Second try: modulate breathing I

On-Off (binary) till user relaxed or stoped

on for 7.5 segs inhalate

off for 7.5 segs exhalate

Speaker (pure tone 440Hz = ‘la’) 2 flashlights (red/white) Array of leds

(intensity manually controlled)

Second try: modulate breathing II

Progressive Signal increases for 7.5 segs inhalate

Signal stops hold breath

Signal decreases for 7.5 segs exhalate

Array leds (12 blue) Speaker: cromatic musical scale 1/8 (‘la’’la’)

On going: dynamic rate modulation depends on BL

On-going works

Ambient light

Sourrounding light that changes of colour

▪ Green: calm

▪ Red: stress

▪ Blue: measuring baseline

▪ White: reporting emotional state

Moves from Green to Red

On-going works

Vibrator (touch sense)

Alerts before the rec is necessary

On skin / table / chair

e.g., extend an intelligent

cushion developed by R. Barba

in his Final Career project to

detect user’s movements on the

chair → IJDSN (Accepted)

Pilot studies

first try tell

unary (on= relax) second try modulate

binary (on-off)

Pilot 1: 6 participants (1 blind) Pilot 2: 4 participants

Evaluation approach

System Usability Scale (Brooke, 1996)

Ad-hoc questions: Q1: Did you feel relaxed during the experience? Q2: Do you consider that the recommendations provided

during the experience had an impact on your performance?

Q3: Do you think that the recommendations had been provided at the right time?

Q4: Do you consider that the recommendation format is appropriate? Would you prefer any other alternative format?

Q5: Do you consider that the recommendation was effective in modifying or improving your performance?

Evaluation outcomes

SUS system not usable (agree!) Unary approach Light hardly perceived

Sound too strong (distracts / warning) Binary approach Array of leds visible

Progressive (light) more intuitive Placement is important Keyboard vs. top of the screen

Alternative formats? Textual / Speach Posibility of selecting the channel

learner

features

recs

selected

Services in the architecture

data

processed

device

capabilities

learner

features

device

data

learner

data

request

recommendation

with context data

Emotional

Data

Processor

Multimodal

Emotional

Detector

SAERS

server

Emotional

Delivery

Component

SAERS

admin

Recs

modelled

TORMES methodology

User

Model

Device

Model

environment

data

collected

emotions

detected

recs

recs

affective

personalized

educational

recommendations

sense of arrows:

Initiator of info flow

Learner accessing the e-learning system with a certain device

in an environment with Sensors and Actuators

Discussion

AIED 2015 22-29 June 2015 Madrid

Importance of labeling on the results Who labels the emotions

▪ User itself

▪ External viewer

How the emotions are labeled ▪ Numeric scale

▪ Categorical values ▪ Which values are shown?

▪ Free labeling ▪ Which machine learning techniques can be used?

What time window length should be considered for emotion detection?

102

Discussion

AIED 2015 22-29 June 2015 Madrid

Emotions not only depend on what you are doing, the context around you also may affect:

▪ Music, noise, etc

▪ Light conditions

▪ People around

▪ Other things you migh have in mind

103

Discussion

Computational costs 2 different prediction problems

▪ The second one with a smaller dataset

Importance of FP and FN in the first step prediction FP (Predicting an emotional change when there is no

emotion): ▪ System reaction may change the student’s flow (if the state

was the right one).

FN: (Not predicting emotions when there are some affective changes): ▪ Being late on reacting to negative state changes.

AIED 2015 22-29 June 2015 Madrid 104

Discussion

AIED 2015 22-29 June 2015 Madrid

Data sources dependencies

Task to be solved

User interface & skill

User special needs

105

Trabajos Futuros / Diseminación

Áreas de trabajo Colegios / laboratorios

Algunos artículos relacionados: C&E’15, NRHM’15, IJDSN’15, EDM’15, AIED’15, UMAP -PALE’15, SWJ´14, SCP’14, SWJ´14, AIED’13, HCI’13, UMAP’13-14, EDM’13-14, SCP14,, ERS&T12, IJAIT’13, ESWA11, EXSY13, IJWBC12, UMUA’11, … IJAIED’16 Springer volume on Recommender Systems for

Technology Enhanced Learning (13)

Experiencias: intra- / inter-sujeto colegios / lab

References & Acknowledgements

aDeNu research group https://adenu.ia.uned.es

Background related projects https://adenu.ia.uned.es/web/projects o European:

aLFanet, ALPE, EU4ALL, ALTER-NATIVA

o National: FAA, CISVI, AMI4INCLUSION, ATODOS, A2UN@,

ADAPTAPlan, MAMIPEC, BIG-AFF ‘