THE FULL YOU!Beyond Self-Reporting….Polymeasures: from Behavioural Genetics, Biomarketing and Human-Computer Interaction to Neurophysiology and Neuroscience.

Professor Luiz Moutinho

Foundation Chair of Marketing

Adam Smith Business School

University of Glasgow,

Scotland

Self-reporting measures (e.g.,survey data via questionnaires )have numerous limitations, such as triggering a rational tendency in respondents and discouraging spontaneous responses (Hupp et al. 2008). Several researchers (e.g., Oatley 1992; LeDoux 1996; Babin et al. 1998; Bagozzi et al. 1999; Lee et al.,2007;) affirmed that the measurement of emotions must supersede self-reporting measures and call for collaboration with other research fields in order to move emotion and social science research forward.

Researchers are making great efforts to give computers affective perceptual abilities and design new instruments to measure physiology and behaviour. Academic research on human-computer interaction (HCI) combines the experimental methods and intellectual framework of cognitive psychology with effective tools from computer science. It emphasizes controlled experiments which yield more objective and reliable results.

PolymeasuresBio-Physio - Biological Measures +

Physiological measures

Experience - Sampling Measures (ESM)

Implicit/ Indirect Measures

Self - Reporting Measures

Lab-Mechanical ObservationNeuroscience

Self reporting of mental processesValidity-Correlation ESM Measures with physio measures, psychological tests and Behavioural indices

Functional Properties?Objective Property of the Measurement?- Not self assessment, but another behaviour.- They do not allow to register Stable Structures

in, but, they can provide unique insight into effects of automatic processing on real-life behaviour. Mood, Emotion, Voice Prints, …. Facial Electromyography

Concerns about validity of causal conclusions - systematic response distortions. Method variance, monomethod bias, and the psychometric properties (reliability + validity) of questionnaire scales. Context-design of studies, statistical analyses.

Biological Measures (Biometrics) – Mechanical Observation

Facial ElectroMyography (EMG),Facial Coding, Iris Recognition, Retinal Scan, Fingerprint recognition, Handwritten biometrics, Voice Prints, Voice Pitch Analysis, Utterance Analysis, Human-Computer Interaction, Vein Matching, Palm Vein Prints, Palm Dorsa, Scan Brain Waves, Pupillometrics, Heart Beat, Degree of Sweating, Glasses Mounted Cameras, Telemetric Measures for Electrodermal Activity,…..

Physiological Measures – Neuroscience (1)

(participant cannot change /L about response)

BRAIN PHYSIOLOGY• EEG – Electroencephalography – records electrical activity along the scalp. Voltage

fluctuations from ionic current flows within neurons of the brain. Neural oscillations.

• PET – Positron emission tomography – 3D image/picture of functional processes in the body. Pairs of Gamma Rays. Tracer Concentration.

• MRI – or nuclear (NMRI) or MRT (tomography) – to visualise internal structures of the body. Uses magnetic field gradients that allow spatial information.

• fMRI – measures brain activity by detecting associated changes in blood flow. BOLD/contrasts (hemodynamic response).

• TMS – Depolarisation or hyperpolarisation in the neurons of the brain. Uses electromagnetic induction with weak electric currents. Using a rapidly changing magnetic field.

Brain Physiology

Physiological Measures – Neuroscience (2)

tDCS – form of neurostimulation which uses constant, low current directly to areas of the brain via small electrodes. Can increase cognitive performance.

fNIRS – DOT (Functional Near – Infrared Spectroscopy) –( Diffuse Optimal Tomography). Advantages- used outside hospital environments, allows interaction with participants, less stringent in selection of participants. Disadvantages – only reaches surface cortex and not possible to quest all the brain simultaneously.

Brain Physiology

Complementarity

Physio data is unlikely to capture the full experience of emotion. Physio data is continuous and can be measured without the conscious introspection required for self-reporting. The big problem with conscious introspection about emotion is that is divorced from the way in which we tend to experience emotion in real life. Physio data is objective, can be reliably measured. Physio markers-oxytocin, tuf-alpha, heart rate variability, galvanic skin response, dopamine,….

In the last decade, scientists have found evidence showing the importance of emotion in intelligent human functioning (Picard 2003). In controlled experiments, research on human-computer interaction (HCI) combines the experimental methods and intellectual framework of cognitive psychology with effective tools from computer science. Voice recognition is a research stream within the HCI discipline that conducts computational studies of emotion in speech. Most of our studies in the area of HCI utilize the emotion recognition architecture based on speech processing and pattern recognition methods in emotional space theory (Pao et al. 2008).

Signal-based evaluation tools address some of the restrictions of the self-report measures. For instance, it is possible to capture and analyze speech signals of advertising slogans and elicit emotions from the signal data. This is a more natural way of measuring emotions than analyzing the recalled data from self-reporting measures.

The purpose of the Voice Emotion Response is to give computers affect recognition abilities, ideally at a level which enables researchers to label the emotional states of other people. However, researchers meet complex problems when they attempt to teach computers how to do this, and the complexities involved could be enormous. Hence, only partial solutions may be obtained, but nevertheless, such partial solutions can still be of value.

Researchers in the Department of Computer Science and Engineering at Tatung University, Taiwan, developed a user interface (also known as a human-computer interface)—the Voice Emotion Response to classify emotion. This includes short utterances covering 12 full-blown emotions in Mandarin speech: happiness, anger, sadness, boredom, and neutrality (unemotional) (Murray and Arnott 1993). The term Bfull-blown emotion^ is used to describe fully-developed emotional expression, which is typically impassioned, and has inherent factors considered relevant for emotional expressions. When Bemotion^ is used in this sense, a positive or negative orientation can be conveyed to others (Plutchik 1994).

The Mandarin Chinese Emotional Corpus consists of 19,136 short utterances recorded by voice actors/actresses and 472,017 emotional annotations. Speech features, such as jitter, shimmer, 16 linear predictive coefficients (LPCs), 20 mel-frequency cepstral coefficients (MFCCs), 12 log frequency power coefficients (LFPCs), 14 perceptual linear predictions (PLPs), etc., are applied to decide the k value with WD-KNN (K-nearest neighbour) in all utterances. The Fibonacci weighting series is adapted from WD-KNN (Pao et al. 2008), developed by a team from Tatung University. The weighted D-KNN (distance K-NN), which is a classification algorithm, evaluates the emotions in the speech data, achieving 81.4 % accuracy (Pao et al. 2008).

Development of the Slogan Validator .Emotional speech corpusIn the initial stage, an emotional corpus needs to be built in order to form a

base for eliciting emotions from speech signals. This exercise yielded a corpus of 3,400 utterances. Human judges then evaluated this preliminary corpus, filtering out ambiguous emotional utterances for further recognition analysis. The next table shows the performance confusion matrix of the judges. The rows and columns represent the simulated and the evaluated categories, respectively. For example, the first row illustrates that the judges evaluated 89.6% of the angry utterances as truly angry, 4.3% as truly happy, 0.9% as sad, 0.8% as bored, 3.5% as neutral, and 0.9% as none of the above.

Angry Happy Sad Bored Neutral Others

Angry 89.6 4.3 0.9 0.8 3.5 0.9

Happy 6.7 73.2 3.3 2.4 13.6 0.9

Sad 2.9 1.0 82.8 9.3 3.3 0.7

Bored 1.3 0.4 8.6 75.2 13.7 0.9

Neutral 1.7 0.9 1.6 12.3 83.5 0.1

Table -Human Judge’s Performance Confusion Matrix.

The core architecture of this interface was developed through a number of stages of computer engineering. In the first stage, pre-processing, the interface locates the endpoints of the input speech signal and high-pass filters the speech signal, which is then partitioned into pieces of frame. The Hamming window is applied to each frame, minimizing the signal discontinuities at both the beginning and the end of each frame, and converting the frames into several types of parametric representations.

The next stage is the speech feature extraction, arguably the most challenging issue when building an emotion recognition system for speech signals (Petrushin 2002; Schuller et al. 2003). The regression selection extracts possible candidates from more than 200 speech features in an attempt to discover a suitable combination of features. Feature extraction methods include MFCC-Mel Frequency and LPCC-Linear Prediction Cepstral Coefficients. The feature vector quantization stage processes occur when 20 MFCCs and 12 LPCCs of each speech frame extract the parameters of each utterance as a feature vector. A vector quantization method (Pao et al. 2005) obtains the mean of the feature parameters corresponding to each frame in one utterance. The weighted D-KNN (Distance K-NN) (Pao et al. 2005), which is a classification algorithm, evaluates the emotions in the speech data.

A participant spoke out Volvo slogan for the third time

NEUROSCIENCEFunctional Magnetic Resonance Imaging (fMRI):

Advantages:- non invasive;- ethically acceptable in healthy participants;- good spatial definition;- more than 15 years of neuropsychological functional studies.

Functional Magnetic Resonance Imaging (fMRI):

Disadvantages:- noisy;- hospital environment;- participant limiting (claustrophobia, metal particles in the body, metallic ornaments, psychotropic drugs, etc.);- limited interaction with participants.

Functional Magnetic Resonance Imaging (fMRI) signal:

Thus, absolute paradigms are not allowed (only relative questions).

Low firing rate

High firing rate

Difference

WHY fMRI?BOLD SIGNALTHE BASELINEPARADIGMSGLMANNs

Very important issues in fMRI studies:

- absolute paradigms are not allowed in common studies (only relative questions);

- BOLD signal is an indirect measure of brain activity; fMRI measures disturbances in the magnetic field caused by differential ratios in hemoglobin / oxyhemoglobin caused by requirement of oxygen caused by increased neuronal spiking rate caused by cognitive recruitment.

The basis of Blood-Oxygen-Level Dependent (BOLD) signal:

Pixels & Voxels

Voxels = pixels with thickness

Z = -0.20

Z = -3.59

Z = +4.49

Nothing52% postcentral gyrus

Deactivation47% precuneous cortex

Activation57% paracingulate gyrus

These statistics allow to emphasize the voxels that most follow the paradigm course according to a established criterion. In this case:

- activation: z > 2.3- deactivation: z < -2.3

+ better signal in fMRI / simplicity- induce strategies in subjects

Stimulus

30 seconds

Stimulus

30 seconds

Baseline

30 seconds

Stimulus

30 seconds

Baseline

30 seconds

WHY fMRI?BOLD SIGNALTHE BASELINEPARADIGMSGLMANNs

Paradigm: block design

Represent the sense of the physiological condition of the body: INTEROCEPTION(Craig, 2002)

The neural correlates of empathy are the anterior Insular Cortex and the Frontal Operculum Cortex.(Singer et al., 2004)

Empathy: is the ability to represent other's feelings or experiences by imagining how it would be in their situation.

z =

-08

(MN

I152

coo

rd.)

R

anterior Insular CortexFrontal Operculum

2.3 5.3

(z > 2.3; p = 0.05)

Research questionsStep 1 – Reading brands Research design FindingsStep 2 – Positive vs. Indifferent Comparison with Step 1 Preferring positive brandsStep 3 – Introducing Non-brands Comparing with Step 2 Emotion & Social Relevance Extras from Step 3 study

The model of Emotions:

vmPFC

Amy

Effectors

Primary inducers

Secondary inducers

BODY

Body Loop Brainstem

As If Body Loop

Insula/SII, SI

(Damásio, 1994; Bechara, 2004; Bechara & Damásio, 2005)

z =

-08

(MN

I152

coo

rd.)

R

anterior Insular CortexFrontal Operculum

2.3 5.3

(z > 2.3; p = 0.05)

Feeling

Research questionsStep 1 – Reading brands Research design FindingsStep 2 – Positive vs. Indifferent Comparison with Step 1 Preferring positive brandsStep 3 – Introducing Non-brands Comparing with Step 2 Emotion & Social Relevance Extras from Step 3 study

z =

+36

(MN

I152

coo

rd.)

Paracingulate Gyrus

2.3 5.3

(z > 2.3; p = 0.05)

x =

-08

(MN

I152

coo

rd.)

P

The Paracingulate Gyrus is thought to have a relevant role in Theory of Mind, mentalizing, meta-representations and reflexive meta-representations.

(Gallagher & Frith, 2003; Rilling et al., 2004; Amodio & Frith, 2006; Saxe, 2006; Frith, 2007)

Research questionsStep 1 – Reading brands Research design FindingsStep 2 – Positive vs. Indifferent Comparison with Step 1 Preferring positive brandsStep 3 – Introducing Non-brands Comparing with Step 2 Emotion & Social Relevance Extras from Step 3 study

At this point, fMRI revealed that assorted brands have:

R

R

- neural correlates with the Emotional system by the assessment carried out by the Insula and the Frontal Operculum Cortex (extraction of the Feeling);

- neural correlates with the Social system by the meta-representational work carried by the Paracingulate Gyrus.

Research questionsStep 1 – Reading brands Research design FindingsStep 2 – Positive vs. Indifferent Comparison with Step 1 Preferring positive brandsStep 3 – Introducing Non-brands Comparing with Step 2 Emotion & Social Relevance Extras from Step 3 study

+ reduce strategies in subjects (psychologically more robust) may improve temporal resolution- lower signal in fMRI (prone to noise) increased complexity in data analysis

Baseline

5 seconds

WHY fMRI?BOLD SIGNALTHE BASELINEPARADIGMSGLMANNs

Paradigm: event related – jittered ISI

+++

++

+ + + …

Baseline Baseline Baseline Baseline

5 seconds 2 seconds 5 seconds 7 seconds 5 seconds 4 seconds 5 seconds 5 seconds

Jittered ISI (interstimuli interval)

R

Activations in the Paracingulate Gyrus (extraction of social relevant meanings):

R

Positive Indifferent

z =

+36

(MN

I152

coo

rd.)

2.3 5.3

(z > 2.3; p = 0.05)

Step 1

x =

-08

(MN

I152

coo

rd.)

P

R

Conjunction

P

R

PP

Research questionsStep 1 – Reading brands Research design FindingsStep 2 – Positive vs. Indifferent Comparison with Step 1 Preferring positive brandsStep 3 – Introducing Non-brands Comparing with Step 2 Emotion & Social Relevance Extras from Step 3 study

Very similar activation pattern

PPP

Positive > NEW Indifferent > NEW Conjunction

x = -08(MNI152 coord.)

2.3 5.3

(z > 2.3; p = 0.05)

Step 2Mixed designThinking on the rates

Step 3Event-related designRates with a button box

Comparing Step 2 and Step 3 studies

P P P

Research questionsStep 1 – Reading brands Research design FindingsStep 2 – Positive vs. Indifferent Comparison with Step 1 Preferring positive brandsStep 3 – Introducing Non-brands Comparing with Step 2 Emotion & Social Relevance Extras from Step 3 study

Activations in the Paracingulate Gyrus (extraction of social relevant meanings)

- frontal pole: self-reflexive processes;

- paracingulate gyrus: meta-representations (thinking about what others are thinking);

- posterior cingulate & precuneous: autobiographical memories and self-referred experiences;

- angular & supramarginal gyri: understanding social intentions, beliefs, and goals; mentalizing.

the same network that humans use to navigate in the social environment is also active during meaningful brands appraisal

(Lou et al., 2004; Jackson et al.,2006; Volz et al., 2009)

WHAT IS A BRAND?FIRST APPROACHSECOND STEPTHIRD ITERATIONTHEORETICAL MODEL‘MIND READING’

The Amygdala activated for Positive brands:z

= -2

0(M

NI1

52 c

oord

.)

2.3 5.3

(z > 2.3; p = 0.05)

y =

-04

(MN

I152

coo

rd.)

R

R

R

R

Positive > NEW Positive > Indifferent

The Amygdala is a primary emotional inducer.

(Damásio,1994; Adolphs, Tranel & Damásio,1998; Bechara et al., 1999; Adolphs, 2003; Bechara, 2004; Bechara & Damásio, 2005)

Research questionsStep 1 – Reading brands Research design FindingsStep 2 – Positive vs. Indifferent Comparison with Step 1 Preferring positive brandsStep 3 – Introducing Non-brands Comparing with Step 2 Emotion & Social Relevance Extras from Step 3 study

fNIRS-DOT (Functional Near InfraRed Spectroscopy – Diffuse Optical Tomography)

Next step:

Disadvantages- not possible to quest all the brain simultaneously;

- only reaches surface cortex.

Advantages- possible outside the hospital;- allows interaction with the participants;

- allows more participants (no noise, nor claustrophobia, etc.).

FINAL REMARKS

TMS (Transcranial Magnetic Stimulation)Next step:

- allows “virtual lesions”, i.e. temporary blocking of selected brain structures, which allows challenging the role of such structures in the process.

FINAL REMARKS

We PERCEIVE the SMILE because we read the pixels’ distribution pattern and NOT because we analyze each pixel individually (univariate approach)

Solution: multivariate analysis, e.g. ANN (artificial neural networks), tensor-PICA (tensor probabilistic independent component analysis), random trees, SVM (support vector machine)…

WHAT IS A BRAND?FIRST APPROACHSECOND STEPTHIRD ITERATIONTHEORETICAL MODEL‘MIND READING’

An example with ANNs:

Data pre-processing

Test data:

Train data:ICs’ timecoursesoutput by PICA

ICs’ maps output by PICA Binaries of ICs’ maps

Original BOLDtimecourses

Masked averages of original BOLD timecourses

WHY fMRI?BOLD SIGNALTHE BASELINEPARADIGMSGLMANNs

We used Probabilistic Independent Component Analysis (PICA) in a data set that encompasses all individuals’ data sets concatenated.

In the end of this stage we had 161 independent components (ICs).

It is assumed that the voxels inside the same IC have similar timecourses.

WHY fMRI?BOLD SIGNALTHE BASELINEPARADIGMSGLMANNs

An example with ANNs:

Dimensionality reduction

Application of an ANN (bootstrap n-1):

1 2 3 4fic ind new pos fic ind new pos fic ind new pos fic ind new pos

1 Pos 0,0001 0,9995 0,0001 0,0003 0,4516 0,0000 0,0001 0,5483 0,0000 0,0001 0,0001 0,9998 0,0000 0,0001 0,0000 0,99992 Ind 0,0000 0,9997 0,0002 0,0000 0,0012 0,0000 0,9962 0,0026 0,0001 0,9998 0,0000 0,0001 0,9990 0,0007 0,0001 0,00023 Fic 0,9975 0,0014 0,0010 0,0002 0,9995 0,0003 0,0002 0,0000 0,9832 0,0151 0,0011 0,0006 0,9996 0,0003 0,0001 0,00014 NEW 0,0000 0,0000 0,9998 0,0002 0,0000 0,0000 1,0000 0,0000 0,0016 0,0003 0,9981 0,0000 0,0488 0,0000 0,9458 0,0054

5 6 7 8fic ind new pos fic ind new pos fic ind new pos fic ind new pos

1 Pos 0,0004 0,0877 0,0004 0,9115 0,9958 0,0000 0,0003 0,0039 0,0000 0,0000 0,0002 0,9998 0,0000 0,0000 0,0002 0,99982 Ind 0,0011 0,9986 0,0003 0,0000 0,0020 0,0387 0,0005 0,9589 0,0075 0,9901 0,0001 0,0022 0,0000 0,0018 0,0002 0,99793 Fic 0,6960 0,3003 0,0037 0,0000 0,9996 0,0003 0,0000 0,0000 0,0146 0,0000 0,0013 0,9841 0,9930 0,0031 0,0037 0,00024 NEW 0,0059 0,0024 0,9917 0,0000 0,2501 0,0006 0,7493 0,0000 0,0000 0,0000 0,9995 0,0005 0,0000 0,0000 1,0000 0,0000

9 10 11 12fic ind new pos fic ind new pos fic ind new pos fic ind new pos

1 Pos 0,0000 0,0001 0,0000 0,9999 0,6645 0,3340 0,0007 0,0007 0,0000 0,0000 0,0000 0,9999 0,0045 0,0067 0,0013 0,98742 Ind 0,0019 0,6870 0,0000 0,3110 0,0391 0,9599 0,0002 0,0008 0,0011 0,0006 0,5790 0,4194 0,0000 0,9923 0,0077 0,00003 Fic 0,2840 0,7123 0,0003 0,0034 0,9977 0,0008 0,0015 0,0000 0,9996 0,0003 0,0000 0,0000 0,9992 0,0004 0,0003 0,00014 NEW 0,0000 0,0000 0,9998 0,0002 0,0008 0,0002 0,9991 0,0000 0,2245 0,0000 0,7022 0,0733 0,0000 0,0000 0,9995 0,0005

13 14 15 16fic ind new pos fic ind new pos fic ind new pos fic ind new pos

1 Pos 0,0001 0,0013 0,0001 0,9986 0,0000 0,0002 0,0000 0,9998 0,0000 0,0001 0,0001 0,9998 0,0001 0,9476 0,0006 0,05162 Ind 0,0004 0,9990 0,0001 0,0005 0,0034 0,9910 0,0002 0,0053 0,0000 1,0000 0,0000 0,0000 0,0000 0,9997 0,0000 0,00033 Fic 0,9992 0,0006 0,0002 0,0000 0,0082 0,9893 0,0003 0,0023 0,0199 0,9684 0,0012 0,0105 0,9998 0,0001 0,0001 0,00004 NEW 0,0001 0,0000 0,9998 0,0001 0,0000 0,3271 0,6381 0,0348 0,0000 0,0001 0,9407 0,0592 0,0007 0,0000 0,9993 0,0000

17 18 Totalfic ind new pos fic ind new pos fic ind new pos

1 Pos 0,0000 0,0002 0,0001 0,9997 0,0000 0,0001 0,0000 0,9999 1 Pos 2 1 0 14 77,8%2 Ind 0,0000 1,0000 0,0000 0,0000 0,0001 0,9819 0,0000 0,0180 2 Ind 1 13 2 2 72,2%3 Fic 0,9994 0,0002 0,0002 0,0001 0,9980 0,0003 0,0016 0,0001 3 Fic 14 3 0 1 77,8%4 NEW 0,0000 0,0000 0,9998 0,0002 0,0000 0,0000 0,6232 0,3768 4 NEW 0 0 18 0 100,0%

We guess brand preference better than chance

Input: 18 nodes (each one is a sphere of 81 voxels with 2 2 2 mm that represent interesting brain regions)

Hidden: 3 nodes (to allow interpretable psychological processes)Output: 3 nodes (categories positive, indifferent, and fictitious)

18 participants provided 1452 epochs, which were randomly sorted and• 80% were used in training• 20% were kept for the test stage

AMORE package implemented in R (R Development Core Team) • global learning rate: 0.01• global momentum: 0.5

WHAT IS A BRAND?FIRST APPROACHSECOND STEPTHIRD ITERATIONTHEORETICAL MODEL‘MIND READING’

Voxel 1 60% temporal occipital fusiform cortexVoxel 2 54% paracingulate gyrusVoxel 3 86% frontal poleVoxel 4 70% precuneous cortexVoxel 5 43% angular gyrusVoxel 6 69% amygdalaVoxel 7 41% central opercular cortexVoxel 8 59% postcentral gyrusVoxel 9 68% lateral occipital cortex – sup. div.Voxel 10 88% caudate

Voxel 1 Voxel 2 Voxel 3 Voxel 4 Voxel 5 Voxel 6 Voxel 7 Voxel 8 Voxel 9 Voxel 10-6.000

-4.000

-2.000

0.000

2.000

4.000

6.000

8.000

PositiveIndifferentFictitiousNEW

Application of an ANN

Voxel 1

Voxel 2

Voxel 3

Voxel 4

Voxel 5

Voxel 6

Voxel 7

Voxel 8

Voxel 9

Voxel 10

Hidden1

Hidden2

Hidden3

Hidden4

Pos

Ind

Fic

NEW

1,94

-5,53

3,43

-8,4

4

2,70

Frontal pole(self-referential processes)

-3,03

Central operculum(SII; Damasio’s feeling)

2,55

Precuneus(autobiographical memories)

Application of an ANN:

The frontal pole, precuneus, and the central operculum support a psychological process that leads to brand preference (positive), and simultaneously does not leads to indifference.