BayCHI Speaker Series

April 14, 2015

Nilo Sarraf

Towards Smart Emotional Neuro Search Engines: An Extension of the Human Brain

PhD Gateway Program

SJSU & QUT School of Information

Disclaimer

❖ This presentation is an overview of my ‘in-progress’ PhD

dissertation (Third year)

❖ I am not a neuroscientist but have research background in

neuroscience and HCI

❖ My PhD dissertation work is NOT affiliated with the

company where I work

❖ This deck was presented at the BayCHI April 2015

speaker series: http://www.baychi.org/calendar/20150414/

2

Research Background & Interests

❖ Research at Stanford University - Neuroscience

❖ Research in the industry - User Experience Research

❖ Passion for Neuroscience and Information Retrieval

❖ Positioning in ‘Neuro Information Science’

❖ Neurophysiological methods in Information Science

❖ Wearable Computing

❖ Artificial Neural Networks

3

Two Parts of this Presentation

1.Overview of my doctoral dissertation (in-progress)

❖ Positioning in Neuro Information Science (Gwizdka, 2012)

❖ “How does Affect Impact Search Performance?”

2.Proposal to the industry

❖ Improve search results based human neurological

feedback through wearable computing devices

❖ Artificial Neural Networks architecture improvements by

adding human emotion data input

Dissertation Overview

Introduction

❖ Purpose: Examine the potential effects of emotions on

information retrieval, as revealed in search processes

❖ Focus: Examine the effects of emotions on search

performance, in terms of search effectiveness and search

efficiency

❖ Conclusions will be drawn about the effects, if any, of

different dimensions of emotions and their effects on

search performance

6

Affect in Design

❖Affective component of information retrieval system

design is becoming increasingly essential

❖Expressions such as “pleasurable engineering” or

“emotional design” have become the driving factors

in system design and these expressions have also

been extended to information retrieval system design

(Nahl & Bilal, 2007)

Historical Evolution of Information Science Research

❖ System Oriented Approach

❖ Precision and Recall

❖ User Oriented Approach

❖ User behavior

❖ Cognitive Oriented Approach

❖ Affect Oriented Approach

8

Research Questions

❖ The gap: The effects of physiological and neurological emotion

responses in information retrieval, more specifically on web search

❖ Two dry runs and two pilot studies

❖ Q1: How do dimensions of emotions affect search effectiveness?

❖ Q2: How do dimensions of emotions affect search efficiency?

❖ Q3: Are there any interactional effects between dimensions of

emotions and search performance?

❖ The hypothesis is that positive emotional states have positive effects

on information retrieval and negative emotional states affect users’

web search performance negatively

9

Human Emotions

Theoretical Models and Framework

❖Emotion in Information Seeking – ISP Model (Kuhlthau (1991)

❖Six steps of the ISP model: Initiation, Selection, Exploration,

Formulation, Collection, Presentation

❖Emotion in Information Processing – CPM Model (Scherer,

2001)

❖Emotions elicit as individuals evaluate continuous events,

objects, or situations

❖Evaluations with respect to the effect they may have on

individual’s goals

11

Structures of Emotions

❖ Discrete

❖ Darwin, the father of the discrete approach, claimed that there exist six basic

emotions: fear, happiness, surprise, anger, sadness, and disgust (Darwin, 1872;

Ekman, 1992)

❖ Continuous

❖ Addresses different ‘dimensions’ of emotions (Russel & Mehrabian, 1977; Russel,

1994). These theorists state that there are two dimensions of emotions, valence and

arousal (Russell, 1994; Russell & Mehrabian, 1977; Russell & Steiger, 1982; Barrett

& Russell, 1999)

❖ Valence: indicates the positivity versus the negativity of an emotion, ranging from

highly positive to negative states

❖ Arousal: measures the calmness versus the excitement of an emotion, ranging

from calming to exciting (or agitating) states

12

Emotional Dimensions Associated with Brain Waves

❖ Russel’s (1989) research shows that the following two emotional dimensions

are associated with various brain waves:

❖ Theta waves, also seen in meditative states (Cahn & Polich, 2006), show

arousal or drowsiness in adults

❖ Alpha waves are exhibited when closing the eyes and during relaxation

❖ Beta waves, linked with motor behavior, occur when the individual is actively

moving (Pfurtscheller and da Silva, 1999)

❖ Low beta frequencies are often associated with concentration and/or active

thinking

❖ Gamma waves represent cognitive or motor functions (Niedermeyer & da

Silva, 2004)

14

Emotiv EPOC Neuro Headset

Research Design

❖Experimental Research Design

❖Two-Way Repeated Measures ANOVA

❖Independent Variable (IV) - Emotional State

❖Three sets of 20 IAPS pictures at neutral, pleasant, and unpleasant level

❖Brain Activities: Alpha and Beta waves

❖SAM Self-Report: Valence and Arousal

❖Dependent Variable (DV) - Search Performance

❖Time on Task

❖Task Completion

❖Number of Search Queries

❖Number of Web Pages

17

EEG Data Analysis

❖Careful data processing and analysis must be done when collecting EEG raw data sets

❖To analyze EEG raw data:

❖Pass each channel through a high/low pass filter

❖Perform a transform on the data

❖Filter for a key frequency band

1.Preprocessing: The raw EEG data usually is not clean and some preprocessing steps are needed:

❖ Applying high-pass and low-pass filters

❖ High-pass filter: Removes the low frequencies

❖ Low-pass filter: Removes the high frequency brain waves

2.Feature Extraction: Divide the signals in chunks of time in order to extract features out of each one of

these pieces

❖ MatLab has many functions for filtering these signals where one could set band pass filters

❖ E.g. alpha waves are between 8Hz and 12Hz

❖ EEGLab

18

EEG Data Analysis: Recognizing Dimensions of Emotions

❖In recognizing the dimensions of emotions (Valence &

Arousal) through EEG devices studies appear to suggest

that:

❖Valence - Alpha asymmetry (present on states of low

cognitive load) on the frontal lobes

❖Arousal - The ratio of beta-alpha waves (present on

states of high cognitive load) on the frontal lobes

19

Challenges & Limitations

❖ Some of the challenges and limitations when it comes to

the proposed model for developing Affective Neuro

Search

❖ The complex human motor movements may contribute

to ‘noise’ when it comes to reading the brain signals

❖ The EEG devices in the market today may not be able

to fully suppress all the noise emanating from major

body movements, such as head/hand movements

21

Proposal to the Industry

Designers & Architects

Problem Statement

❖The existing Artificial Neural Networks architectures are based

solely on digital data input

❖System programmers and architectures fail to approach

modeling the human brain holistically

❖ The main component, human emotion, is missing from this

equation

❖I propose that adding one additional data input of human

emotion may improve these artificial neural networks

23

Why Ponder on These Issues?

❖This is the era of brain-controlled devices

❖This is the era of physical connection/collaboration between the

human brain and physical devices

❖ In an era when humans are creating brain controlled airplanes,

neuro-gaming, and robots that learn behavior by reading human

emotions, there appear to be no limits in having search engines

read human emotions in order to improve search results based

on the neurological feedback they receive from brain waves

24

Research Contribution

❖ One of the main contributions of my

dissertation is my proposal to include human

emotions readings via wearable computing

devices as an additional data input for

statistical learning algorithms when creating

artificial neural networks

27

Smart Emotional Neuro Search Engines

❖ I envision my dissertation contribute to the body of knowledge of Neuro

Information Science in developing search engines that, through wearable

computing devices that are able to read brain waves and dimensions of

emotions in order to improve search results based on the neurological

feedback that the search engines receive from brain waves

❖ Search engines become an extension of the human brain by receiving

brain waves that constantly provide neurological feedback in terms of the

search results that they provide

❖ Search engine reads brain waves by receiving the brain signals through

wearable computing devices and ‘learn to improve’

28

Thank You!

nilosarraf.com

@nilosarraf