Affect in recommender systems

Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..

Affect in recommender systems

Marko TkalčičUniversity of Ljubljana


Presentation overview

I: LDOS presentation & Motivation II: What are emotions? III: Emotion in recsys – related work IV: Role of emotions in the MM consumption chain V: Affect in the decision-making stage Conclusions

Note: some material is not ours ... Fair use ...


Part I: LDOS group at UL FE and underlying assumption


LDOS group at UL FE University of Ljubljana

– Faculty of electrical engineering• LDOS (Digital signal processing laboratory)

– Approx 15 members

Relevant people

Head: prof. Jurij Tasič

Andrej Košir

Marko Tkalčič

Ante Odić

Matevž Kunaver Tomaž Požrl


LDOS work on recommender systems 2002-2009: public movie datasets

– CBR– CF

2009-2012– Emotions– Context

2012 – – Decision making (affective + cognitive attributes)

• Ajzen model• Kahneman/Tversky model• ...

Basic RecSys

AffectiveRecSys

AffectiveComputing

Decision making Modeling in RecSys


Underlying presumption of our work Recommender System = predictor of users‘ decision

making Decision making: EMOTIONS DO INFLUENCE

(c) Dilbert.com


PART II : What is affect/emotions/mood/personality


PART II : What are affect/emotions/mood/personality

Not well defined (wikipedia):

– Emotion = subjective, conscious experience

– Affect = experience of emotion (interchangable)

– Emotion vs. Mood:• Emotion = high arousal, short term• Mood = low arousal, long term

– Personality = accounts for the individual differences in the users’ emotional, interpersonal, experiential, attitudinal and motivational styles [John and Srivastava, 1999]

Time (duration)

emotion mood personality

CHANGES FIXED

Psychophysiologicalexpressions

Biologicalreactions

Mental states


Overview of emotions/moods Several definitions We take simple models, easy to incorporate in computers:

– Basic emotions– Dimensional model– Circumplex model


Basic emotions Discrete classes model Different sets Charles Darwin: Expression of emotions in man and

animal Paul Ekman definition (6 + neutral):

– Happiness– Anger– Fear– Sadness– Disgust– Surprise

(c) Paul Ekman


Dimensional model Three dimensions

– Valence (positive vs. Negative)– Arousal (high vs. Low)– Dominance (power(less) over emotions)


Dimensional model Three dimensions

– Valence– Arousal– Dominance

– (c) Lang, P. J. (1980)

Each emotive state is a point in the VAD space


Circumplex model Maps basic emotions dimensional model (Posner et al.)

Arousal

Valence

high

negative positive

low

neutral

sadness

fear

disgust

surprise

joyanger


How to detect emotions Emotions are characterized:

– psychophysiological expressions, – biological reactions – mental states

SENSORS !!!


How to detect emotions? Explicit vs. Implicit Explicit

– Questionnaires (SAM)


How to detect emotions? Explicit vs. Implicit Explicit

– Questionnaires (SAM) Implicit:

– Work done in the affective computing community– Different modalities (sources):

• Facial actions (video)• Physiological signals ( GSR, EEG)• Voice• Posture• ...

– ML techniques• Classification (basic emotions)• Regression (dimensional model)


Lie To Me Part1.avi (c) 20th Century Fox Main Character Cal Lightman = Paul Ekman

Defined the FACS(Facial Action Coding System)


LDOS Experiment 2 datasets:

– Posed (Cohn-Kanade dataset)– Spontaneous (LDOS-PerAff-1 dataset)

Input: Video streams of facial expressions as responses to visual stimuli

Output: emotive states as distinct classes

Gabor features kNN

Emotive state


Results and conclusions Posed dataset: accuracy = 92 % Spontaneous dataset: accuracy = 62% Reasons for bad results:

– Weak learning supervision– Non optimal video acquisition (face rotation, occlusions,

changing lightning ...)– Non extreme facial expressions

Upcoming paper: IEEE Transactions on Multimedia


Personality Definition:

User personality accounts for the individual differences in the users’ emotional, interpersonal, experiential, attitudinal and motivational styles [John and Srivastava, 1999]

Ever-lasting Several models

Five Factor Model (FFM or Big5): Openness (inventive/curious vs. consistent/cautious) Conscientiousness (efficient/organized vs. easy-going/careless) Extraversion (outgoing/energetic vs. solitary/reserved) Agreeableness (friendly/compassionate vs. cold/unkind) Neuroticism (sensitive/nervous vs. secure/confident)

How to measure? Questionnaires:

International Personality Item Pool ( http://ipip.ori.org/ )

http://ipip.ori.org/




LDOS PerAff-1 dataset Emotive responses Ratings Personality data Videos of facial expressions 50 users, 70 items, sparsity=0 http://slavnik.fe.uni-lj.si/markot/Main/LDOS-PerAff-1


PART III : Related work on emotions in recsys


PART III : Related work on emotions in recsys

Emotions and personality Scattered work


Emotions in recsys

Gonzalez, 2007 ? Emotions as context in recsys?

Arapakis et al., 2009User affective feedback from automatic facial expression analysis


Emotions in recsys


Arapakis, 2009User affective feedback from automatic facial expression analysis

Tkalčič et al., 2010 Affective user model


Emotions in recsys

Kaminskas, Ricci2011

find an appropriate musical score that would reinforce the affective state induced by the touristic attraction.





Emotions in recsys

Kaminskas, Ricci2011

find an appropriate musical score that would reinforce the affective state induced by the touristic attraction.




Lops et al, 2012 Ongoing work: emotion detection in the phase of presentation of the recommendations for generating unexpected and seredipitous recommendations


Personality in recsys

Nunes et al., 2007 Personality as ???

Tkalčič et al., 2009Personality-based user similarity measure For the cold start problem

Rong Hu and Pearl Pu,

Personality-based user similarity measure For the cold start problem

Dennis and Masthoff,2012

Adapting persuasive (learning) Technologies to personality traits


PART IV: Emotions in the MM consumption chain


PART IV: Emotions in the MM consumption chain

Scattered work on emotions in RecSys

Unifying framework (too ambitious?)


The proposed framework - 1

Content application

Give conten

t

time

Entry stage Consumption stage Exit stage

Give recommendati

ons

choice



Content application

Entry mood

Detect entrymood

Give conten

t

Exit mood

time


Give recommendati

ons

choice

• Context• Decision making• Influence• Diversification



Content application

Entry mood

Detect entrymood

Give conten

t

Content-induced affective state

Observe user

time


Give recommendati

ons

choice

• Affective tagging• Affective user profiles



Content application

Entry mood

Detect entrymood

Give conten

t

Content-induced affective state Exit mood

Observe user

time


Give recommendati

ons

choice

Detect exit

mood

• Implicit feedback• Evaluation metrics



Content application

Entry mood

Detect entrymood

Give conten

t


Observe user

time


Give recommendati

ons

choice

Detect exit

mood





PART V: Affect in the decision making step


PART V: Affect in the decision making step Stage 2 and 3 are straightforward Stage 1 is interesting = new research avenues

Content application

Entry mood

Detect entrymood

Give content


Observe user

time


Give recommendations

choice

Detect exit

mood





From data-centric to user-centric The community is problem-solving oriented

– „The existing datasets are real, why building synthetic ones?“ (??, RecSys 2011)

The data-centric approach is still rooted in the research community:– „It‘s about music, not about recommenders“ (?? at

RecSys 2011) Solving existing problems is only a part of research ...

... the other part is generating new knowledge (on how the world works) ...

... which in turn generates new problems ...

... which in turn opens new publishing/funding/citing possibilities


General user modeling framework Data-centric = uses data that

– Is available (genres, actors, directors ...)– Easy to acquire (rating, „liking“ ...)

But NOT necessarily data that carry information

USER MODEL

Controlled variables

Uncontrolled variables

Prediction accuracy

?


LET‘S MOVE FORWARD Try new models! Generate new kind of data! Find out how the world really works!

Model DECISION MAKING:– Ajzen model (Andrej‘s talk)– Kahneman model


System 1 / System 2 (c) Kahneman, 2003


Decision Making Modeling in RecSys

System 1 model System 2 model

Aggregation

Decision prediction

Emotiondetection

Personalitydetection Affective stimuli

detection

Contentmetadata


SoA Modeling in RecSys

System 1 model System 2 model

Aggregation

Decision prediction

Emotiondetection

Personalitydetection Affective stimuli

detection

Contentmetadata


Conclusions

RecSys = decision making predictor

Assumption = emotions do influence

Scattered work Unifying framework

Our wish = Focus on stage 1: decision making:– System 1 / System 2 modeling


Future work

Improve models

Generate dataset

Validate

Affect in recommender systems

Technology

Transcript of Affect in recommender systems