Affect in recommender systems

Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..

Marko TkalčičUniversity of Ljubljana

Presentation overview

I: LDOS presentation & Motivation II: What are emotions? III: Emotion in recsys – related work IV: Role of emotions in the MM consumption chain V: Affect in the decision-making stage Conclusions

Note: some material is not ours ... Fair use ...

Part I: LDOS group at UL FE and underlying assumption

LDOS group at UL FE University of Ljubljana

– Faculty of electrical engineering• LDOS (Digital signal processing laboratory)

– Approx 15 members

Relevant people

Head: prof. Jurij Tasič

Andrej Košir

Marko Tkalčič

Ante Odić

Matevž Kunaver Tomaž Požrl

LDOS work on recommender systems 2002-2009: public movie datasets

– CBR– CF

2009-2012– Emotions– Context

2012 – – Decision making (affective + cognitive attributes)

• Ajzen model• Kahneman/Tversky model• ...

Basic RecSys

AffectiveRecSys

AffectiveComputing

Decision making Modeling in RecSys

Underlying presumption of our work Recommender System = predictor of users‘ decision

making Decision making: EMOTIONS DO INFLUENCE

(c) Dilbert.com

PART II : What is affect/emotions/mood/personality

PART II : What are affect/emotions/mood/personality

Not well defined (wikipedia):

– Emotion = subjective, conscious experience

– Affect = experience of emotion (interchangable)

– Emotion vs. Mood:• Emotion = high arousal, short term• Mood = low arousal, long term

– Personality = accounts for the individual differences in the users’ emotional, interpersonal, experiential, attitudinal and motivational styles [John and Srivastava, 1999]

Time (duration)

emotion mood personality

CHANGES FIXED

Psychophysiologicalexpressions

Biologicalreactions

Mental states

Overview of emotions/moods Several definitions We take simple models, easy to incorporate in computers:

– Basic emotions– Dimensional model– Circumplex model

Basic emotions Discrete classes model Different sets Charles Darwin: Expression of emotions in man and

animal Paul Ekman definition (6 + neutral):

– Happiness– Anger– Fear– Sadness– Disgust– Surprise

(c) Paul Ekman

Basic emotions Discrete classes model Different sets Charles Darwin: Expression of emotions in man and

animal Paul Ekman definition (6 + neutral):

– Happiness– Anger– Fear– Sadness– Disgust– Surprise

(c) Paul Ekman

Dimensional model Three dimensions

– Valence (positive vs. Negative)– Arousal (high vs. Low)– Dominance (power(less) over emotions)

Dimensional model Three dimensions

– Valence– Arousal– Dominance

– (c) Lang, P. J. (1980)

Each emotive state is a point in the VAD space

Circumplex model Maps basic emotions dimensional model (Posner et al.)

Arousal

Valence

negative positive

neutral

sadness

disgust

surprise

joyanger

How to detect emotions Emotions are characterized:

– psychophysiological expressions, – biological reactions – mental states

SENSORS !!!

How to detect emotions? Explicit vs. Implicit Explicit

– Questionnaires (SAM)

How to detect emotions? Explicit vs. Implicit Explicit

– Questionnaires (SAM) Implicit:

– Work done in the affective computing community– Different modalities (sources):

• Facial actions (video)• Physiological signals ( GSR, EEG)• Voice• Posture• ...

– ML techniques• Classification (basic emotions)• Regression (dimensional model)

Lie To Me Part1.avi (c) 20th Century Fox Main Character Cal Lightman = Paul Ekman

Defined the FACS(Facial Action Coding System)

LDOS Experiment 2 datasets:

– Posed (Cohn-Kanade dataset)– Spontaneous (LDOS-PerAff-1 dataset)

Input: Video streams of facial expressions as responses to visual stimuli

Output: emotive states as distinct classes

Gabor features kNN

Emotive state

Results and conclusions Posed dataset: accuracy = 92 % Spontaneous dataset: accuracy = 62% Reasons for bad results:

– Weak learning supervision– Non optimal video acquisition (face rotation, occlusions,

changing lightning ...)– Non extreme facial expressions

Upcoming paper: IEEE Transactions on Multimedia

Personality Definition:

User personality accounts for the individual differences in the users’ emotional, interpersonal, experiential, attitudinal and motivational styles [John and Srivastava, 1999]

Ever-lasting Several models

Five Factor Model (FFM or Big5): Openness (inventive/curious vs. consistent/cautious) Conscientiousness (efficient/organized vs. easy-going/careless) Extraversion (outgoing/energetic vs. solitary/reserved) Agreeableness (friendly/compassionate vs. cold/unkind) Neuroticism (sensitive/nervous vs. secure/confident)

How to measure? Questionnaires:

International Personality Item Pool ( http://ipip.ori.org/ )

LDOS PerAff-1 dataset Emotive responses Ratings Personality data Videos of facial expressions 50 users, 70 items, sparsity=0 http://slavnik.fe.uni-lj.si/markot/Main/LDOS-PerAff-1

PART III : Related work on emotions in recsys

Emotions and personality Scattered work

Emotions in recsys

Gonzalez, 2007 ? Emotions as context in recsys?

Arapakis et al., 2009User affective feedback from automatic facial expression analysis

Emotions in recsys

Arapakis, 2009User affective feedback from automatic facial expression analysis

Tkalčič et al., 2010 Affective user model

Emotions in recsys

Kaminskas, Ricci2011

find an appropriate musical score that would reinforce the affective state induced by the touristic attraction.

Emotions in recsys

Kaminskas, Ricci2011

find an appropriate musical score that would reinforce the affective state induced by the touristic attraction.

Lops et al, 2012 Ongoing work: emotion detection in the phase of presentation of the recommendations for generating unexpected and seredipitous recommendations

Personality in recsys

Nunes et al., 2007 Personality as ???

Tkalčič et al., 2009Personality-based user similarity measure For the cold start problem

Rong Hu and Pearl Pu,

Personality-based user similarity measure For the cold start problem

Dennis and Masthoff,2012

Adapting persuasive (learning) Technologies to personality traits

PART IV: Emotions in the MM consumption chain

Scattered work on emotions in RecSys

Unifying framework (too ambitious?)

The proposed framework - 1

Content application

Give conten

Entry stage Consumption stage Exit stage

Give recommendati

choice

Content application

Entry mood

Detect entrymood

Give conten

Exit mood

Give recommendati

choice

• Context• Decision making• Influence• Diversification

Content application

Entry mood

Detect entrymood

Give conten

Content-induced affective state

Observe user

Give recommendati

choice

• Affective tagging• Affective user profiles

Content application

Entry mood

Detect entrymood

Give conten

Content-induced affective state Exit mood

Observe user

Give recommendati

choice

Detect exit

• Implicit feedback• Evaluation metrics

Content application

Entry mood

Detect entrymood

Give conten

Observe user

Give recommendati

choice

Detect exit

PART V: Affect in the decision making step

PART V: Affect in the decision making step Stage 2 and 3 are straightforward Stage 1 is interesting = new research avenues

Content application

Entry mood

Detect entrymood

Give content

Observe user

Give recommendations

choice

Detect exit

From data-centric to user-centric The community is problem-solving oriented

– „The existing datasets are real, why building synthetic ones?“ (??, RecSys 2011)

The data-centric approach is still rooted in the research community:– „It‘s about music, not about recommenders“ (?? at

RecSys 2011) Solving existing problems is only a part of research ...

... the other part is generating new knowledge (on how the world works) ...

... which in turn generates new problems ...

... which in turn opens new publishing/funding/citing possibilities

General user modeling framework Data-centric = uses data that

– Is available (genres, actors, directors ...)– Easy to acquire (rating, „liking“ ...)

But NOT necessarily data that carry information

USER MODEL

Controlled variables

Uncontrolled variables

Prediction accuracy

LET‘S MOVE FORWARD Try new models! Generate new kind of data! Find out how the world really works!

Model DECISION MAKING:– Ajzen model (Andrej‘s talk)– Kahneman model

System 1 / System 2 (c) Kahneman, 2003

Decision Making Modeling in RecSys

System 1 model System 2 model

Aggregation

Decision prediction

Emotiondetection

Personalitydetection Affective stimuli

detection

Contentmetadata

SoA Modeling in RecSys

System 1 model System 2 model

Aggregation

Decision prediction

Emotiondetection

Personalitydetection Affective stimuli

detection

Contentmetadata

Conclusions

RecSys = decision making predictor

Assumption = emotions do influence

Scattered work Unifying framework

Our wish = Focus on stage 1: decision making:– System 1 / System 2 modeling

Future work

Improve models

Generate dataset

Validate

Affect in recommender systems

Technology

Transcript of Affect in recommender systems

Recommender Systems

Charla recommender systems

Interactive Recommender Systems

Trust Networks for Recommender Systems · Trust Networks for Recommender Systems Vertrouwensnetwerken voor Aanbevelingssystemen ... 5 Social Recommender Systems 101 ... Apart from

Personalizing & Recommender Systems

Recommender Systems Bener

Recommender systems tutorial.pdf

Intention Recognition for Mixed-Initiative Recommender Systems · 2017-11-28 · initiative recommender systems. 1 Introduction Conversational recommender systems are good examples

Music Recommender Systems

Recommender Systems

Personalized Recommender Systems: Mining User PreferenceRecommender Types of Recommender Systems - 6 - Recommender Systems Collaborative Filtering Memory-based Model-based. Hybrid

Recommender Introduction to Recommender Systems and ...

Recommender Systems an Introduction Chapter07 Evaluating Recommender Systems

INTRODUCTION TO RECOMMENDER - Leuphana … · SYSTEMS AND THEIR EVALUATION Olga ... recommender systems Introduction Properties of recommender ... 11 Tutorial: Recommender Problems

Recommender Systems - Universidade NOVA de Lisboactp.di.fct.unl.pt/~jmag/ws/slides/b08 Recommender systems.pdf · Recommender systems •Recommender systems aim at suggesting new

Recommender Systems & Personalization...Recommender Systems & Personalization Pattie Maes MAS 961- Week 6. Recommender Systems: ... movies, products, … User profile (info about user)

Tutorial: Recommender Systemswelling/teaching/CS77B... · Tutorial: Recommender Systems ... Recommender systems implementation & evaluation Product configuration systems Web mining

Hybrid recommender systems

Recommender Systems. Outline Limitations of Recommender Systems SMARTMUSEUM Case Study.

Movie genome: alleviating new item cold start in movie … · MM Multimedia RS Recommender systems VRS Video recommender systems MRS Movie recommender systems MMRS Multimedia recommender