Joemon M Jose (with Ioannis Arapakis & Ioannis Konstas) Department of Computing Science.
-
date post
19-Dec-2015 -
Category
Documents
-
view
213 -
download
0
Transcript of Joemon M Jose (with Ioannis Arapakis & Ioannis Konstas) Department of Computing Science.
Exploring the role of affective feedback in Interactive IR
Joemon M Jose(with Ioannis Arapakis & Ioannis Konstas)
Department of Computing Science
Affective Feedback 2
Questions?What is the role of emotions in the
information seeking process?
Do they correspond to any form of relevance feedback?
How can we effectively employ them in information retrieval scenarios?
02/03/2009
Affective Feedback 3
Relevance Feedback• Relevance assessments can contribute in the
disambiguation of the user’s information need
• This is achieved through the application of various feedback techniques
02/03/2009
Affective Feedback 4
Explicit Relevance Feedback• Feedback which is obtained through the explicit
and intended indication of documents as relevant (positive feedback) or irrelevant (negative feedback)
02/03/2009
Affective Feedback 5
Explicit Relevance FeedbackBenefits Drawbacks
Robust method for inferring relevance feedback
Interrupts the flow of the search process
Better query reformulationsIntroduces the cognitive
burden of explicit relevance judgments
Improves considerably the retrieval performance of a
system
Trade-off between the users perusing documents because
the system expects them to do so and because they are
genuinely interested
02/03/2009
Affective Feedback 6
Implicit Relevance Feedback• Implicit Feedback: a passive form of feedback,
which is applied in an intelligent and unobtrusive manner
• Can be used to individualize a system’s responses or develop user models (UM)
02/03/2009
Affective Feedback 7
Implicit Relevance FeedbackBenefits Drawbacks
Disengages users from the cognitive burden of document
rating and relevance judgments
Difficult to interpret
Large amount of data can be obtained very easily
Unreliable (compared to explicit feedback techniques)
Does not account for the individual differences of users
02/03/2009
Affective Feedback 8
Common aspects• Both categories of feedback techniques
determine relevance by considering what occurs on the cognitive and situational level of interaction
• However, they do not account for the affective dimension of the conversational interplay between the user and the system
02/03/2009
Affective Feedback 9
Affective ComputingAffective computing aims in the development of
more natural and flexible systems.Human-machine interactive systems capable of
sensing affect states (stress, inattention, etc) and capable of adapting and responding appropriately to these are likely to be perceived as more natural, efficient and trustworthy (Pantic, Sebe, Cohn, Huang, 2005).
Can we build a multimodal retrieval system that exploits more than one modality?
02/03/2009
Affective Feedback 10
Affective FeedbackCan affective feedback be of any value to IR?
Likely yes, since it is considered a qualitatively rich source of human affect indications, which can be potentially exploited to enhance the information retrieval process.
Affective feedback can be defined asthe sum of all the human affective
expression/indications, which are communicated implicitly to (or identified by) a computer system and can be therefore used to facilitate a more natural, effective and robust interaction.
02/03/2009
Affective Feedback 11
Affective Interaction• Users interact with intentions, motivations and
feelings besides real-life problems and information objects…• Intentions, motivations and emotions are all critical
aspects of cognition and decision-making
02/03/2009
Affective Feedback 12
Affective Interaction• Information systems equipped with the ability to detect
and respond to user emotions could potentially:
1. Improve the naturalness of human-computer interaction
2. Progressively optimize their retrieval strategy
3. Offer a more personalized experience
4. Determine more accurately the relevancy of an information object
02/03/2009
Affective Feedback 13
Affective Interaction• What are the possible reasons of emotion?
1. System?
2. Search strategy & search results?
3. Content design and aesthetics?
4. Other
02/03/2009
Affective Feedback 14
Emotion in IR – Some Conclusions• The co-occurrence of emotions during an
information seeking process, among other physiological, psychological and cognitive processes
• Patterns of emotional variance, which reveal a progressive transition from positive to negative valence as the degree of task difficulty increases
• Depending on their frequency of occurrence the value of the conveyed affective information may potentially vary?
02/03/2009
Affective Feedback 15
Test Collection• For the indexing we used TREC 9 (2000) Web Track
• 1.69 million document subset of the VLC2 collection
• We retained the original content of the TREC topics, but presented them using the structural framework of the simulated information need situations• Introduce a layer of realism, while preserving well-
defined relevance criteria
02/03/2009
Affective Feedback 16
Search Tasks
02/03/2009
Affective Feedback 17
Facial Expression Analysis• Facial expression analysis was applied on the
video recordings of each session
• For each key-frame of the video eMotion calculated the probability of the detected facial expression (assuming there was one) corresponding to any of the seven detectable emotion categories (Neutral, Happiness, Surprise, Anger, Disgust, Fear, Sadness)
02/03/2009
Affective Feedback 18
eMotion• eMotion is an automatic facial expression recognition
system• Developed by Nicu Sebe’s group in Amsterdam/Trento
• It follows a model-based approach, in which a 3-dimensional wireframe model of the face is constructed, once certain facial landmark features are detected
• Head motion of facial deformation can then tracked and measured in terms of motion-units (MU’s), which are eventually classified into one (or more) of the seven detectable emotion categories
02/03/2009
Affective Feedback 19
eMotion
02/03/2009
Affective Feedback 20
Classifier• eMotion has been trained using a generic static
classifier
• The classifier has been developed from a subset of the Cohn-Kanade database
• It performs reasonably well across all individuals, independently of ethnicity-specific features
02/03/2009
Tools & ModalitiesTools:
1) eMotion (Facial Expression Recognition System) + 2d camera2) Pasion (Facial Expression Recognition System) + 3d camera3) Polar RS800 Heart Rate Monitor4) BodyMedia SenseWear Pro3 Armband
• Modalities:Facial Expressions (emotion categories)1
Facial Expressions (motion units)1
HR3
GSR4
Heat Flux4
Skin Temperature4
Acceleration4
Affective Feedback 22
Facial Expressions-
02/03/2009
Affective Feedback 23
Facial Expressions-
02/03/2009
Affective Feedback 24
Biometrics
02/03/2009
Affective Feedback 25
Findingsusers' affective responses will vary across the
relevance of perused information items.
the results also indicate that prediction of topical relevance is possible and
to a certain extent models can benefit from taking into account user affective behaviour.
02/03/2009
Affective Feedback 26
Open Questions
How to select different modalities?Large-scale body movements; Hand-gesture
recognition; Gaze-detection; Speech/voice analysis
How to integrate multiple modalities?Modelling challenge?
How to develop a practical system that respond to users emotional behaviour?
02/03/2009