Your Reactions Suggest You Liked the Movie Automatic Content Rating via Reaction Sensing

Xuan Bao, Songchun Fan, Romit Roy Choudhury, Alexander Varshavsky, Kevin A. Li

Rating Online Content (Movies)

Manual rating not incentivized, not easy … does not reflect experience

Our Vision

Overall star rating

Reaction tags

Reaction-based

highlights

Our Vision

Overall star rating

Reaction tags

Reaction-based

highlights

Automatically

Key Intuition Multi-modal sensing / learning

Reactions / Ratings

02:43 - Action09:21 - Hilarious

Overall – 5 Stars

……

12:01 - Suspense

Specific Opportunities• Visual

– Facial expressions, eye movements, lip movements …

• Audio– Laughter, talking

• Motion– Device stability

• Touch screen activities– Fast forward, rewind, checking emails and IM chats …

• Cloud– Aggregate knowledge from others’ reactions– Labeled scores from some users

Pulse: System Sketch

Applications (Beyond Movie Ratings)• Annotated movie timeline

– Slide forward to the action scenes

• Platform for ad analytics– Assess which ads grabbing attention … – Customize ads based on scenes that user reacts to

• Personalized replays and automatic highlights– User reacts to specific tennis shot, TV shows personalized replay– Highlights of all exciting moments in the superbowl game

• Online video courses (MOOCs)– May indicate which parts of lecture needs clarification

• Early disease symptom identifcation– ADHD among young children, and other syndromes

First Step: A Sensor Assisted Video Player

• Developed on Samsung Galaxy tablet (Android)– Sensor meta-data layered on video as output

Sensing threads control

Observe the user from front cam

Media player control functions monitored

Pulse Media Player

Basic Design

Features from Raw Sensor ReadingsMicrophone, Camera, Acc, Gyro, Touch, Clicks

Reactions: Laugh, Giggle, Doze, Still, Music …Signals to Reactions (S2R)

Reaction to Rating & Adjective (R2RA)

English Adjectives

Numeric Rating

Tag Cloud Final Rating

Data Distillation Process

Basic Design

Features from Raw Sensor ReadingsMicrophone, Camera, Acc, Gyro, Touch, Clicks

Reactions: Laugh, Giggle, Doze, Still, Music …Signals to Reactions (S2R)

Reaction to Rating & Adjective (R2RA)

English Adjectives

Numeric Rating

Tag Cloud Final Rating

Data Distillation Process

Cloud

Visual Reactions• Facial expressions (face size, eye size, blink,

etc.)– Track viewers’ face through the front camera– Track eye position and size (challenging with spectacles)– Track partial faces (via SURF points matching)

Face Tracking Eye Tracking (Green)Blink (Red)

Partial Face

Visual Reactions• Facial expressions (face size, eye size, blink,

etc.)– Track viewers’ face through the front camera– Track eye position and size (challenging with spectacles)– Track partial faces (via SURF points matching)– Detect blinks, lip size

Look for difference between frames

Acoustic Reactions• Laughter, Conversation, Shout-outs …

– Cancel out (known) movie sound from recorded sound – Laughter detection, conversation detection

Even with knowledge of the original movie audio (Blue), it is hard to identify user conversation (distinguish Red and Green)

Acoustic Reactions• Separating movie from user’s audio

– Spectral energy density comparison not adequate

– Different techniques for different volume regimes

High VolumeLow Volume

Acoustic Reactions• Laughter, Conversation, Shout-outs …

– Cancel out (known) movie sound from recorded sound – Laughter detection, conversation detection

Early results demonstrate promise of detecting acoustic reactions

Motion Reactions• Reactions also leave footprint on motion

dimensions– Motionless during intense scene– Fidget during boredom

Intense scene Calm sceneTime to stretch

Motion Reactions• Reactions also leave footprint on motion

dimensions– Motionless during intense scene– Fidget during boredom

Motion Reactions• Reactions also leave footprint on motion

dimensions– Motionless during intense scene– Fidget during boredom

Motion readings correlate with changing in ratings …

Motion Reactions

Motion readings correlate with changing in ratings …Timing of motions also correlate with timing of scene changes

• Reactions also leave footprint on motion dimensions– Motionless during intense scene– Fidget during boredom

Extract Reaction Features – Player control• Collect users’ player control operations• Pause, fast forward, jump, roll back, …• All slider movement

Seek bar

Challenges in Learning

Problem – A Generalized Model Does Not Work• Directly trained model does not capture the rating

trendWhy?

The Reason it Does Not Work is …• Human behaviors are heterogeneous

– Users are different– Environments are different even for same user (home vs.

commute)

home commute

Sensed motion patterns very different when the same movie wateched during a bus commute vs. in bed at home.

The Reason it Does Not Work is …• Human behaviors are heterogeneous

– Users are different– Environments are different even for same user (home vs.

commute)

• Gyroscope readings from same user (at home and office)

The Reason it Does Not Work is …• Human behaviors are heterogeneous

– Users are different– Environments are different even for same user (home vs.

commute)

• Gyroscope readings from same user (at home and office)

• Naïve solution build specific models one by one– Impossible to acquire data for all <User, Context, Movie>

tuplesOffice Home Commute …

Challenges in Learning

Approach:Bootstrap from Reaction Agreements

Approach: Bootstrap from Agreement• Thoughts

– What behavior means positive/negative for a particular setting– How do we acquire data without explicitly asking the user every

time

• Approach: Utilize reactions that most people agree on

TimeClimax Boring Cloud Knowledge

(Other users’ opinions)

Sensor Reading

Ratings

Approach: Bootstrap from Agreement• Solution: spawn from consensus

– Learn user reactions during the “climax” and the “boring” moments

– Generalize this knowledge of positive/negative reactions – Gaussian process regression (ratings) and svm (labels)

GPR SVM

Evaluation

User Experiment Setting• 11 participants watch preloaded movies (~50 movies)

• 2 comedies, 2 dramas, 1 horror movie, 1action movie

• Users provide manual ratings and labels – For ground truth

• We compare Pulse’s ratings with manual ratings

Pulse Truth 1 2 3 4 5

1 0 1 0 0 0

2 0 4 2 0 1

3 0 1 17 0 1

4 0 0 2 5 25 0 0 2 1 7

Preliminary Results – Final (5 Star) Rating

Difference with true 5 star manual rating

Preliminary Results – Final (5 Star) Rating

Preliminary Results – Myth behind the Error• Final ratings can deviate significantly from the

average segment ratings

• User-given scores may not be linearly related to quality

Preliminary Results – Lower Segment Rating Error• Final ratings come from averaging segment ratings• Our system outperforms other methods

Mean Error(5-point scale)

Random ratings Collaborativefiltering

Our system

3 4 4 2 2 2 5Per-segment ratings

Preliminary Results – Better Tag Quality• Tags capture users’ feelings better than SVM alone

Happy Intense Warm

Happy Intense Warm

Preliminary Results – Reasonable Energy Overhead• Reasonable energy overhead compared to without

sensing

More tolerable on tablets. May need duty-cycling on smart phones

Closing Thoughts• Human reactions are in the mind

– However, manifest into bodily gestures, activities

• Rich, multi-modal sensors on moble devices– A wider net for “catching” these reactions

• Pulse is an attempt to realize this opportunity– Distilling semantic meanings from sensor streams– Rating movies … tagging any content with reaction meta data

• Enabler for– Recommendation engines– Content/video search– Information retrieval, summarization

Thoughts?

Backup – potential questions• Privacy concern

– Like every technology, pulse may attract early adoptors– If only final ratings are uploaded, the privacy level is similar to current ratings

• Why not just emotion sensing/just laughter detection– Emotion sensing is a broad and challenging problem…but the goal is different than ours (rating)… – Explicit signs like laughter usually only account for a small duration of movie viewing, we need to

explore other opportunities (motion)– Our approach takes advantage of the specific task – 1. we know the user is watching a movie 2. we

can observe the user for a longer duration (than most emotion sensing work) 3. we know other users’ opinions

• How is this possible…human mind is too complex– Human thoughts are complicated… but they may produce footprints in behaviors– Using collaborative filtering explicitly uses knowledge of other users’ thoughts to bootstrap our

algorithm

• The sample size is small…only 11 users– The sample size is limited, but– Each user watched multiple movies (50+ movies viewed)… segment ratings are for 1-minute

segments (thousands of points)– Collaborative filtering shows that even within this data set, the ratings can diverge and naïve

solution does not work as well as ours

Preliminary Results – Better Retrieval Accuracy• Viewers care more about the highlights of a movie• Find the contribution by using sensing

Gain Additional error

Total goalOverall achieved performance

Challenges in Learning

Problem – A Generalized Model Does Not Work

• Directly trained model does not capture the rating trend

Why?

The Reason it Does Not Work is …

• Human behaviors are heterogeneous– Users are different– Environments are different (e.g., home vs. commute)

home commute

Sensed motion patterns very different when the same movie wateched during a bus commute vs. in bed at home.

The Reason it Does Not Work is …

• Human behaviors are heterogeneous– Users are different– Environments are different (e.g., home vs. commute)

• Impact of sensor readings histograms

• Human behaviors are heterogeneous– Users are different– Environments are different (e.g., home vs. commute)

• Impact on sensor readings histograms

• Naïve solution build specific models one by one– Impossible to acquire data for all <User, Context, Movie> tuples

Office Home Commute …

The Reason it Does Not Work is …

Challenges in Learning

Approach:Bootstrap from Reaction Agreements

Approach: Bootstrap from Agreement

• Thoughts– What behavior means positive/negative for a particular setting– How do we acquire data without explicitly asking the user every time

• Approach: Utilize reactions that most people agree on

Time

Climax Boring

Cloud Knowledge(Other users’

opinions)

Sensor Reading

Ratings

Approach: Bootstrap from Agreement

• Solution: spawn from consensus– Learn user reactions during the “climax” and the “boring” moments – Generalize this knowledge of positive/negative reactions – Gaussian process regression (ratings) and svm (labels)

GPR SVM

Approach: Bootstrap from Agreement

GPR

A Simple Example of GPR

Approach: Bootstrap from Agreement

• On GPR and SVM - SVM– SVM is a supervised learning method for classification

– Identify hyperplanes in high-dimensional space that can best separate observed samples

– For our purpose, we used non-linear SVM with RBF kernel for its wide applicability

User Experiment Setting

• 11 participants watch preloaded movies (46 movies)

• Two comedy, two dramas, one horror movie, one action movie

• Users give manual ratings and labels

• Evaluate by comparing generated ratings with manual ratings

Evaluation

Pulse Truth 1 2 3 4 5

1 0 1 0 0 0

2 0 4 2 0 1

3 0 1 17 0 1

4 0 0 2 5 25 0 0 2 1 7

Preliminary Results – Good Final Rating

Preliminary Results – Myth behind the Error

• Final ratings can deviate significantly from segment rating• User-given scores may not be linearly related to quality

Preliminary Results – Lower Segment Rating Error

• Final ratings come from averaging ratings for each segment• Our system outperforms other methods

Mean Error(5-point scale)

Random ratings Collaborativefiltering

Our system

3 4 4 2 2 2 5Movie

segments

Preliminary Results – Better Retrieval Accuracy

• Viewers care more about the highlights of a movie• Find the contribution by using sensing

Gain Additional error

Total goalOverall achieved performance

Preliminary Results – Better Tag Quality

• Generated tags captures users’ feelings much better than using SVM alone

Happy Intense Warm

Happy Intense Warm

Preliminary Results – Reasonable Energy Overhead

• Reasonable energy overhead compared to without sensing

More tolerable on tablets. May need duty-cycling on smart phones

Closing Thoughts• Human reactions are in the mind

– However, manifest into bodily gestures, activities

• Rich, multi-modal sensors on moble devices– Opportunity for “catching” these activities– Multi-modal capability – whole is greater than sum of parts

• Pulse is an attempt to realize this opportunity– Distilling semantic meanings from sensor streams– Rating movies … tagging any content with reaction meta data

• Enabler for– Recommendation engines– Content/video search– Information retrieval, summarization

Questions?

Extract Reaction Features – Player control• Player control and taps

– Pause, fast forward, jump, roll back, …– All slider movement

Seek bar

Approach: Bootstrap from Agreement

GPR

A Simple Example of GPR

Approach: Bootstrap from Agreement• On GPR and SVM - SVM

– SVM is a supervised learning method for classification

– Identify hyperplanes in high-dimensional space that can best separate observed samples

– For our purpose, we used non-linear SVM with RBF kernel for its wide applicability