1Hayley Hung, TUDelft

Socially Intelligent Sensing

Hayley Hung

Pattern Recognition and Bioinformatics GroupIntelligent Systems Department

2Hayley Hung, TUDelft

What is social behaviour?

3Hayley Hung, TUDelft

What is Human Face-to-face interaction for ?

Makes our lives easier:Relationships TrustCo-operationPersuade/influence othersInformation sharing How could technology help

us to understand/interpret/socially relevant behaviour?How could this help to influence/enhance our experience?

4Hayley Hung, TUDelft

Research Mission Statement

To develop algorithms that can model and understand non-verbal human social behaviour in real life situations. And through this, to understand how to build systems that can enhance people's quality of life by behaving with socially aware intelligence.

Develop algorithms that are perceptive to human social behaviour: Social Signal Processing, Machine LearningEnhancing people's quality of life: Human Machine Interaction, Ambient Intelligent Environments, Design, Architecture.Socially aware intelligence: Social and Behavioural Psychology, Ethnography.

Hayley Hung, TUDelft

What can you say about this picture?

Hayley Hung, TUDelft

What can you say about this picture?

Relaxed postureGestures

Vocal Behaviour

Mutual Gaze

Interpersonal Distance

7Hayley Hung, TUDelft

Current Research Frontier

Person detection

Person tracking

Gaze detection

Body pose estimation

Group detection

Social and Behavioural Pscychology, Ethnography

Activity modelling

Action recognition

Attraction Estimation

Rapport Estimation

Role Recognition

Personality estimation

Dominance Estimation

8Hayley Hung, TUDelft

Current Research Frontier

Relationship intimacy estimation

Conversation quality estimation

Person detection

Person tracking

Gaze detection

Body pose estimation

Group detection

Conversational event estimation

Personality estimation

Relationship quality estimation

Social and Behavioural Pscychology, Ethnography

Activity modelling

Action recognition

New Problem

Definitions

9Hayley Hung, TUDelft

How to model social behaviour

Sensor Data Feature and Cue Extraction

Data AnnotationSocial BehaviourModelling and Classification

Model PerformanceEvaluation

10Hayley Hung, TUDelft

Task 1: Estimating Attraction

Source: http://catinbag.blogspot.nl/2010/07/fatal-attraction.htmlVeenstra and Hung, “Do They Like Me? Using Video Cues to Predict Desires during Speed-dates” in ICCV Workshops 2011

11Hayley Hung, TUDelft

Speed Dating, Non-verbal cues and Attraction

Can proximity-related video cues be used to automatically predict attraction in speed-dates?

12Hayley Hung, TUDelft

Automated Position Extraction

- =

13Hayley Hung, TUDelft

Automated Position Extraction

14Hayley Hung, TUDelft

A Sped up Speed Date:

15Hayley Hung, TUDelft

Speed Dating Results

Predicting attractionVariance in position is best feature predictor for women (70%).Variance in position of the women and synchrony both perform well (70%) for men.

Fusion of all synchrony features

Fusion of all movement features

16Hayley Hung, TUDelft

Speed Date Experiments : Conclusion

The video channel can indeed be a source of valuable information in speed-dates Results differ per gender:

Movement synchrony information is more important for males than females.For females, information on the movement of their male counterpart gives good results

17Hayley Hung, TUDelft

Task 2: Classifiying Social Actions using a Single Wearable Accelerometer

Hung, Englebienne, Kools, “Estimating Social Actions”, Ubicomp 2013

18Hayley Hung, TUDelft

Modelling Human Social Behaviour in Dense Crowds

How can we model instantaneous social behaviour in extremely large crowds?

19Hayley Hung, TUDelft

Our Goal

To develop methods to automatically measure socially relevant behaviour and moods in dense crowds using just a single tri-axial accelerometer

First step: detect socially relevant actions

Speaking; Laughing; Gesturing; Stepping; Drinking

20Hayley Hung, TUDelft

Our Goal

Use insights from Social Psychology:

Speakers move more than listeners (McNeill 2000)

Laughter and joking correlated with sudden bursts of motion (Kendon 1990)

Synchronised motion during conversation (Kendon 1990,Chartrand and Bargh 1999)

21Hayley Hung, TUDelft

Data: The Scenario

32 volunteers (mostly mutual strangers)Experiment Stages: Briefing; Meeting and Mingling; Team formation (groups of 4); Quiz; Award GivingPrizes for top 3 teams5mx6m recording area

22Hayley Hung, TUDelft

Data: The Scenario

32 volunteers (mostly mutual strangers)Experiment Stages: Briefing; Meeting and Mingling; Team formation (groups of 4); Quiz; Award GivingPrizes for top 3 teams5mx6m recording area

23Hayley Hung, TUDelft

Data: The Scenario

32 volunteers (mostly mutual strangers)Experiment Stages: Briefing; Meeting and Mingling; Team formation (groups of 4); Quiz; Award GivingPrizes for top 3 teams5mx6m recording areaEach participant wore a sensing badge

24Hayley Hung, TUDelft

Classifying social actions

Social Actions: speaking, laughing, gesturing, drinking, or stepping

(%) Gesture Step Drink Laugh SpeechPrecision 59 100 100 100 64Recall 24 21 21 38 82F-measure

34 35 35 56 72

25Hayley Hung, TUDelft

Social Action Conclusion

It is possible to detect socially relevant behaviour.Can we detect when people are in the same conversational group?Could we even detect

personality traits quality of people's interaction?Quality of people's relationships?...etc

26Hayley Hung, TUDelft

Looking to the Future...

The way we behave socially can exhibit strong detectable patterns, which are robust to noise.

How simple can the extracted features be?How could socially aware systems benefit better design?

27Hayley Hung, TUDelft

Potential Applications: Human Robot Interaction

28Hayley Hung, TUDelft

Applications: Urban Planning

Behaviour in Public SpacesHow can we measure statistically generaliseable changes as a result of interventions?

29Hayley Hung, TUDelft

Potential Applications: Organisational behaviour

30Hayley Hung, TUDelft

Acknowledgements/Collaborators

Arno Veenstra

Gwenn Englebienne Jeroen Kools

Ben KroseMaarten van Steen

Matt Dobson Claudio Martella

Domenic Vossen

31Hayley Hung, TUDelft

Workshop at ACM Multmedia (acmmm13.org/)Barcelona , October 22

Human Behaviour Understanding for the Interactions in Arts, Creativity, Entertainment and Edutainment

Albert Ali Salah (Bogazici University, Turkey)Oya Aran (Idiap Research Institute, Switzerland)Hatice Gunes (Queen Mary University of London, UK)

32Hayley Hung, TUDelft

Data:Sensors

Each participant wore:indoor positioning deviceProximity and Acceleration sensor

12 participants wore wireless microphone (for annotation)3 fish eye cameras (for annotation)3 accelerometer readings failed (software bug)