Biologically-driven Musical Instrument - eNTERFACE Workshops
ENTERFACE ’10 Amsterdam, July-August 2010 Hamdi Dibeklio ğ lu Ilkka Kosunen Marcos Ortega Albert...
-
date post
21-Dec-2015 -
Category
Documents
-
view
217 -
download
1
Transcript of ENTERFACE ’10 Amsterdam, July-August 2010 Hamdi Dibeklio ğ lu Ilkka Kosunen Marcos Ortega Albert...
eNTERFACE ’10Amsterdam, July-August 2010
Hamdi Dibeklioğlu
Ilkka Kosunen
Marcos Ortega
Albert Ali Salah
Petr Zuzánek
Goal of the Project
Responsive photograph frame◦ User interaction leads to different responses
Modules of the project◦ Video segmentation module
Dictionary of responses◦ Behaviour understanding
Offline: Labelling dictionary Online: Cluster user action
◦ System logic Linking user actions to responses
System Design
External Data(Videos)
Automatic Segmentation
Segment Library
Program Logic(Learning)
Segment Selection
Visual Feature Analysis
Automatic Segmentation
Segment Library
Module 1: Offline segmentation
Module 5: Dual frame modeModule 3:System
response
Module 2: Real-time Feature Analysis
Module 4: Interface
5 video recordings (~1.5-2 min.)◦ Same individual◦ Different actions and expressions
Manual annotation of videos◦ ANVIL tool◦ Annotated by different individuals
Automatic segmentation◦ Segmentation based on actions◦ Optical flow: amount of activity over time
Module 1: Offline Segmentation
Optical flow calculation
Activity calculation based on feature tracking over the sequence
Feature detection◦ Shi-Tomasi corner detection algorithm
Feature tracking◦ Lucas-Kanade feature tracking algorithm◦ Pyramidal implementation (Bouguet)
Optical flow based segmenting To find a calm segment, just search for long
period of frames with calculated optical flow below some treshold (we used 40% of average optical flow calculated from all frames)
To find an active segment, search for frames with lot of optical flow, and then search forward and backward for the calm segments.
Module 2: Real-time Feature Analysis
Face detection activates the system◦ Viola-Jones face detector
User’s behaviour can be monitored via◦ Face detection◦ Eye detection
Valenti et al., isophote-curves based eye detection◦ Optical flow energy
OpenCV Lucas-Kanade algorithm◦ Colour features◦ Facial feature analysis
The eMotion system
Facial feature tracking
Face model: 16 surface patches Face model: 16 surface patches embedded in Bezier volumes.embedded in Bezier volumes.
Piecewise Bezier Volume Deformation Piecewise Bezier Volume Deformation (PBVD) tracker is used to trace the (PBVD) tracker is used to trace the motion of the facial features.motion of the facial features.
* R. Valenti, N. Sebe, and T. Gevers. Facial expression recognition: A fully integrated approach. In ICIAPW, pages 125–130, 2007.
Expression Classification
12 motion units12 motion units NNaive Bayesaive Bayes (NB) (NB) classifier for classifier for
categorizing expressionscategorizing expressions NB Advantage: the posterior probabilities NB Advantage: the posterior probabilities
allow a soft output of the systemallow a soft output of the system
Module 3: System Response
Linking user actions and system responses An action queue is maintained
◦ Different user inputs (transitions) lead to different responses (states)
The responses (segments) are ‘unlocked’ one by one
Sleeping
Wake-up Neutral ResponseFace
detected
Period of inactivity
User input
Module 4: Interface
Currently two external programs are employed:◦ SplitCam◦ eMotion
Glyphs are used to provide feedback to the user Glyph brightness is related to distance to
activation Once a glyph is activated, the same user activity
will elicit the same response Each user can have different behaviours
activating glyphs