100 DAY REVIEWEVA HANSER

SUPERVISORS:

SceneMaker : Intelligent Multimodal Visualisation

of Natural Language Scripts

School of Computing & Intelligent Systems, Faculty of Computing & Engineering, University of Ulster

Prof. PAUL MC KEVITTDr. TOM LUNNEYDr. JOAN CONDELL

Presentation Outline

Project Aims & Objectives

Literature ReviewProject ProposalConclusion

Project Aims & Objectives

Extraction of Visual and Emotional Information

from Play/Film Scripts

Structuring for Visualisation Purposes

Synchronised Presentation in all Modalities (Vision, Speech, Illumination, Camera Work)

Compelling, Life-like Animations

1. Detecting Emotions and Personality

2. Modeling Affective Behaviour of Virtual Actors

3. Multimodal Visualisation of 3D Scenes

Literature Review

Literature Review1. Detecting Emotions and Personality

Which Emotions exist? Which are needed for realistic animation?

How to categorise all Emotions and Personality?

Literature Review1. Detecting Emotions and Personality

Psychological Theories of Emotion

6 Basic, Universal Emotions (Ekman and Rosenberg, 1997) : Happiness, Sadness, Anger, Disgust, Fear, Surprise

Pleasure-Arousal-Dominance (PAD) Model:Intensity Value Scale

OCC Model (Ortony et al. 1988): 22 Emotion Categories arranged by Goals, Likelihood, Familiarity

Personality ModelOCEAN (De Raad, 2000): Openness, Conscientiousness, Extroversion,

Agreeableness, Neuroticism

Social Roles

Story Character Roles

Literature Review 1. Detecting Emotions and Personality

Fuzzy Rule Based System (Su et al., 2007)

Annotation of Scene Scripts:XML-based Scripting Languages

BEAT (Cassell et al., 2001), MPML (Breitfuss et al., 2007)

Tags to Control Body Language & Utterances

ALMA (Gebhard, 2005), SCREAM (Prendinger and Ishiyuka, 2002)

Emotion Expression Mapped to Body Language

Literature Review 2. Modeling Affective Virtual Actors

Gandalf, Rea (Vilhjálmsson and Thórisson, 2007), Max (Kopp et al., 2008), Greta (Pelachaud, 2005)

Face-to-Face ConversationLanguage Understanding and GenerationGaze Responding to GesturesSpeaker and Listener FeedbackFace and Hand Expressions

Casino VirtuellGreta

Image Source: Pelachaud, 2005 Image Source: Gebhard and Schröder, 2008

Literature Review 3. Visualisation of 3D Scenes

Sound and Music Effects

Cinematic Principles (Camera, Lighting, Staging)

EML : Expression Mark-up Language (De Melo and Paiva, 2006)

Multimodal Expressions for Virtual Human and Environment

Literature Review 3. Visualisation of 3D Scenes

WordsEye (Coyne and Sproat, 2001)

Static 3D scenesRestricted Language Input

CONFUCIUS (Ma, 2006)

Multimodal 3D Animation of NL stories

Image Source: Coyne and Sproat, 2001 Image Source: Ma, 2006

Literature Review 3. Visualisation of 3D Scenes

Madame Bovary(Cavazza et al., 2007)

Scene Projection into RoomInteraction with Human Actor

Image Source: Cavazza et al., 2007

Virtual Theatre Interface(Virtual Theatre, 2004) Different Audience ViewsStage Lighting and Positioning

Image Source: Virtual Theatre, 2004

Project Proposal

1. THE PROJECT IDEA

2. SCENEMAKER ARCHITECTURE

3. SOFTWARE ANALYSIS

4. RELATION TO OTHER WORK

5. PROJECT SCHEDULE

Affective Actors:Emotionally Influenced Body Language

Environmental Expression: Emotionally Influenced Multimedia :Music, Sound, Illumination, Timing and Camera Work

Software Prototype: SceneMakerGenerates 3D Scenes from Screenplays

Web Based User Interface for Manual ManipulationAccessible on Mobile Devices Testing with Sample Screenplays of Different Genres

Project ProposalThe Project Idea

Project ProposalSceneMaker Architecture

Project ProposalSoftware Analysis

Constructionist Design Methodology (CDM) Constructionist AI Methodology (CAIM) (Thórisson, 2007)

Psyclone (Thórisson et al., 2004) , OpenAIR (Thórisson, 2007)

Unified Messaging Format between Various Input and Output Modules/Modalities

CONFUCIUS (Ma, 2006)

Natural Language Processing Animated 3D Scenes with H-Amin 3D Models

CSLU Text-To-Speech Synthesis (Sutton et al., 1998)

Project ProposalRelation to Other Work

Emotion Visualisation from NL Text

Emotional Influence on Various Modalities

Mobile User Interface

Relation to Other Work

Project ProposalProject Schedule

 Research Activities 2008 2009 2010 2011  Oct-Dec Jan-Mar Apr-Jun Jul- Sep Oct- Dec Jan- Mar Apr-Jun Jul-Sep Oct-Dec Jan-Mar Apr-Jun Jul-Sep

Perform Literature Review                        100 Day Review and Presentation                        Submission to ISEA2009                        Investigation on User Requirements, Interviews with Actors/Directors          

            

Submission to MobileHCI 2009                        In-Depth Review of Systems and Approaches Relevant for Integration into SceneMaker

         

 

           Submission to ICMI-MLMI 2009 Conference

          

           Submission to ACII 2009 Conference                        Confirmation                      Design Automated Scene Production System

          

           Submission to AICS 2009                        Implementation of Automated Scene Production System          

            

2nd Year Poster                        Implementation of SceneMaker GUI in Accordance with HCI Guidelines                        Submission to IEEE Pervasive Computing Journal          

            

Test and Evaluate                        Submission to ACM Transactions on Multimedia Computing, Communications and Applications          

 

           3rd year presentation                        Thesis write up                        

ActivitiesSubmissionsDeliverables

Multimodal System, SceneMaker, for Automatic Visualisation of Affective Expressions in Story Scripts

Assisting Directors, Drama Students, Writers and Animators to Test Ideas

User Interface for Manual Adjustments

Existing Systems Partially Solve Aspects of NLP, ECAs, Emotion Modeling and Multimodal Storytelling

Test Basis: Screenplays of Different Genres

Conclusion

QUESTIONS OR COMMENTS?

Thank you for your attention.

www.infm.ulst.ac.uk/~eva