Post on 28-Jan-2016
description
http://www.infm.ulst.ac.uk/~paul/
Magee Campus
1
Imagineering Creative Technologies
Professor Paul Mc Kevitt
Intelligent Systems Research CentreSchool of Computing & Intelligent Systems
Faculty of Computing & EngineeringUniversity of Ulster, Magee
Derry/Londonderry Northern Ireland
http://www.infm.ulst.ac.uk/~paul/
Magee Campus
2
Background• Intelligent MultiMedia & MultiModal Systems• Integration of natural language, speech/audio,
graphics/VR, vision• Semantic representation, fusion & synchronisation• Affective computing• Integration & reuse of existing software (Thórisson, 2007)• Relations with cognitive science• Deployment on mobile devices
(Dell Axim, Smartphones, Nintendo DS)• Applications in arts: music, drama, dance, design,
computer games, film, TV/radio, art, storytelling
http://www.infm.ulst.ac.uk/~paul/
Magee Campus
3
http://www.infm.ulst.ac.uk/~paul/
Magee Campus
4
http://www.infm.ulst.ac.uk/~paul/
Magee Campus
5
http://www.infm.ulst.ac.uk/~paul/
Magee Campus
6
http://www.infm.ulst.ac.uk/~paul/
Magee Campus
7
http://www.infm.ulst.ac.uk/~paul/
Magee Campus
8
http://www.infm.ulst.ac.uk/~paul/
Magee Campus
9
CHAMELEON:a platform for developing intelligent multimedia applications
Tom Brøndsted, Paul Dalsgaard, Lars Bo Larsen, Mike Manthey, Paul Mc Kevitt(CPK, Aalborg University, Denmark)
• Speech input/output, image processing & laser
• IntelliMedia Workbench campus information system application
• Semantic representation with frames (Minsky, 1975)
• Blackboard for semantic representation
• Bielefeld DACS distributed communications platform (Fink et al., 1996)
• Implemented with C, C++, DACS, Infovox TTS, grapHvite
• Applications in multimodal systems development
http://www.infm.ulst.ac.uk/~paul/
Magee Campus
Architecture of CHAMELEON
10
http://www.infm.ulst.ac.uk/~paul/
Magee Campus
IntelliMedia Workbench
11
http://www.infm.ulst.ac.uk/~paul/
Magee Campus
CHAMELEON Semantic Representation
• Semantic Frames (Minsky, 1975)
12
Input frame:
[SPEECH-RECOGNISER UTTERANCE: (Point to Hanne's office) INTENTION: instruction! TIME:timestamp]
Output frame:
[LASER INTENTION: description (pointing) LOCATION: coordinates (5, 2) TIME: timestamp]
Integration frame:
[NLPINTENTION: description (pointing) LOCATION: office (tenant Hanne) (coordinates (5, 2)) UTTERANCE: (This is Hanne's office) TIME: timestamp]
http://www.infm.ulst.ac.uk/~paul/
Magee Campus
CONFUCIUS:automatic conversion of natural language to 3D animation
Minhua (Eunice) Ma, Paul Mc Kevitt
• Lexical Visual Semantic Representation (LVSR)Use of Jackendoff’s (1990) Lexical Conceptual Structure (LCS)
& Schank’s (1977) scripts
• Use of Fellbaum’s (1998) semantic classes in WordNet
• Implementation with Java, VRML, WordNet, Connexor parser, FreeTTS (Java)
• Applications in education, computer games, animation production,
digital storytelling
13
http://www.infm.ulst.ac.uk/~paul/
Magee Campus
Architecture of CONFUCIUS
14
http://www.infm.ulst.ac.uk/~paul/
Magee Campus
MultiModal semantic representation
15
http://www.infm.ulst.ac.uk/~paul/
Magee Campus
CONFUCIUS examples
16
• Contact verb example (e.g. hit, collide, scratch, touch)
“The car collided with a wall.” Collision detection Nonspeech sound effects No human role involved
• Human action verb examples“Ipke gave Brigitte a loaf of bread.”
“Bernie put a cup of coffee on the table.” H-Anim Site node locative tags of object (on tag for table object)
“Paul pushed the door.”
“Gert ate the bread.”
http://www.infm.ulst.ac.uk/~paul/
Magee Campus
17
“Paul sat on the chair.”
“Teri was waving goodbye.”
“Helge went to the gym.”
“Gert put on his hat.”
States“Anke is sad.”
“Anke is ashamed.”
http://www.infm.ulst.ac.uk/~paul/
Magee Campus
18
http://www.infm.ulst.ac.uk/~paul/
Magee Campus
SceneMaker:automatic visualisation of screenplays Eva Hanser, Joan Condell, Tom Lunney, Paul Mc Kevitt
• Automatically generates affective virtual scenes from screenplays/play scripts
• Realistic visualisation of emotional aspects
• Extraction of genre/moods from screenplays
• Use of OCC (Ortony-Clore-Collins) affective model,ConceptNet, Open Mind Common Sense (OMCS), WordNet-Affect
• Enhance believability of virtual actors and scene presentation
• Multimodal representation with 3D animation, speech, audio & cinematography
• Applications in assisting directors, actors, drama students, scriptwriters
19
http://www.infm.ulst.ac.uk/~paul/
Magee Campus
Architecture of SceneMaker
20
http://www.infm.ulst.ac.uk/~paul/
Magee Campus
MemoryLane:a mobile digital storytelling companion
Sheila Mc Carthy, Heather Sayers, Mike McTear, Paul Mc Kevitt
• Multimodal stories (speech, text, video) from life-cached memory data (poems, songs, photos, home videos)
• Extensive HCI requirements analysis with focus on older people• Multimodal preferences and abilities of users determines
story modalities and user interface configuration
• Prudent on sensitive life-cached data
• Creates new stories on the fly
• Applications in medical informatics and digital care
• Implemented with Microsoft Visual Studio
• Deployed on Dell Axim X51v PDA
21
http://www.infm.ulst.ac.uk/~paul/
Magee Campus
Architecture of MemoryLane
22
http://www.infm.ulst.ac.uk/~paul/
Magee Campus
MemoryLane Prototype
23
http://www.infm.ulst.ac.uk/~paul/
Magee Campus
AmbiLearn:an ambient intelligent multimodal learning environment for children
Jennifer Hyndman, Tom Lunney, Paul Mc Kevitt
• Educational games & virtual learning environments
• Child - Computer Interaction (ChiCI)
• AmbiLearn architecture & TreasureLearn application
• Implementation with DevKit Pro (Nintendo homebrew)
• Deployment on mobile device (Nintendo DS)
• Application to collaborative learning for children
24
http://www.infm.ulst.ac.uk/~paul/
Magee Campus
Architecture of AmbiLearn
25
http://www.infm.ulst.ac.uk/~paul/
Magee Campus
PlayPhysics:an emotional games learning environment for teaching Physics
Karla Munoz Esquivel, Tom Lunney, Paul Mc Kevitt
• Virtual Learning Environment (VLE) for teaching Physics
• Incorporating serious games & intelligent tutoring systems (ITS)
• Modelling of emotions & personality of student
• Use of Bayesian Networks for modelling student knowledge
• Affective feedback through audio-visual effects
• On-line web application with Student, Lecturer, Head of Department, Administrator views
• Implemented with Elvira Bayesian Network tool, Flash, Java, Tomcat Apache Server, MySQL database, ConceptNet, 3D Studio Max & Maya
• Applications in educational software & computer games
26
http://www.infm.ulst.ac.uk/~paul/
Magee Campus
Olympia architecture
27
http://www.infm.ulst.ac.uk/~paul/
Magee Campus
PlayPhysics interface (astronaut game)
28
http://www.infm.ulst.ac.uk/~paul/
Magee Campus
TeleMorph & TeleTuras:bandwidth determined mobile multimodal presentation
Tony Solon, Kevin Curran, Paul Mc Kevitt
• Multimedia limitations of mobile networks, mobile device capabilities
• User preferences on modalities
• Transmoding between modalities with cross-modality adaptation
• Use of Fuzzy (logic) Inference system (FIS)
• TeleMorph platform & TeleTuras tourist application
• Implemented with MATLAB
• Applications in streaming media, computer networks, multimodal presentation
29
http://www.infm.ulst.ac.uk/~paul/
Magee Campus
30
Architecture of TeleMorph Fuzzy Inference System (FIS)
hi veryHiaboveAvgavgbelowAvglo veryLonull
hi veryHiaboveAvgavgbelowAvglo veryLonull
aboveAvgavgbelowAvgveryLonull lo< Avg > Avg
hi veryHi
Packet Loss
bpsCurr,vCurr, vLowest, vHighest
aiCurr, aiLowest, aiHighestitCurr, itLowest, itHighestaCurr, aLowest, aHighesttCurr, tLowest, tHighest
18 Inputs
pktLoss
adaptFreq
hi veryHiaboveAvgavgbelowAvglo veryLonull
hi veryHiaboveAvgavgbelowAvglo veryLonull
hi veryHiaboveAvgavgbelowAvglo veryLonull
hi veryHiaboveAvgavgbelowAvglo veryLonull
hi veryHiaboveAvgavgbelowAvglo veryLonull
hi veryHiaboveAvgavgbelowAvglo veryLonull
IfAndThenRule 1
Rule7464
IfAndThen
hi veryHiaboveAvgavgbelowAvglo veryLonull
hi veryHiaboveAvgavgbelowAvglo veryLonull
hi veryHiaboveAvgavgbelowAvglo veryLonull
Aggregation
Defuzzification
Implication – Apply 7,464 Rules
AdaptFreq
5 outputs
v - video qualityai - audio & image qualityit - image & text qualitya - audio qualityt - text quality
bpsCurr, vCurr, etc...
http://www.infm.ulst.ac.uk/~paul/
Magee Campus
31
TeleTuras GUI
http://www.infm.ulst.ac.uk/~paul/
Magee Campus
32
Available media qualities in 6 sample test scenarios
http://www.infm.ulst.ac.uk/~paul/
Magee Campus
Adaptation trace data from test scenario 6
33
http://www.infm.ulst.ac.uk/~paul/
Magee Campus
MediaHub:Bayesian decision-making in an intelligent multimedia distributed platform hubGlenn Campbell, Tom Lunney, Aiden McCaughey, Paul Mc Kevitt
• Interpret/generate semantic representations of multimodal input/output
• Fusion & synchronisation of multimodal data
• Decision-making with Bayesian Causal Probabilistic Networks (CPNs)
• Implemented with Psyclone platform, HuginLite Bayesian modelling tool, Java, NetBeans IDE
• Applications in decision-making for key problems in multimodal systems: anaphora resolution, domain knowledge awareness, multimodal presentation, turn-taking, dialogue-act recognition & parametric learning
34
http://www.infm.ulst.ac.uk/~paul/
Magee Campus
Architecture of MediaHub
35
http://www.infm.ulst.ac.uk/~paul/
Magee Campus
Decision-making layers
36
http://www.infm.ulst.ac.uk/~paul/
Magee Campus
psyProbe view of MediaHub Whiteboard
37
http://www.infm.ulst.ac.uk/~paul/
Magee Campus
Alternative `Turn-taking' Bayesian network in Hugin
38
http://www.infm.ulst.ac.uk/~paul/
Magee Campus
SoFI:SOng Form Intelligence for repairing streaming musicacross wireless bursty networks
Jonathan Doherty, Kevin Curran, Paul Mc Kevitt
• Large packet loss dropouts of 5 - 15 s.• Use audio pattern recognition & matching to repair gaps• K-means clustering classifier determines self-similarity matches• Match current sections of song received with previous
sections played• Typicality of Western Tonal Format (WTF)• Use of MPEG-7 Audio Spectrum Envelope (ASE) multimedia
content description standard giving features for extraction• Implemented with G-streamer for steaming media, Ices2, Icecast2
for audio broadcast & MPEG-7 Audio Spectrum Envelope (ASE)• Applications in repair of streaming music
39
http://www.infm.ulst.ac.uk/~paul/
Magee Campus
Architecture of SoFI
40
http://www.infm.ulst.ac.uk/~paul/
Magee Campus
Spectrogram representation
41
http://www.infm.ulst.ac.uk/~paul/
Magee Campus
42
http://www.infm.ulst.ac.uk/~paul/
Magee Campus
43
http://www.infm.ulst.ac.uk/~paul/
Magee Campus
Steganoflage:strengthening steganography in digital imagesAbbas Cheddad, Joan Condell, Kevin Curran, Paul Mc Kevitt
• Hiding/embedding data in transmission medium• Use of 1D hash algorIthm coupled with
2D iFFT (irreversible Fast Fourier Transform)• Encryption of digital documents in 2D spatial domain• Use of steganography to add security layer resistant to
noise & JPEG compression• Exploitation of Jarvis kernel (Jarvis et al., 1976) to generate error
diffusion signal• Use of Wavelet-based Inverse Halftoning via De-convolution
(WInHD) to recover approximate original signal• Implemented with MATLAB & PHP• Applications in security, forensics, forgery, digital watermarking
44
http://www.infm.ulst.ac.uk/~paul/
Magee Campus
Architecture of Steganoflage
45
http://www.infm.ulst.ac.uk/~paul/
Magee Campus
Securing confidential data
46
The traditional method of securing individuals’ data with the notion of security by obscurity that Steganography introduces
App
licat
ion
http://www.infm.ulst.ac.uk/~paul/
Magee Campus
National & international collaboration
• InstitutoTecnológico de Estudios Superiores de Monterrey, Campus Ciudad de México (ITESM-CCM), Mexico
• Aalborg University, Denmark• CADIA, Reykjavik University, Iceland• University of Bielefeld, Germany• LIMSI (CNRS), Orsay, Paris, France• Sheffield University, England• Stanford University, California, USA
47
http://www.infm.ulst.ac.uk/~paul/
Magee Campus
48
http://www.infm.ulst.ac.uk/~paul/
Magee Campus
49
http://www.infm.ulst.ac.uk/~paul/
Magee Campus
50
Noel Sharkey thinks AI is an illusion
http://www.infm.ulst.ac.uk/~paul/
Magee Campus
51
http://www.infm.ulst.ac.uk/~paul/
Magee Campus
Intellectural Property (IP)Steganoflage:
United Kingdom Patent Application [No. 0819982.0]:“Method for Skin Tone Detection”,University of Ulster,Abbas Cheddad, Joan Condell, Kevin Curran, Paul Mc Kevitt.
SoFI:United Kingdom Patent Application [No. 0908879.0]: “A system and method for streaming music repair and error concealment”,University of Ulster,Jonathan Doherty, Kevin Curran, Paul Mc Kevitt.
52
http://www.infm.ulst.ac.uk/~paul/
Magee Campus
Industrial collaborationInterTradeIreland Fusion project, 2008/10:Nvolve Ltd., Letterkenny, Co. Donegal, Ireland, Michael McNeill, School of Computing & Information Engineering, Coleraine; Prof. Paul Mc Kevitt.
Enterprise Ireland Innovation Voucher, IV-2008 1047, 2008/09:``Navigatour'',Backseat Tour Guides Ltd., Letterkenny, Co. Donegal, Ireland,Anton Hutton, School of Creative Arts & EyeSpyFX; Prof. Paul Mc Kevitt.
53
http://www.infm.ulst.ac.uk/~paul/
Magee Campus
University of Ulster, Office of Innovation, Proof-of-Principle (PoP), 2008/09:``Identity cards employing steganography``,Abbas Cheddad, Joan Condell, Kevin Curran, Paul Mc Kevitt.
University of Ulster, Office of Innovation, Proof-of-Principle (PoP), 2009:``System and method for streaming music repair and error concealment``,Jonathan Doherty, Kevin Curran, Paul Mc Kevitt.
Invest NI Proof-of-Concept (PoC) project, 2009/10:``Secure digital watermarks (SDW):secure data hiding in image frames using steganography``,Joan Condell, Kevin Curran, Paul Mc Kevitt.
54
http://www.infm.ulst.ac.uk/~paul/
Magee Campus
Creative People
55
• Abbas Cheddad• Conrad Deighan• Jonathan Doherty• Eoghan Furey• Eva Hanser• Jennifer Hyndman• Sheila McCarthy• Karla Munoz-Esquivel• Dr. Glenn Campbell (Nvolve Ltd., Letterkenny)• Dr. Minhua (Eunice) Ma
(University of Derby)• Dr. Tony Solon (Bath)
• Dr. Joan Condell• Dr. Kevin Curran• Anton Hutton
(Creative Arts & EyeSpyFX)• Dr. Tom Lunney• Aiden McCaughey• Prof. Paul Mc Kevitt• Dr. Michael McNeill
(Coleraine)• Prof. Mike McTear
(Jordanstown)• Dr. Heather Sayers
http://www.infm.ulst.ac.uk/~paul/
Magee Campus
Conclusion & future work• Imagineering creative technologies
• MultiModal semantic representation
• Affective computing
• Deployment on mobile devices
• Applications in creative industries
56