Emotion-based multimedia browsing (R&D Proposals)
-
Upload
antoniofeynmann -
Category
Documents
-
view
216 -
download
0
Transcript of Emotion-based multimedia browsing (R&D Proposals)
-
8/14/2019 Emotion-based multimedia browsing (R&D Proposals)
1/38
Project ProposalsProject Proposals
Emotion-Controlled Digest Generation for MultimediaContents
Emotion-Controlled Variable Multimedia Transmission
Virtual Meeting Place
Agent Technology for Entertainment/Consumer Services
An Agent Digest Distribution System for Audio and VideoContents
-
8/14/2019 Emotion-based multimedia browsing (R&D Proposals)
2/38
EmotionEmotion--ControlledControlled
Digest GenerationDigest Generation
for Multimedia Contentsfor Multimedia Contentsby
A. Barletta
-
8/14/2019 Emotion-based multimedia browsing (R&D Proposals)
3/38
Multimedia DigestMultimedia Digest
Give an fast preview of multimedia content
Video, audio, books, pictures, etc
Automatic algorithm Who is consuming the digest ?
Are the information contained in a movie
sufficient for generating a digest ?
What about the Human factor ?
-
8/14/2019 Emotion-based multimedia browsing (R&D Proposals)
4/38
EmotionEmotion--controlled Digestcontrolled Digest
Generation (I)Generation (I)
Digest
-
8/14/2019 Emotion-based multimedia browsing (R&D Proposals)
5/38
EmotionEmotion--controlled Digestcontrolled Digest
Generation (II)Generation (II)Emotion Engine
DigestGenerator
Model
Bio Sensors:skin,
blood pressure,retinal activity, etc
Emotions: Attention, Stress, Suspense, etc
audio
videoStream:-Video-Audio-Text
Using the inputfrom the emotionEngine we select asubset of the entireMovie, song, book
-
8/14/2019 Emotion-based multimedia browsing (R&D Proposals)
6/38
EmotionEmotion--controlled Digestcontrolled Digest
Generation (III)Generation (III)networkednetworked --
Emotion Engine
Real TimeDigest
Close Loop
Control
Media Server
-
8/14/2019 Emotion-based multimedia browsing (R&D Proposals)
7/38
Content AnalysisContent Analysis
-
8/14/2019 Emotion-based multimedia browsing (R&D Proposals)
8/38
Emotion Engine (I)Emotion Engine (I)
Already available several solutions in terms of emotion detection
Several input methods:
Galvanic skin Response
Electrocardiogram
Elecromyogram
BVP (Blood, Volume, Pressure) Photoplethysmyograph - Respiration Hall effect respiration sensor -
Temperature (using sensors or infrared cameras)
General Somatic Machine (GSM)
Facial Action Coding System
Posture, Gesture, Voice, Face
Pupillary dilation
Stomach motility
Penis size Cortisol level in saliva
Mature models applied to education, Human Machine Interaction, AutomaticMultimedia Selection, CE interface, etc
No current usage in the digest generation
-
8/14/2019 Emotion-based multimedia browsing (R&D Proposals)
9/38
Emotion Engine (II)Emotion Engine (II)
-
8/14/2019 Emotion-based multimedia browsing (R&D Proposals)
10/38
Digest Generator ModelDigest Generator Model
It collects emotions and synchronized with content
It uses the data collected for filtering composite media (video,audio, text) and for creating a digest
The multimedia content is subdivided in time quanta and ateach of these quanta is associated the collected humanfeedback
It models the brain activity during the consuming ofinformation: we try to synthesize this activity
With approximated emotion detection and a good empiricalmodel we can simulated the human mind and select only themore important information that needs to be stored
-
8/14/2019 Emotion-based multimedia browsing (R&D Proposals)
11/38
Possible ApplicationPossible Application
Producing general digest for preview and
eCommerce
Producing real time digest for specific users
(networking close loop case) for late complete
vision
Fast consuming of multimedia contents (no digest)
Video thumbnails
Emotion based CE feedback (gaming)
-
8/14/2019 Emotion-based multimedia browsing (R&D Proposals)
12/38
ReferencesReferences
http://affect.media.mit.edu/AC_research/sensing.html
http://www.almaden.ibm.com/cs/BlueEyes/index.html[BlueEyes-enabled television ]
web.media.mit.edu/~carsonr/pdf/hcii01.pdf []
-
8/14/2019 Emotion-based multimedia browsing (R&D Proposals)
13/38
-
8/14/2019 Emotion-based multimedia browsing (R&D Proposals)
14/38
EmotionEmotion--Controlled VariableControlled Variable
Multimedia TransmissionMultimedia Transmission
by A. Barletta
-
8/14/2019 Emotion-based multimedia browsing (R&D Proposals)
15/38
MultimediaMultimedia
Fix stream adapted to the available
bandwidth
Who is the consumer ? A machine or ahuman ?
Human have not a constant consuming rate
Variable bandwidth multimedia distribution
dependent on the user attention
-
8/14/2019 Emotion-based multimedia browsing (R&D Proposals)
16/38
EmotionEmotion--Controlled MultimediaControlled Multimedia
DistributionDistribution
Emotion Engine
Variable
BandwidthUsage
Close Loop
Control
Media Server
-
8/14/2019 Emotion-based multimedia browsing (R&D Proposals)
17/38
Variable Quality/BandwidthVariable Quality/Bandwidth
The Quality Of Service of the streamedmultimedia content can be changed in a widerange
Human information assumption is not constant: itdepends on the degree of attention
We can adapt the quality of the content to thestatus of the user
Examples: Switching the video stream when the user is notactively focusing on the screen (audio is maintained)
-
8/14/2019 Emotion-based multimedia browsing (R&D Proposals)
18/38
Possible ApplicationsPossible Applications
Limited bandwidth devices (mobile phone)
Video stream
-
8/14/2019 Emotion-based multimedia browsing (R&D Proposals)
19/38
-
8/14/2019 Emotion-based multimedia browsing (R&D Proposals)
20/38
Virtual Meeting PlaceVirtual Meeting Place
How to meet your friends onAmazon.com!
(by A. Barletta)
-
8/14/2019 Emotion-based multimedia browsing (R&D Proposals)
21/38
Internet ExperienceInternet Experience
Web site are virtual meeting place
No support for communicating with visiting
people
Meeting at the library ? Why not at Amazon ?
Meeting at the News Kiosk ? Why not at
BBC.com ?
Looking for a VAIO ? Ask people visiting
Sony.com !
-
8/14/2019 Emotion-based multimedia browsing (R&D Proposals)
22/38
Today Internet ExperienceToday Internet Experience
-
8/14/2019 Emotion-based multimedia browsing (R&D Proposals)
23/38
Internet: A Virtual meeting PlaceInternet: A Virtual meeting Place
Anybody knows S. Lew ?
Did you receive thebook you ordered ?
I did, but I didnt likeit !
-
8/14/2019 Emotion-based multimedia browsing (R&D Proposals)
24/38
Existing SolutionsExisting Solutions
http://www.groove.net/
Business oriented
virtual shared spaces
Business application already available
-
8/14/2019 Emotion-based multimedia browsing (R&D Proposals)
25/38
-
8/14/2019 Emotion-based multimedia browsing (R&D Proposals)
26/38
Agent Technology forAgent Technology for
Entertainment ServicesEntertainment Services
A system for multimedia commerce
by A. Barletta, B. Moser
-
8/14/2019 Emotion-based multimedia browsing (R&D Proposals)
27/38
The problemThe problem
How to send digital contents ?
How to sell digital content ?
How to exchange digital content ? Which encoding mechanism ?
A Agent framework can solve this problem
-
8/14/2019 Emotion-based multimedia browsing (R&D Proposals)
28/38
Computing ParadigmsComputing Paradigms
Client/Server
HTTP Server, FTP, etc
Remote execution
postscript
Mobile code
Applet
Agents
Aglets
-
8/14/2019 Emotion-based multimedia browsing (R&D Proposals)
29/38
Client/Server Paradigm
Agent Paradigm
-
8/14/2019 Emotion-based multimedia browsing (R&D Proposals)
30/38
Agent ParadigmAgent Paradigm
Reduce terminal-server interactions
Embedded and Proactive Autonomous Behaviors
[Data + Code] is moved to each node of the network
Several prototype solutions for e-Business and networkmanagement
Bartering, Searching, Monitoring, etc
-
8/14/2019 Emotion-based multimedia browsing (R&D Proposals)
31/38
AgEntAgEnt : Agent + Entertainment: Agent + Entertainment
An Agent Distribution System for Audio
and Video Contents
-
8/14/2019 Emotion-based multimedia browsing (R&D Proposals)
32/38
An Agent Digest Distribution SystemAn Agent Digest Distribution System
for Audio and Video Contentsfor Audio and Video Contents Autonomous software agents deliver audio and
video digest
Expiration of contents
Close system (security, proprietary encoder)
Flexible: the terminals can be updated automatically
The full chain is under control of the distributor
Content formatting, distribution, decoding
The user control only the selection of the contents(Simple interface: select, try, buy, play)
-
8/14/2019 Emotion-based multimedia browsing (R&D Proposals)
33/38
System ViewSystem View
Music/Video Agent:- responsible of rendering the content and looking forthe right decoder-Encryption logic (public key, private key)- maintain expiration/control (without calling back the
media server)- in case of purchasing it connects via a secureconnection to the server and retrieve/load/displaycontents- User has not access to the contents itself- It manages group of friends to exchange
Personal Agents Box (PAB)- Under controlled of Sony- provide support formanaging/loading/
activating/destroy Agentsmodule- upgradable decoders- manage security issues
Agent Distribution System (ADS):- it produces Active Agents- it can use a proprietary Sonyprotocol for distributing (usertransparent)- it uses a Sony encodersynchronized with the decodersin the PABs
Music Agent- it contains code and the
digest data
-
8/14/2019 Emotion-based multimedia browsing (R&D Proposals)
34/38
Use case (I)Use case (I)1
2
3
4
Digest
Song
Agent Distribution System(ADS)
Personal Agents Box (PAB)
-
8/14/2019 Emotion-based multimedia browsing (R&D Proposals)
35/38
Use case (II)Use case (II)
Antonio has got a VAIO system with a Personal Agents Box (PAB) applicationprovided by Sony
Antonio navigates in a Sony Audio Web sites and he looks for new songs
The Sony Web site offers an Agent Distribution System based on agents
Antonio chooses a collection of MusicAgents he likes [2]
Antonio decides to buy the song: after payment transaction, the content isdownloaded inside the agent, and played. Antonio unplug the VAIO from thenetwork and enjoy the music; after an expiration time the agent disables theplayback and destroy the content; only the digest is available [3].
Antonio decide to send this Agent to a friend, Boris (only the agent with thedigest is sent) [4]
Boris open the agent, listen in the digest and decide to play the song: the agent
after a payment transaction, download the content from the Sony Web site andplays the song
-
8/14/2019 Emotion-based multimedia browsing (R&D Proposals)
36/38
Secure Agent DistributionSecure Agent Distribution
Each Personal Agent Box has a private andpublic key
The Agent Distribution System (ADS)
receives the public key of each PAB
A group of friends can exchange the Publickeys
Transmission of Agent is secured using thepublic key of the receiving PAB
-
8/14/2019 Emotion-based multimedia browsing (R&D Proposals)
37/38
Secure Content DistributionSecure Content Distribution
Case 1:
Each MusicAgent has one key
Request of content with key
Server encode with Agent key
Agent decode with its own key
Case 2
Server has a private key
MusicAgent has a public key of the server An asymmetric encryption is done
-
8/14/2019 Emotion-based multimedia browsing (R&D Proposals)
38/38
ReferencesReferences
http://www.trl.ibm.com/aglets/index_e.htm
Aglets Home Page: IBM framework for building agents
No product available on the market from IBM