© Ramesh Jain 1 Fall 2008

© Ramesh Jain 1

Fall 2008

CS 212

Multimedia Systems and Applications

Instructor: Ramesh Jain

© Ramesh Jain 2

Projects Based on Research in Experiential Computing Laboratory

Based on the concept of EventWeb Two majore research projects

Personal Media Management Environment to Environment Connectivity (E2E)

© Ramesh Jain 3

Tim Berners-Lee thought:

Suppose all the information stored on computers everywhere were linked.

Suppose I could program my computer to create a space in which anything could be linked to anything.

Others – including Bush -- had that idea earlier but the technology was not ready.

© Ramesh Jain 4

That resulted in the Web

DocumentWeb Each node is a ‘Page’ or a document. Pages are linked through explicit referential links

© Ramesh Jain 5

Continuing the Evolution of the Web

Consider a Web in which each node Is an event Has informational as well as experiential data Is connected to other nodes using

Referential links Structural links Relational links Causal links

Explicit links can be created by anybody

This EventWeb is connected to other Webs.

© Ramesh Jain 6

Different Facets of an Event

What When Where Who

Why How

© Ramesh Jain 7

Tim

e

1- dimensional SpaceEvents Happen

© Ramesh Jain 8

Tim

e

1- dimensional SpaceEventWeb

© Ramesh Jain 9

Event Base

Atomic Event Entry

Stage 1: Event Ingestion

© Ramesh Jain 10

Stage 2: Event Clustering/Linking

Event Grouping, Linking, Assimilation

Event Base

Atomic Event Entry

© Ramesh Jain 11

Stage 3: Event Presentation, Navigation

Event Base

Event Presentation/ Navigation


Atomic Event Entry

© Ramesh Jain 12

Multimedia Storytelling

Collect information about events All relevant events Appropriate information In right media

Stories are sequence of events. Stories/Novels Drama Movies

Present right event information using right media in right order.

© Ramesh Jain 13

Tim

e

1- dimensional SpaceMultimedia Storytelling

Text

Video

Photo

© Ramesh Jain 14

Experiential Media Management Environment

Event-based Photo-taking is an event Photos are taken of an event Albums represents events and objects Presentations are to share and communicate

experiences

First Photos then other media.

© Ramesh Jain 15

EMME Event Cycle

Event Base



Atomic Event Entry

EXIF

Features

Tags/ Context

© Ramesh Jain 16

Modern Cameras

Are more than ‘Camera Obscura’. Many sensors capture scene context and

store it along with intensity values. EXIF data is all metadata related to the Event.

And we throw away all this valuable information!!!

© Ramesh Jain 17

Examples:Photos are Assigned tagsonly based onEXIF

© Ramesh Jain 18

EMME Event Cycle

Event Base



Atomic Event Entry

EXIF

Features

Tags/ Context

Photo stream Segment.Event

Ontology

UserAnnot-ations

© Ramesh Jain 19

Photo Stream Segmentation

Definition: given a photo stream }{ ipP =

Segment

Event 1 Event 2 Event 3

© Ramesh Jain 20

EMME Event Cycle

Event Base



Atomic Event Entry

EXIF

Features

Tags/ Context

Photo stream Segment.Event

Ontology

UserAnnot-ations

StoryTelling

Search

Explore

© Ramesh Jain 21

Using EMME

Searching for photo I2R visit

Creating Albums: Professional in Singapore Tourism in Summer 2008

Telling stories What did I do in Summer 2008

Scenario: In December 2008, I have 20,000 pictures taken in 2008. How do I (semi-automatically) select 25 to send to My mother The uncle that I hate My personal friend My professional friend …

© Ramesh Jain 22

Sentient EventWeb

Use sensors to create events. This class

Infrastructure to create and link all such events. Index Search Present

Archive events and put them on the Web.

© Ramesh Jain 23

First Version: E2E

Person to Person Device to Device

Telephone Video Conferencing

Environment to Environment (E2E)

Communication Modes:

© Ramesh Jain 24

Event Cycle

Event Base



Atomic Event Entry

© Ramesh Jain 25

E2E

Com

mun

icat

ion

Natural interaction

Semantic interaction

Seamlessinteraction

Bi-directional

connectivity

Not depend on physical similarity

Handle privacy

Event-based architecture

Scalable architecture

Sensor abstraction

Multimodal information

No fixed application

Live and archived modes

© Ramesh Jain 26

Environment

Fixed objects Active Objects Events and Activities

Sensor to capture Objects and Events Cameras, microphones, pressure, RFID, …

Displays and Actuators Projectors, Displays, Speakers, …

© Ramesh Jain 27

Key Idea

Put enough sensors and devices to make mediation (sensors and devices) disappear.

System detects events and objects and sends appropriate data streams and displays/renders incoming streams on appropriate devices

Bring back Person to Person communication

© Ramesh Jain 29

Node Architecture

Data Acquisition & Analysis

Situation Model

Env. Model

+Security settings

Event ServerMultimedia Database

Presentation

EventBase

Distribution/ Networking

Sensors

© Ramesh Jain 30

Event

e.g. E1_Cam2,E1_Mic2

e.g. Lip-motion=true, Volume=High

e.g. Env.1, Position: 102,145 Visitor Chair

e.g. Telemedicine App., Consultation room

e.g. Patient talking

Data Data Acquisition Acquisition & Analysis& Analysis

Env. Env. ModelModel

Situation Situation ModelModel

Event Detection Stages

© Ramesh Jain 34

Many Potential Projects: Photos Features to classify images:

Detect specific objects (sky, grass, mountains, …) and label images. Face detection and recognition. Person recognition – faces to other body information to person. Some simple ‘event’ (like a talk, or a wedding) recognition from

photographs. Using calendar information to assign information to images. Integrating Social Network Information In Personal Media Management Mobile App for uploading Event information. Can use Android APIs Select 1 or a few images from a group to represent the group. This

may be based on quality of images or content of images. Prepare a collage of images to represent a group. Use some theme for

selecting and displaying your images. Voice annotation to photos and use of speech recognition to convert

voice tags to text tags. Summarize an Event from Photos from many sources.

Many people shoot a photo for an event. How to create a good event representation from photos from multiple sources?

© Ramesh Jain 35

Many Potential Projects: E2E Tracking people using multiple sensors. Mobile E2E mode: switching from a mobile

phone to a much richer environment. Creating ‘virtual camera’ for heterogeneous

camera connections. Anywhere projector IP Audio streaming: Send and receive audio

by just specifying the IP address – without knowing the devices connected there.

© Ramesh Jain 1 Fall 2008

Documents

Transcript of © Ramesh Jain 1 Fall 2008