Multimedia Synchronization Brian P. Bailey Spring 2006.
-
Upload
makayla-dagg -
Category
Documents
-
view
222 -
download
2
Transcript of Multimedia Synchronization Brian P. Bailey Spring 2006.
![Page 1: Multimedia Synchronization Brian P. Bailey Spring 2006.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649c755503460f94929901/html5/thumbnails/1.jpg)
Multimedia Synchronization
Brian P. BaileySpring 2006
![Page 2: Multimedia Synchronization Brian P. Bailey Spring 2006.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649c755503460f94929901/html5/thumbnails/2.jpg)
Announcements
![Page 3: Multimedia Synchronization Brian P. Bailey Spring 2006.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649c755503460f94929901/html5/thumbnails/3.jpg)
MM Synchronization
• Applications composed of more than one media (at least one continuous)
• Express desired relationships– content, spatial, temporal, and interaction– combinations of each
![Page 4: Multimedia Synchronization Brian P. Bailey Spring 2006.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649c755503460f94929901/html5/thumbnails/4.jpg)
Content and Spatial Relationships
• Content– define how views relate to data sources– e.g., a graph linked to a table of data
• Spatial– define relative positions of media objects– subdivide the space; express relationships– e.g., pack command in Tcl, layout managers
in Java, tables in HTML, etc.
![Page 5: Multimedia Synchronization Brian P. Bailey Spring 2006.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649c755503460f94929901/html5/thumbnails/5.jpg)
Temporal Relationships
• Define how media are coordinated in time– audio should not drift from video by > 80ms– voice narration should accompany a slide and
end when user navigates elsewhere– display different caption for each video scene;
and update it in response to user interaction
• Intra-media and inter-media relationships
• Time-independent and dependent media
![Page 6: Multimedia Synchronization Brian P. Bailey Spring 2006.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649c755503460f94929901/html5/thumbnails/6.jpg)
Lip Synchronization
Left: audio after video; Right: audio before video
![Page 7: Multimedia Synchronization Brian P. Bailey Spring 2006.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649c755503460f94929901/html5/thumbnails/7.jpg)
Lip Synchronization
Tolerable
Nottolerable
Notdetectable
Tolerable
Nottolerable
![Page 8: Multimedia Synchronization Brian P. Bailey Spring 2006.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649c755503460f94929901/html5/thumbnails/8.jpg)
Tele-pointer Synchronization
Left: pointer before audio; Right: pointer after audio
![Page 9: Multimedia Synchronization Brian P. Bailey Spring 2006.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649c755503460f94929901/html5/thumbnails/9.jpg)
Synchronization Guidelines
• Lip synchronization within 80ms– video before audio is more tolerable
• Other fine-grained synchronization should typically be within range of 500ms
![Page 10: Multimedia Synchronization Brian P. Bailey Spring 2006.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649c755503460f94929901/html5/thumbnails/10.jpg)
Interaction Relationships
• Define how interaction affects playback– e.g., if user transitions to next slide in narrated
slide show, narration should change as well
• Classes of interaction– navigation, participation, and control– asynchronous and synchronous
![Page 11: Multimedia Synchronization Brian P. Bailey Spring 2006.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649c755503460f94929901/html5/thumbnails/11.jpg)
Synchronization Model
• Enables expression of media and synchronization relationships
• An effective model should support:– spatial and temporal relations (fine & coarse)– rich interaction (beyond VCR control)– efficient runtime (interaction monitoring)– be usable and comprehensible
![Page 12: Multimedia Synchronization Brian P. Bailey Spring 2006.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649c755503460f94929901/html5/thumbnails/12.jpg)
Models
• Timeline
• Hierarchical
• Petri net
• Interval
• Event-based
• Common threads– provide language to
express relationships– runtime system to
monitor relationships– policies to enforce
relationships
![Page 13: Multimedia Synchronization Brian P. Bailey Spring 2006.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649c755503460f94929901/html5/thumbnails/13.jpg)
Timeline Model
• Uses a single global timeline
• Actions triggered when the time marker reaches a specific point along timeline
![Page 14: Multimedia Synchronization Brian P. Bailey Spring 2006.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649c755503460f94929901/html5/thumbnails/14.jpg)
Example
• Define a timed sequence of images, each image has a caption that goes with it
I1
C1
t1
I2
C2
t2
I3
C3
t3
![Page 15: Multimedia Synchronization Brian P. Bailey Spring 2006.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649c755503460f94929901/html5/thumbnails/15.jpg)
Example (Cont.)
• Rule language
– At (t1), show (I1, C1)– At (t2), show (I2, C2)– At (t3), show (I3, C3)
• Visual environment
![Page 16: Multimedia Synchronization Brian P. Bailey Spring 2006.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649c755503460f94929901/html5/thumbnails/16.jpg)
Hierarchical Model (SMIL)
• Based on sequential and parallel
• Apply operators to only the start/end points of each media object
I1 I2 I3 I1 T1
![Page 17: Multimedia Synchronization Brian P. Bailey Spring 2006.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649c755503460f94929901/html5/thumbnails/17.jpg)
Example
• Narrated slide show– image, text, audio on each slide– select link to move to the next slide
S1 A1 T1
…
I1 S2 A2 T2 I2
![Page 18: Multimedia Synchronization Brian P. Bailey Spring 2006.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649c755503460f94929901/html5/thumbnails/18.jpg)
Timed Petri Nets
• tokens, places, transitions, and arcs
![Page 19: Multimedia Synchronization Brian P. Bailey Spring 2006.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649c755503460f94929901/html5/thumbnails/19.jpg)
Example
11ms 11ms 11ms
33ms
11ms 11ms
33ms
Specify audio video synchronization
![Page 20: Multimedia Synchronization Brian P. Bailey Spring 2006.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649c755503460f94929901/html5/thumbnails/20.jpg)
Interval Model
• 13 relationships between two intervals
BeforeA B
MeetsA B
During A
B
Overlaps A
B
Starts A
B
Ends A
BEqual A
B
![Page 21: Multimedia Synchronization Brian P. Bailey Spring 2006.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649c755503460f94929901/html5/thumbnails/21.jpg)
• Associate actions with expressions
• Expressions may contain scalars, clocks, variables, relations, and connectives
• When the expression becomes TRUE, invoke associated action
When “Time > Q.end + 5 && !Response” Answer=WRONG
Event Model (Nsync)
![Page 22: Multimedia Synchronization Brian P. Bailey Spring 2006.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649c755503460f94929901/html5/thumbnails/22.jpg)
Background and Time Model
• Each media object attached to a clock
• Clock implements logical time– Value = Rate * System + Offset
• Express temporal behavior as relationships among clocks
• Interactive events tied to variables
![Page 23: Multimedia Synchronization Brian P. Bailey Spring 2006.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649c755503460f94929901/html5/thumbnails/23.jpg)
OverviewOverview
NoNo
YesYes
MoreMoreInfo?Info?
DetailedDetailedNarrationNarration
More InfoMore InfoMore InfoMore Info
Example: Delayed Transition
![Page 24: Multimedia Synchronization Brian P. Bailey Spring 2006.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649c755503460f94929901/html5/thumbnails/24.jpg)
Model Specification
When “Narration >= Overview && !MoreInfo” NextSlide
When “Narration >= Overview && MoreInfo” PlayDetails
When “Narration >= Overview + Details” NextSlide
Narration: narration’s logical timeline
Overview: normal transition point
Details: additional narrative details
MoreInfo: records kitchen info status
![Page 25: Multimedia Synchronization Brian P. Bailey Spring 2006.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649c755503460f94929901/html5/thumbnails/25.jpg)
Reactive Interface
![Page 26: Multimedia Synchronization Brian P. Bailey Spring 2006.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649c755503460f94929901/html5/thumbnails/26.jpg)
Model Specification
When “Video >= 0 && Video < T1” Select Kitchen
When “Video >= T1 && Video < T2” Select Deck
When “Video >= T2 && Video <= T3” Select Yard
![Page 27: Multimedia Synchronization Brian P. Bailey Spring 2006.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649c755503460f94929901/html5/thumbnails/27.jpg)
Expression Evaluation
• Propositional logic breaks down– returns logic value only at present time– requires polling to catch future transitions
• Predictive logic– returns logic value at present time along with
a prediction of any future transition– eliminates need for intermittent polling/timers
![Page 28: Multimedia Synchronization Brian P. Bailey Spring 2006.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649c755503460f94929901/html5/thumbnails/28.jpg)
Predictive Logic States
• WBT(t) False now, but Will Become Trueat future time t
• WBF(t) True now, but Will Become Falseat future time t
![Page 29: Multimedia Synchronization Brian P. Bailey Spring 2006.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649c755503460f94929901/html5/thumbnails/29.jpg)
Prediction Example
When “Video > 10” ActionWhen “Video > 10” Action
1010
Vid
eo T
ime
Vid
eo T
ime
Rate = 1
Rate = 1
(then - now)t = ----------------- rate
System TimeSystem Time
t = (10 - 0) / 1 = WBT(10)
00
![Page 30: Multimedia Synchronization Brian P. Bailey Spring 2006.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649c755503460f94929901/html5/thumbnails/30.jpg)
Prediction Example
When “Video > 10” ActionWhen “Video > 10” Action
00
1010
Vid
eo T
ime
Vid
eo T
ime
Rate = 1
Rate = 1
(then - now)? = ----------------- rate
System TimeSystem Time
? = (10 - 3) / 2 = WBT(3.5)
?
Rat
e =
2
Rat
e =
2
33
![Page 31: Multimedia Synchronization Brian P. Bailey Spring 2006.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649c755503460f94929901/html5/thumbnails/31.jpg)
Evaluation Rules for “AND”
WBT(x) && WBT(y) = WBT( max(x, y) )
WBF(x) && WBF(y) = WBF( min(x, y) )
WBF(x) && WBT(y) =
WBT(infinity) if (x < y)
WBT(y) then WBF(x) otherwise
![Page 32: Multimedia Synchronization Brian P. Bailey Spring 2006.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649c755503460f94929901/html5/thumbnails/32.jpg)
Take Home Exercise
WBT(x) | | WBT(y) = ?
WBF(x) | | WBF(y) = ?
WBF(x) | | WBT(y) = ?
![Page 33: Multimedia Synchronization Brian P. Bailey Spring 2006.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649c755503460f94929901/html5/thumbnails/33.jpg)
Pros
• Complements current languages– adds ability to express combinations of
interactive and temporal behavior– syntax can easily be translated into mark up
• Predictive logic useful in run-time engines– eliminates need for polling/timers– enables look-ahead pre-fetching
![Page 34: Multimedia Synchronization Brian P. Bailey Spring 2006.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649c755503460f94929901/html5/thumbnails/34.jpg)
Cons
• Difficult to visualize rule propagation– makes system difficult to debug
• Rules are not groups into hierarchies– enable divide and conquer strategy
• Lack of scope– all rules always active– guard actions with complex expressions
![Page 35: Multimedia Synchronization Brian P. Bailey Spring 2006.](https://reader036.fdocuments.us/reader036/viewer/2022062320/56649c755503460f94929901/html5/thumbnails/35.jpg)
Take Home Exercise
• Be able to model relationships within relatively simple applications
• Weigh tradeoffs between models