A Semantic-based Middleware for Multimedia Collaborative Applications Agustín J. González Advisor:...
-
Upload
kya-fender -
Category
Documents
-
view
216 -
download
0
Transcript of A Semantic-based Middleware for Multimedia Collaborative Applications Agustín J. González Advisor:...
![Page 1: A Semantic-based Middleware for Multimedia Collaborative Applications Agustín J. González Advisor: Dr. Hussein Abdel-Wahab Doctoral Dissertation Defense.](https://reader033.fdocuments.us/reader033/viewer/2022051515/55178a6055034645368b5451/html5/thumbnails/1.jpg)
A Semantic-based Middleware for Multimedia Collaborative Applications
Agustín J. GonzálezAdvisor: Dr. Hussein Abdel-
WahabDoctoral Dissertation Defense
Old Dominion University
February 2000
![Page 2: A Semantic-based Middleware for Multimedia Collaborative Applications Agustín J. González Advisor: Dr. Hussein Abdel-Wahab Doctoral Dissertation Defense.](https://reader033.fdocuments.us/reader033/viewer/2022051515/55178a6055034645368b5451/html5/thumbnails/2.jpg)
2
Outline
Introduction
MiddlewareObjectives
Extension of operating systems network services
Stream synchronization
Floor control framework
Protocol for dynamic image transmission
Experimental results
ConclusionsQuestions
![Page 3: A Semantic-based Middleware for Multimedia Collaborative Applications Agustín J. González Advisor: Dr. Hussein Abdel-Wahab Doctoral Dissertation Defense.](https://reader033.fdocuments.us/reader033/viewer/2022051515/55178a6055034645368b5451/html5/thumbnails/3.jpg)
3
Introduction • Large-scale Multimedia Applications
* Desktop computer performance increase
* Internet growth in bandwidth and # of hosts
• A challenging class of applications* Processing power & bandwidth
* Scalability
* Heterogeneity (Ethernet/modem, WinNT/Solaris, MPEG/H263)
* Timely data delivery
• Traditional services* Network layer: UDP & TCP (real time was not a concern)
* Operating systems: Abstractions are not adequate for multimedia.» Example: Real time is not well supported.
• Gap between multimedia requirements and system services
more
more
more Outline
![Page 4: A Semantic-based Middleware for Multimedia Collaborative Applications Agustín J. González Advisor: Dr. Hussein Abdel-Wahab Doctoral Dissertation Defense.](https://reader033.fdocuments.us/reader033/viewer/2022051515/55178a6055034645368b5451/html5/thumbnails/4.jpg)
4
Multimedia Resource Requirements
Bandwidth
Processing
Quality
Q1Q2
Q3
CPU performance
![Page 5: A Semantic-based Middleware for Multimedia Collaborative Applications Agustín J. González Advisor: Dr. Hussein Abdel-Wahab Doctoral Dissertation Defense.](https://reader033.fdocuments.us/reader033/viewer/2022051515/55178a6055034645368b5451/html5/thumbnails/5.jpg)
5
Source: Dr. David Patterson University of Virginia Distinguished Lecture Series, May 19,1998. http://www.cs.berkeley.edu/~pattrsn/talks/Stanford.pdf
Processor Performance Increase
3
Processor Performance Increase
µProc60%/yr.
1
10
100
10001980
1981
1983
1984
1985
1986
1987
1988
1989
1990
1991
1992
1993
1994
1995
1996
1997
1998
1999
2000
CPU
1982
Performance
Time
Effect on MM
![Page 6: A Semantic-based Middleware for Multimedia Collaborative Applications Agustín J. González Advisor: Dr. Hussein Abdel-Wahab Doctoral Dissertation Defense.](https://reader033.fdocuments.us/reader033/viewer/2022051515/55178a6055034645368b5451/html5/thumbnails/6.jpg)
6
Multimedia Resource Requirements
Bandwidth
Processing
MultimediaRequirements
Quality
Q1Q2
Q3
Bandwidth
![Page 7: A Semantic-based Middleware for Multimedia Collaborative Applications Agustín J. González Advisor: Dr. Hussein Abdel-Wahab Doctoral Dissertation Defense.](https://reader033.fdocuments.us/reader033/viewer/2022051515/55178a6055034645368b5451/html5/thumbnails/7.jpg)
7
Multimedia Resource Requirements
Bandwidth
Processing
MultimediaRequirements
Quality
Q1Q2
Q3
High processing + high bandwidth + Others
Introduction
![Page 8: A Semantic-based Middleware for Multimedia Collaborative Applications Agustín J. González Advisor: Dr. Hussein Abdel-Wahab Doctoral Dissertation Defense.](https://reader033.fdocuments.us/reader033/viewer/2022051515/55178a6055034645368b5451/html5/thumbnails/8.jpg)
8
Internet Growth
56 M
Introduction
![Page 9: A Semantic-based Middleware for Multimedia Collaborative Applications Agustín J. González Advisor: Dr. Hussein Abdel-Wahab Doctoral Dissertation Defense.](https://reader033.fdocuments.us/reader033/viewer/2022051515/55178a6055034645368b5451/html5/thumbnails/9.jpg)
9
Traditional requirementsTrad. Services
Trad. Application
Gap between system services and application requirements
Developers need to fill this gap by implementing common services for multimedia applications.
Real-timeScalabilityHeterogeneity
Multimedia applications
Middleware
outline
![Page 10: A Semantic-based Middleware for Multimedia Collaborative Applications Agustín J. González Advisor: Dr. Hussein Abdel-Wahab Doctoral Dissertation Defense.](https://reader033.fdocuments.us/reader033/viewer/2022051515/55178a6055034645368b5451/html5/thumbnails/10.jpg)
10
Objective “Our main objective is to investigate and propose
heterogeneous, scalable, reliable, flexible, and reusable solutions and enhancements to common needs in developing multimedia collaborative applications.”
Needs we addressed:
* Extension of network services
* Media synchronization
* Floor control
* Data sharing Outline
![Page 11: A Semantic-based Middleware for Multimedia Collaborative Applications Agustín J. González Advisor: Dr. Hussein Abdel-Wahab Doctoral Dissertation Defense.](https://reader033.fdocuments.us/reader033/viewer/2022051515/55178a6055034645368b5451/html5/thumbnails/11.jpg)
11
Extension of Network Services
• New services* Asynchronous data reception
* Quality of service monitoring
* Transmission traffic rate control
• New convenient facilities* Unified Multicast/Unicast API
* Efficient buffer management for Application Data Unit more
more
more
more
more
Outline
![Page 12: A Semantic-based Middleware for Multimedia Collaborative Applications Agustín J. González Advisor: Dr. Hussein Abdel-Wahab Doctoral Dissertation Defense.](https://reader033.fdocuments.us/reader033/viewer/2022051515/55178a6055034645368b5451/html5/thumbnails/12.jpg)
12
Asynchronous data reception
Network packet arrivals
packet
Thread watching
packet“application function”
Ext. Net. Srvs
Event-driven model
![Page 13: A Semantic-based Middleware for Multimedia Collaborative Applications Agustín J. González Advisor: Dr. Hussein Abdel-Wahab Doctoral Dissertation Defense.](https://reader033.fdocuments.us/reader033/viewer/2022051515/55178a6055034645368b5451/html5/thumbnails/13.jpg)
13
Quality of services monitoring
Traffic monitoring
packet packet“application function”
3,67 Bps789
2,15 Bps789
packet packet“application function”
Ext. Net. Srvs
ti-3 ti
si
Time
Packet Size
ti-1ti-2 t
Window k=3
ki
i
kijj
k tt
s
tSTTR
1)(
![Page 14: A Semantic-based Middleware for Multimedia Collaborative Applications Agustín J. González Advisor: Dr. Hussein Abdel-Wahab Doctoral Dissertation Defense.](https://reader033.fdocuments.us/reader033/viewer/2022051515/55178a6055034645368b5451/html5/thumbnails/14.jpg)
14
Transmission traffic rate control
Ext. Net. Srvs
ti-2 ti-1
si
Time
Packet Size
t`i ti
Send() call Actual Tx time
packet
i
2,15 Bps789
packet“application function”
Traffic Limit2,200 Bps
i-1
![Page 15: A Semantic-based Middleware for Multimedia Collaborative Applications Agustín J. González Advisor: Dr. Hussein Abdel-Wahab Doctoral Dissertation Defense.](https://reader033.fdocuments.us/reader033/viewer/2022051515/55178a6055034645368b5451/html5/thumbnails/15.jpg)
15
Unified Multicast/Unicast API
• Datagram transmission* A send to a machine or a multicast group does not make a difference.
• Datagram reception* if the given IP address is a multicast, join group.
* if address is not multicast, do not bind (I’m client).
port,addressUnicast
addressMulticast
Ext. Net. Srvs
![Page 16: A Semantic-based Middleware for Multimedia Collaborative Applications Agustín J. González Advisor: Dr. Hussein Abdel-Wahab Doctoral Dissertation Defense.](https://reader033.fdocuments.us/reader033/viewer/2022051515/55178a6055034645368b5451/html5/thumbnails/16.jpg)
16
Efficient buffer management for Application Data Unit
A B
AH BH
Tx Rx
BA
BHAH
* Goal: to prevent payload movements in memory
* Sender modules create an output buffer that can hold following “headers” and “tails” .
* Receiver module needs to allocate worst case buffer size.
Ext. Net. Srvs
![Page 17: A Semantic-based Middleware for Multimedia Collaborative Applications Agustín J. González Advisor: Dr. Hussein Abdel-Wahab Doctoral Dissertation Defense.](https://reader033.fdocuments.us/reader033/viewer/2022051515/55178a6055034645368b5451/html5/thumbnails/17.jpg)
17
Stream Synchronization• Problem: processing times and network delays are not
deterministic.
• The objective of synchronization is to faithfully reconstruct the temporal relationship between events (“pieces of data”).
• Main characteristics of our solution:* It depends on one-way messages only
» No need of feedback
* It only requires sender’s and receivers’ clock rates to be constant. » These clocks might be off.
» These clocks might even have different rates of change.
» No need of globally synchronized clocks
* It supports policies to handle late packets and delay adjustments.
Details
![Page 18: A Semantic-based Middleware for Multimedia Collaborative Applications Agustín J. González Advisor: Dr. Hussein Abdel-Wahab Doctoral Dissertation Defense.](https://reader033.fdocuments.us/reader033/viewer/2022051515/55178a6055034645368b5451/html5/thumbnails/18.jpg)
18
Stream Synchronization (details)
• Time model• Intra-stream synchronization• Inter-stream synchronization• Clock skew estimation and removal
more
more
more
more
Outline
![Page 19: A Semantic-based Middleware for Multimedia Collaborative Applications Agustín J. González Advisor: Dr. Hussein Abdel-Wahab Doctoral Dissertation Defense.](https://reader033.fdocuments.us/reader033/viewer/2022051515/55178a6055034645368b5451/html5/thumbnails/19.jpg)
19
Time ModelVirtual Observer
Network
Original Phenomenon
Capture
TimestampingEqualization
Buffer
Remote view
Playout
tiai
ciqi pi
Virtual Observer’smachine time
Receivertime
Absolute time
ci
i
pi
ti
TransmissionPath
Sync
![Page 20: A Semantic-based Middleware for Multimedia Collaborative Applications Agustín J. González Advisor: Dr. Hussein Abdel-Wahab Doctoral Dissertation Defense.](https://reader033.fdocuments.us/reader033/viewer/2022051515/55178a6055034645368b5451/html5/thumbnails/20.jpg)
20
ci
pi
ai
Receiver
Sender
ti-1 ti
ai-1 ai
cici-1
qi-1
qi
pi-1 pi
Perception
Timestamping
Arrival
Equalization
Playout
ci-1
pi-1
Intra-stream Synchronization (model)“Seen” by virtual observer
“Seen” by receiver
ctecp ii Synchronization condition:
Virtual delay
cte
Solution
Tradeoff:% late packets
Total delay
Adaptive compromise
![Page 21: A Semantic-based Middleware for Multimedia Collaborative Applications Agustín J. González Advisor: Dr. Hussein Abdel-Wahab Doctoral Dissertation Defense.](https://reader033.fdocuments.us/reader033/viewer/2022051515/55178a6055034645368b5451/html5/thumbnails/21.jpg)
21
Intra-stream Synchronization (solution)
Adjust “virtual delay” to achieve a given % of late packets
otherwise 0
arrival latefor 1*11 ii ll
NASA MBone 1% late packets
sync
more
Slow start1 min !
Estimator for % of late packets:
![Page 22: A Semantic-based Middleware for Multimedia Collaborative Applications Agustín J. González Advisor: Dr. Hussein Abdel-Wahab Doctoral Dissertation Defense.](https://reader033.fdocuments.us/reader033/viewer/2022051515/55178a6055034645368b5451/html5/thumbnails/22.jpg)
22
Fast start refinement
Less than 5 s ! sync
![Page 23: A Semantic-based Middleware for Multimedia Collaborative Applications Agustín J. González Advisor: Dr. Hussein Abdel-Wahab Doctoral Dissertation Defense.](https://reader033.fdocuments.us/reader033/viewer/2022051515/55178a6055034645368b5451/html5/thumbnails/23.jpg)
23
Inter-stream Synchronization
Global synchronization model v/s Differentiated synchronization model
Synchronizes streams coming from one virtual observer
Actual network delay Global Sync Model Differentiated
Synchronizes streams coming from anywhere with worst case delay
Solution
![Page 24: A Semantic-based Middleware for Multimedia Collaborative Applications Agustín J. González Advisor: Dr. Hussein Abdel-Wahab Doctoral Dissertation Defense.](https://reader033.fdocuments.us/reader033/viewer/2022051515/55178a6055034645368b5451/html5/thumbnails/24.jpg)
24
Inter-stream Synchronization (solution)
AudioSync
DataSyncDataSync
VideoSyncInter-stream
coordinatorVirtual delay
Max. virtual delay
sync
![Page 25: A Semantic-based Middleware for Multimedia Collaborative Applications Agustín J. González Advisor: Dr. Hussein Abdel-Wahab Doctoral Dissertation Defense.](https://reader033.fdocuments.us/reader033/viewer/2022051515/55178a6055034645368b5451/html5/thumbnails/25.jpg)
25
Clock skew estimation and removal
Goal: Remove differences in clock frequencies
Correction
The algorithm adjusts a straight line as new packets arrive
sync
Before After
![Page 26: A Semantic-based Middleware for Multimedia Collaborative Applications Agustín J. González Advisor: Dr. Hussein Abdel-Wahab Doctoral Dissertation Defense.](https://reader033.fdocuments.us/reader033/viewer/2022051515/55178a6055034645368b5451/html5/thumbnails/26.jpg)
26
Active Resource
Everywhere Resource
Inactive resourceNode (participant)
Localized Resource
“Audio” “Shared tool”
Lightweight Framework for Floor Control
• Problem: How to manage exclusive resources in large-scale multimedia applications?
• We recognize two cases:
ResourceUserUser Resource Communication Channel
1:1
:
nn
Solution
![Page 27: A Semantic-based Middleware for Multimedia Collaborative Applications Agustín J. González Advisor: Dr. Hussein Abdel-Wahab Doctoral Dissertation Defense.](https://reader033.fdocuments.us/reader033/viewer/2022051515/55178a6055034645368b5451/html5/thumbnails/27.jpg)
27
Floor Control (Solution)
* We propose two protocols for floor control, one per architecture.
* Features: lightweight, scalable, robust
(1) Request(2) Granted
(2) Taken
(1) Request
(3) Granted
Preemptive
Coordinator Floor holder Participant
(1) Request(2) Release
(4) Granted
(3) Taken or Released
TCP connectionHeartbeat
Delayedpreemptive
* The coordinator is stationary for localized resources.
* The coordinator migrates with floor for everywhere resources
Localized res.
![Page 28: A Semantic-based Middleware for Multimedia Collaborative Applications Agustín J. González Advisor: Dr. Hussein Abdel-Wahab Doctoral Dissertation Defense.](https://reader033.fdocuments.us/reader033/viewer/2022051515/55178a6055034645368b5451/html5/thumbnails/28.jpg)
28
Architecture for localized resources
Requester Control
Requester
RequesterListener
Coordinator
RequestReleased
HolderRefreshTakenGrantedRelease
GrantedRelease
RequestReleased
Withdrew
GrantedTakenRelease
NewHolder
getResourceInfo
Object implementing interface xObject related with floor architectureMain floor architecture objectsOptional Object
x
*
NewHolderListener*
ResourceInfo *
ResourceUserListener*
1-1 reliable remote invocation1-N unreliable remote invocationLocal invocation
Monitor/LogListener
*
log
Monitor/LogListener
*
log
HeartbeatHeartbeat
Policy
RequestNotiWithdrawalNotiHolderTimeoutSelectNextHolder
Everywhere res.
Outline
![Page 29: A Semantic-based Middleware for Multimedia Collaborative Applications Agustín J. González Advisor: Dr. Hussein Abdel-Wahab Doctoral Dissertation Defense.](https://reader033.fdocuments.us/reader033/viewer/2022051515/55178a6055034645368b5451/html5/thumbnails/29.jpg)
29
Architecture for everywhere resources
Requester Control
Requester
RequesterListener
RequestReleased
HolderRefreshTakenGrantedRelease
GrantedRelease
RequestWithdrew
Granted
getResourceInfo
ResourceListener*
1-1 Temporary connection for reliable remote invocation1-N unreliable remote invocationLocal invocation
Monitor/LogListener*
log
Monitor/LogListener*
log
Heartbeat
RequestNotiWithdrawalNotiHolderTimeoutSelectNextHolder
Coordinator
Policy
ActivateRequestWithdrewReleased
GrantedTakenRelease
Coordinator
Other Objects(Same Architecture as above)
To all Requesters
Before Granted
After Granted
Heartbeat
Outline
![Page 30: A Semantic-based Middleware for Multimedia Collaborative Applications Agustín J. González Advisor: Dr. Hussein Abdel-Wahab Doctoral Dissertation Defense.](https://reader033.fdocuments.us/reader033/viewer/2022051515/55178a6055034645368b5451/html5/thumbnails/30.jpg)
30
Protocol for Dynamic Image Transmission
• Problem: In addition to audio and video, multimedia sessions needs a component to convey the main idea of discussion.
• Traditional solutions: * Use video (size limitation & high bandwidth)
* Shared tools: XTV, co-browsers, VNC,.. (do not scale well)
• Our solution:
* Video-like protocol tuned to send dynamic images
Solution
![Page 31: A Semantic-based Middleware for Multimedia Collaborative Applications Agustín J. González Advisor: Dr. Hussein Abdel-Wahab Doctoral Dissertation Defense.](https://reader033.fdocuments.us/reader033/viewer/2022051515/55178a6055034645368b5451/html5/thumbnails/31.jpg)
31
Protocol for Dynamic Image Transmission
• Sender:* Temporal redundancy removal
» Sample image at regular period
» Divide image in tiles
» Process only changed tiles
* Spatial redundancy removal» compress and send changed tiles
• Receiver:» Receive data unit
» Decompress tile
» Update tile in image
Losses?
![Page 32: A Semantic-based Middleware for Multimedia Collaborative Applications Agustín J. González Advisor: Dr. Hussein Abdel-Wahab Doctoral Dissertation Defense.](https://reader033.fdocuments.us/reader033/viewer/2022051515/55178a6055034645368b5451/html5/thumbnails/32.jpg)
32
Overcoming losses
• Each tile is retransmitted after a random time
• This also accommodates late comers
Performance Study* How to select a tile compression technique? (JPEG, GIF, PNG?)
* Is there a “best” tile size? What does it depend on?
* How often to sample the image?
* How can two tiles be compared efficiently?
* Maximum data transmission rate? What does it depend on?
Outline
![Page 33: A Semantic-based Middleware for Multimedia Collaborative Applications Agustín J. González Advisor: Dr. Hussein Abdel-Wahab Doctoral Dissertation Defense.](https://reader033.fdocuments.us/reader033/viewer/2022051515/55178a6055034645368b5451/html5/thumbnails/33.jpg)
33
Implementation and Experimental Results
• Implementation:* Network support implemented
* Synchronization: implemented and used with real RTP data in off-line analysis
* Floor control: partially implemented for localized resources
* Image protocol implemented
• Putting everything together: Odust* A prototypical sharing tool built on top of the middleware.
It uses:* Network support, floor control, dynamic image protocol, other
application specific modules.Odust
![Page 34: A Semantic-based Middleware for Multimedia Collaborative Applications Agustín J. González Advisor: Dr. Hussein Abdel-Wahab Doctoral Dissertation Defense.](https://reader033.fdocuments.us/reader033/viewer/2022051515/55178a6055034645368b5451/html5/thumbnails/34.jpg)
34
MulticastNetwork
User: EduardoOS: WinNT
User: RodrigoOS: WinNT
User: AgustínOS: Solaris
User: CeciliaOS: Solaris
Odust Description
Architecture
![Page 35: A Semantic-based Middleware for Multimedia Collaborative Applications Agustín J. González Advisor: Dr. Hussein Abdel-Wahab Doctoral Dissertation Defense.](https://reader033.fdocuments.us/reader033/viewer/2022051515/55178a6055034645368b5451/html5/thumbnails/35.jpg)
35
MulticastNetwork
User: EduardoOS: WinNT
User: RodrigoOS: WinNT
User: AgustínOS: Solaris
User: CeciliaOS: Solaris
Odust Description: Cecilia’s view
Architecture
UNIX
![Page 36: A Semantic-based Middleware for Multimedia Collaborative Applications Agustín J. González Advisor: Dr. Hussein Abdel-Wahab Doctoral Dissertation Defense.](https://reader033.fdocuments.us/reader033/viewer/2022051515/55178a6055034645368b5451/html5/thumbnails/36.jpg)
36
MulticastNetwork
User: EduardoOS: WinNT
User: RodrigoOS: WinNT
User: AgustínOS: Solaris
User: CeciliaOS: Solaris
Odust Description: Rodrigo’s view
Architecture
WinNT
![Page 37: A Semantic-based Middleware for Multimedia Collaborative Applications Agustín J. González Advisor: Dr. Hussein Abdel-Wahab Doctoral Dissertation Defense.](https://reader033.fdocuments.us/reader033/viewer/2022051515/55178a6055034645368b5451/html5/thumbnails/37.jpg)
37
MulticastNetwork
User: EduardoOS: WinNT
User: RodrigoOS: WinNT
User: AgustínOS: Solaris
User: CeciliaOS: Solaris
Odust Description: Eduardo’s view
Architecture
WinNT
![Page 38: A Semantic-based Middleware for Multimedia Collaborative Applications Agustín J. González Advisor: Dr. Hussein Abdel-Wahab Doctoral Dissertation Defense.](https://reader033.fdocuments.us/reader033/viewer/2022051515/55178a6055034645368b5451/html5/thumbnails/38.jpg)
38
MulticastNetwork
User: EduardoOS: WinNT
User: RodrigoOS: WinNT
User: AgustínOS: Solaris
User: CeciliaOS: Solaris
Odust Description: Agustín’s view
Architecture
UNIX
![Page 39: A Semantic-based Middleware for Multimedia Collaborative Applications Agustín J. González Advisor: Dr. Hussein Abdel-Wahab Doctoral Dissertation Defense.](https://reader033.fdocuments.us/reader033/viewer/2022051515/55178a6055034645368b5451/html5/thumbnails/39.jpg)
39
Odust Architecture
Capture andDynamic Compound
Image ProtocolSender
Dynamic CompoundImage Protocol
Receiver and Display
EventInjector Event
Capture
TokenManager
TokenClient
Application A
WinNT
NativeLibrary
Application A Receiver
Sharing Tool Receiver
Application A Sender
Sharing Tool Sender
ApplicationB’s View
ApplicationA’s View
JDesktop
Java VM
Mx Dx
Temporary TCPMulticast Method Invocation
c
a
d
e
g
b
f
h
i
j
k
l
mn
Outline
![Page 40: A Semantic-based Middleware for Multimedia Collaborative Applications Agustín J. González Advisor: Dr. Hussein Abdel-Wahab Doctoral Dissertation Defense.](https://reader033.fdocuments.us/reader033/viewer/2022051515/55178a6055034645368b5451/html5/thumbnails/40.jpg)
40
Conclusion• We observed the convenience of a middleware
Traditional requirementsTrad. Services
Trad. Application
Real-timeScalabilityHeterogeneity
Multimedia applications
Middleware
• Future work* Add more components
* Continue implementation
* Try new ideas (see thesis)
• It offers:* Multimedia network services
* Synchronization
* Floor control
* Dynamic image transmission
Outline