ELIS – Multimedia Lab Davy Van Deursen, Wim Van Lancker Requirements for automatic evaluation of...
-
Upload
bethanie-carson -
Category
Documents
-
view
213 -
download
1
Transcript of ELIS – Multimedia Lab Davy Van Deursen, Wim Van Lancker Requirements for automatic evaluation of...
![Page 1: ELIS – Multimedia Lab Davy Van Deursen, Wim Van Lancker Requirements for automatic evaluation of MF TCs.](https://reader036.fdocuments.us/reader036/viewer/2022072010/56649db05503460f94a9e66a/html5/thumbnails/1.jpg)
ELIS – Multimedia Lab
Davy Van Deursen, Wim Van Lancker
Requirements for automatic evaluation of MF TCs
![Page 2: ELIS – Multimedia Lab Davy Van Deursen, Wim Van Lancker Requirements for automatic evaluation of MF TCs.](https://reader036.fdocuments.us/reader036/viewer/2022072010/56649db05503460f94a9e66a/html5/thumbnails/2.jpg)
2
ELIS – Multimedia Lab
<Automatic evaluation framework for MF TCs><Davy Van Deursen, Wim Van Lancker>
• Parsing of a media fragment• Retrieval of the media fragment scenarios• retrieve the full media resource• UA performs mapping and sends HTTP byte range request• UA sends HTTP time/track range request
• with or without redirect …• with or without setup information …
• Interpretation/rendering of the HTTP response• play/visualize the media fragment
Testing a MF-aware UA
![Page 3: ELIS – Multimedia Lab Davy Van Deursen, Wim Van Lancker Requirements for automatic evaluation of MF TCs.](https://reader036.fdocuments.us/reader036/viewer/2022072010/56649db05503460f94a9e66a/html5/thumbnails/3.jpg)
3
ELIS – Multimedia Lab
<Automatic evaluation framework for MF TCs><Davy Van Deursen, Wim Van Lancker>
• Automatic testing of parsing+retrieval• catch HTTP requests from the UA• information necessary in the TC
• media fragment URI + corresponding HTTP request(s)
• conclusion: should be feasible
• Automatic testing of interpretation/rendering• computer vision algorithms -> lot of work, not feasible• manual testing (inspired on for example SVG test cases)
• create an artificial a/v resource enabling the manual validation of the result (e.g., show a clock for temporal fragment evaluation)
• http://ninsuna.elis.ugent.be/content/TC_spatial_25fps_with_audio.mp4
Testing a MF-aware UA
![Page 4: ELIS – Multimedia Lab Davy Van Deursen, Wim Van Lancker Requirements for automatic evaluation of MF TCs.](https://reader036.fdocuments.us/reader036/viewer/2022072010/56649db05503460f94a9e66a/html5/thumbnails/4.jpg)
4
ELIS – Multimedia Lab
<Automatic evaluation framework for MF TCs><Davy Van Deursen, Wim Van Lancker>
• Parsing of HTTP Range header• Extraction of the requested media fragment• server-triggered redirect (e.g., for tracks)• calculate byte-time/track mapping and return bytes
corresponding to the found byte range
• Automatic testing of parsing and extraction• catch response • information necessary in the TC
• Range header and corresponding response headers• we assume that the server is able to provide the correct
bytes according to its calculated byte ranges
Testing a MF-aware server
![Page 5: ELIS – Multimedia Lab Davy Van Deursen, Wim Van Lancker Requirements for automatic evaluation of MF TCs.](https://reader036.fdocuments.us/reader036/viewer/2022072010/56649db05503460f94a9e66a/html5/thumbnails/5.jpg)
5
ELIS – Multimedia Lab
<Automatic evaluation framework for MF TCs><Davy Van Deursen, Wim Van Lancker>
• Automatic evaluation of the TCs demands detailed TC inputs• existing media resource• media fragment (e.g., t=10,20)• possible MF-specific HTTP request headers (i.e., Range
and Accept-Range-Redirect) combinations• possible MF-specific HTTP response headers (i.e.,
Content-Range-Mapping)
Conclusions
Server validation
UA validation
![Page 6: ELIS – Multimedia Lab Davy Van Deursen, Wim Van Lancker Requirements for automatic evaluation of MF TCs.](https://reader036.fdocuments.us/reader036/viewer/2022072010/56649db05503460f94a9e66a/html5/thumbnails/6.jpg)
ELIS – Multimedia Lab
End2End examples
…
incompliant behavior
compliant behavior, should be specified in the TC
not sure if compliant or not
misuse of MF-specific headers
![Page 7: ELIS – Multimedia Lab Davy Van Deursen, Wim Van Lancker Requirements for automatic evaluation of MF TCs.](https://reader036.fdocuments.us/reader036/viewer/2022072010/56649db05503460f94a9e66a/html5/thumbnails/7.jpg)
7
ELIS – Multimedia Lab
<Automatic evaluation framework for MF TCs><Davy Van Deursen, Wim Van Lancker>
Valid temporal example
t=10UA
get m.ogv
m.ogvget m.ogv
t=10-
get m.ogvbytes=349-
get m.ogv…
get m.ogvt=10- / redirect
Server
206, t=9-
307, bytes=349-
…
…
200
307, m.ogv?t=10
Server
200
307, m.ogv?t=10
Cases to add: include-setup
![Page 8: ELIS – Multimedia Lab Davy Van Deursen, Wim Van Lancker Requirements for automatic evaluation of MF TCs.](https://reader036.fdocuments.us/reader036/viewer/2022072010/56649db05503460f94a9e66a/html5/thumbnails/8.jpg)
8
ELIS – Multimedia Lab
<Automatic evaluation framework for MF TCs><Davy Van Deursen, Wim Van Lancker>
UA-detectable ‘invalid’ temporal example
t=fooUA
get m.ogv
m.ogvget m.ogv
include-setup
get m.ogvbytes=0-22
get m.ogv…
Server
206, include-setup
…
200
Cases to add: redirect
![Page 9: ELIS – Multimedia Lab Davy Van Deursen, Wim Van Lancker Requirements for automatic evaluation of MF TCs.](https://reader036.fdocuments.us/reader036/viewer/2022072010/56649db05503460f94a9e66a/html5/thumbnails/9.jpg)
9
ELIS – Multimedia Lab
<Automatic evaluation framework for MF TCs><Davy Van Deursen, Wim Van Lancker>
UA-undetectable ‘invalid’ temporal example
UA
get m.ogv
m.ogv
get m.ogvt=20-;incl-setup
get m.ogv…
Server
???
Cases to add: redirect
t=20
length of m.ogv is 15s
get m.ogvt=20-
get m.ogvincl-setup Server 206, include-setup
…
200
![Page 10: ELIS – Multimedia Lab Davy Van Deursen, Wim Van Lancker Requirements for automatic evaluation of MF TCs.](https://reader036.fdocuments.us/reader036/viewer/2022072010/56649db05503460f94a9e66a/html5/thumbnails/10.jpg)
10
ELIS – Multimedia Lab
<Automatic evaluation framework for MF TCs><Davy Van Deursen, Wim Van Lancker>
Spatial example
xywh=2,2,20,20
UA
get m.ogv
m.ogvget m.ogv
xywh=2,2,20,20 Server
…
200
307, m.ogv?xywh=2,2,20,20
get m.ogv…
![Page 11: ELIS – Multimedia Lab Davy Van Deursen, Wim Van Lancker Requirements for automatic evaluation of MF TCs.](https://reader036.fdocuments.us/reader036/viewer/2022072010/56649db05503460f94a9e66a/html5/thumbnails/11.jpg)
11
ELIS – Multimedia Lab
<Automatic evaluation framework for MF TCs><Davy Van Deursen, Wim Van Lancker>
Valid track example
track=1UA
get m.ogv
m.ogvget m.ogv
track=1
get m.ogvbytes=…-…
get m.ogv…
get m.ogvtrack=1/redirect
Server
206, track=1
307, bytes=…-…
…
…
200
307, m.ogv?track=1
Server
200
307, m.ogv?track=1
Cases to add: include-setup
![Page 12: ELIS – Multimedia Lab Davy Van Deursen, Wim Van Lancker Requirements for automatic evaluation of MF TCs.](https://reader036.fdocuments.us/reader036/viewer/2022072010/56649db05503460f94a9e66a/html5/thumbnails/12.jpg)
12
ELIS – Multimedia Lab
<Automatic evaluation framework for MF TCs><Davy Van Deursen, Wim Van Lancker>
UA-undetectable ‘invalid’ track example
track=54UA
get m.ogv
m.ogv
get m.ogvinclude-setup
get m.ogvtrack=54
Server
206, include-setup
…
200
Server…
???
get m.ogvtrack=54;incl-setup
Server
…
206, include-setup
get m.ogvbytes=0-22
Cases to add: redirect
![Page 13: ELIS – Multimedia Lab Davy Van Deursen, Wim Van Lancker Requirements for automatic evaluation of MF TCs.](https://reader036.fdocuments.us/reader036/viewer/2022072010/56649db05503460f94a9e66a/html5/thumbnails/13.jpg)
13
ELIS – Multimedia Lab
<Automatic evaluation framework for MF TCs><Davy Van Deursen, Wim Van Lancker>
• TODOs• provide detailed, machine-readable TC descriptions
• and extend Corrib?
• write evaluation software • Questions
• which and how much media resources to use?• we can vary in container formats, codecs, codec profiles, codec settings (intra
frame period), …
• URI fragments vs. URI queries?• for the moment: only URI fragments• URI queries
• e.g., media.mp4?xywh=100,100,50,50 -> transcoding on server?
• Issues• what if the server chooses what it returns (not exactly the shortest
time range possible …)?
TODOs – questions – Issues