Doc.:IEEE 802.11-05/1194r0 Submission November 2005 Philip J. Corriveau - IntelSlide 1 Video Testing...
-
Upload
sherman-evans -
Category
Documents
-
view
213 -
download
0
Transcript of Doc.:IEEE 802.11-05/1194r0 Submission November 2005 Philip J. Corriveau - IntelSlide 1 Video Testing...
November 2005
Philip J. Corriveau - IntelSlide 1
doc.:IEEE 802.11-05/1194r0
Submission
Video Testing Methodology
Notice: This document has been prepared to assist IEEE 802.11. It is offered as a basis for discussion and is not binding on the contributing individual(s) or organization(s). The material in this document is subject to change in form and content after further study. The contributor(s) reserve(s) the right to add, amend or withdraw material contained herein.
Release: The contributor grants a free, irrevocable license to the IEEE to incorporate material contained in this contribution, and any modifications thereof, in the creation of an IEEE Standards publication; to copyright in the IEEE’s name any IEEE Standards publication even though it may include portions of this contribution; and at the IEEE’s sole discretion to permit others to reproduce in whole or in part the resulting IEEE Standards publication. The contributor also acknowledges and accepts that this contribution may be made public by IEEE 802.11.
Patent Policy and Procedures: The contributor is familiar with the IEEE 802 Patent Policy and Procedures <http:// ieee802.org/guides/bylaws/sb-bylaws.pdf>, including the statement "IEEE standards may include the known use of patent(s), including patent applications, provided the IEEE receives assurance from the patent holder or applicant with respect to patents essential for compliance with both mandatory and optional portions of the standard." Early disclosure to the Working Group of patent information that might be relevant to the standard is essential to reduce the possibility for delays in the development process and increase the likelihood that the draft publication will be approved for publication. Please notify the Chair <[email protected]> as early as possible, in written or electronic form, if patented technology (or technology under patent application) might be incorporated into a draft standard being developed within the IEEE 802.11 Working Group. If you have questions, contact the IEEE Patent Committee Administrator at <[email protected]>.
Authors:Name Company Address Phone email
Philip Corriveau Intel HF3-96 5200 NE Elam Young Pkwy, Hillsboro, OR. 97124 (503)-696-1837 [email protected]
Audrey Younkin Intel HF3-96 5200 NE Elam Young Pkwy, Hillsboro, OR. 97124 (503)-696-3947 [email protected]
Chris Olsen Intel HF3-96 5200 NE Elam Young Pkwy, Hillsboro, OR. 97124 (503)-696-7548 [email protected]
Fernald Royce Intel HF3-96 5200 NE Elam Young Pkwy, Hillsboro, OR. 97124 (503)-696-4318 [email protected]
Rik Logan Intel 5200 NE Elam Young Pkwy Hillsboro, OR 87124 (503)-712-1675 [email protected]
Uriel Lemberger Intel PO Box 1659, Matam Industrial Park, Haifa 31015 Israel +972-4-865-5701 [email protected]
Neeraj Sharma Intel 13290 Evening Creek Drive, San Diego, CA 92128 (858)-385-4112 [email protected]
Sasha Tolpin Intel PO Box 1659, Matam Industrial Park, Haifa 31015 Israel +972-4-865-5430 [email protected]
November 2005
Philip J. Corriveau - IntelSlide 2
doc.:IEEE 802.11-05/1194r0
Submission
Agenda
• Video Assessment
• GED Process Overview
• Purpose
• Objective Outputs
• Experimental Design
• Methodology Overview
• Summary
November 2005
Philip J. Corriveau - IntelSlide 3
doc.:IEEE 802.11-05/1194r0
Submission
• Philip Corriveau & Audrey Younkin• User Centered Design • Media and Acoustics Perception Lab
Intel Video Testing Methodology Concepts and Methods
• Olsen Christopher & Fernald Royce
• Platform Systems Technology
• Wireless Integration Team
November 2005
Philip J. Corriveau - IntelSlide 4
doc.:IEEE 802.11-05/1194r0
Submission
Video Assessment~Not a One Step Process~
• The Gross Error Detector (GED) is one of the first steps in assessing video experience. – The GED presents a simple and quick way to measure large errors,
such as dropped or repeated frames or video playback at an incorrect frame rate.
• The GED is designed to communicate an Experiential Value that can be directly correlated to a real end-user.
• Other tools like VQM, then can capture and measure video quality (artifacts).
November 2005
Philip J. Corriveau - IntelSlide 5
doc.:IEEE 802.11-05/1194r0
Submission
Why Use GED?
• The Video Gross Error Detector is a high-level video performance analysis application
• Determines whether a video stream arrives intact, in sequence and at the correct frame rate
• Enables automated, quantitative, repeatable measurements of video quality
• Significantly less expensive and more accurate than manual video quality assessment
November 2005
Philip J. Corriveau - IntelSlide 6
doc.:IEEE 802.11-05/1194r0
Submission
GED Process Overview
0. Source Material 1. Marked Source Material
2. Compressed and Marked
4. GED Analysis
GED Encode WMV Encode
System Under Test
3. Capture Results
GED Decode
November 2005
Philip J. Corriveau - IntelSlide 7
doc.:IEEE 802.11-05/1194r0
Submission
GED Encode
November 2005
Philip J. Corriveau - IntelSlide 8
doc.:IEEE 802.11-05/1194r0
Submission
Purpose
• Currently the Gross Error Detector (GED) measures video performance based of a set of objective metrics.
• However, this does not present the entire picture for video performance and assessment. For more accurate end user analysis, there needs to be a subjective mean opinion score (MOS) coupled with the objective metrics.
November 2005
Philip J. Corriveau - IntelSlide 9
doc.:IEEE 802.11-05/1194r0
Submission
Current Objective Outputs
• Dropped Frames
• Repeated Frames
• Delayed Frames
• Non-sequential Frames
Future Objective Outputs
• Audio Error Detection (end of Q4)
• AV Sync (end of Q4)
• Frame Hashing - detection of blur and video corruption (Q1 ’06)
November 2005
Philip J. Corriveau - IntelSlide 10
doc.:IEEE 802.11-05/1194r0
Submission
Experimental Design
• Currently the GED experimental design is broken into two Phases– Phase I: Expert Subjective Assessment
– Phase II: Non-Expert Subjective Assessment
November 2005
Philip J. Corriveau - IntelSlide 11
doc.:IEEE 802.11-05/1194r0
Submission
Phase I Expert Assessment
• Using a SD (720 x 486) compiled collection of 10 second clips adding up to a total of 321 seconds (~5 minutes).
• The GED tool will do an average over the 10 second clips with respect to metrics.
• The GED is independent of the codec, streaming server and hardware platform.
• Using Windows Media Services with 1-6 Mbps WMVs and the East Fork server with 1-8 Mbps MPEG2 SD files.
November 2005
Philip J. Corriveau - IntelSlide 12
doc.:IEEE 802.11-05/1194r0
Submission
General Methodology
• To create a Subjective Score (1-5) there needs to be an evaluation of what real life situations produce in respect to a wireless connection.
• The wireless path loss and other effects will cause different amounts of frames dropped/repeated that can be captured and compared to the original GED.
November 2005
Philip J. Corriveau - IntelSlide 13
doc.:IEEE 802.11-05/1194r0
Submission
Example of Expected Scoring Value
• However, the temporal factor also needs to be put in place, not only the number of dropped frames, but where they occur in the sequence and how long each error occurs for.
• Each clip is 260 frames long.
Expert Score
Number Frames
Dropped
Percentage Frames
Dropped
5 >4 1.5%
4 >10 3.8%
3 >50 19%
2 >100 38%
1 >200 77%
November 2005
Philip J. Corriveau - IntelSlide 14
doc.:IEEE 802.11-05/1194r0
Submission
Phase II Non-Expert Subjective Assessment
• Subjects:
40-70 non-expert external participants will rate video quality on a scale of 1-5 (half male/half female). Using Market Decisions Corporation for outsourcing of participants.
November 2005
Philip J. Corriveau - IntelSlide 15
doc.:IEEE 802.11-05/1194r0
Submission
General Methodology
• Based on the Expert Analysis, five new sequences (one for each score) will be developed based on the GED outputs and temporal factors.
• One or two participants per test session.
• Upon arrival to the study location, participants will be asked to read through a brief overview of the study consisting of a summary of the test procedure requirements and will be asked to sign a confidentiality agreement and consent form.
November 2005
Philip J. Corriveau - IntelSlide 16
doc.:IEEE 802.11-05/1194r0
Submission
How to Rate
• Prior to each session, participants will be instructed to rate each video sequence based on its real-time smoothness and flow.
• Do not rate experience based on content quality but rather the flow of presentation.
November 2005
Philip J. Corriveau - IntelSlide 17
doc.:IEEE 802.11-05/1194r0
Submission
Practice Trial
• A familiarization and practice trial will carried out to begin the test session.
• Here, participants will be shown the best and the worst quality for frame of reference.
November 2005
Philip J. Corriveau - IntelSlide 18
doc.:IEEE 802.11-05/1194r0
Submission
Randomization
• The presentation order of the video content will be randomized using a pseudo-random number generator tool to prevent ordering effects, i.e. to mask any tendency for a participant observer to rate a clip in relation to the previous one.
• Randomization is a key element of psycho-visual testing to ensure that participants do not see the material in a repeated fashion that would allow a learning effect.
November 2005
Philip J. Corriveau - IntelSlide 19
doc.:IEEE 802.11-05/1194r0
Submission
Voting Opinion
• After each video sequence, participants will be presented with a pop-up window requiring their input of their opinion of the video experience on a subjective rating scale. This scale ranged from excellent (numerical equivalent of five) to bad (numerical equivalent of one).
November 2005
Philip J. Corriveau - IntelSlide 20
doc.:IEEE 802.11-05/1194r0
Submission
Numerical Values
Value Presented to Participant Actual Numerical Value
Excellent 5
Good 4
Fair 3
Poor 2
Bad 1
November 2005
Philip J. Corriveau - IntelSlide 21
doc.:IEEE 802.11-05/1194r0
Submission
Participants will sit at a predetermined viewing distance
20°
5HH
November 2005
Philip J. Corriveau - IntelSlide 22
doc.:IEEE 802.11-05/1194r0
Submission
Post Test
• At the end of the testing session, participants will be asked to fill out a post test questionnaire based on their experience.
November 2005
Philip J. Corriveau - IntelSlide 23
doc.:IEEE 802.11-05/1194r0
Submission
Methodology Overview
Tutorial
Shown “Bad” Video
Shown “Excellent” Video
Practice Trial
Clips Randomized
Subjective Vote
Post-Test Survey
November 2005
Philip J. Corriveau - IntelSlide 24
doc.:IEEE 802.11-05/1194r0
Submission
Summary
• Video Assessment not a one step process.
• We presented one of the tools GED, which is quantitative, repeatable measurements of video quality.
• Experiments concepts were described that help to correlate subjective test results with objective measurements.
• Results from the experiment described within will be presented in the next meeting in January
November 2005
Philip J. Corriveau - IntelSlide 25
doc.:IEEE 802.11-05/1194r0
Submission
Next Steps
• Intel will correlate GED and Subjective data to Link (Phy and Mac) Layer statistics for 802.11
• Develop a model for 802.11 for Video Quality
• Deliver results at January meeting