MOVIS – Participating Institutions...Logs for quality assurance. Media Stream Media feed Content...
Transcript of MOVIS – Participating Institutions...Logs for quality assurance. Media Stream Media feed Content...
www.nr.no
The MOVIS project
Perfomance Monitoring System for Video Streaming Networks
Wolfgang Leister
April 2006
www.nr.no
MOVIS – Participating InstitutionsResearch Project supported by Research Council of Norway
► TV2 Interaktiv, Bergen► Norsk Regnesentral, Oslo► IRT, München► Nextgentel, Sandsli► Lyse Tele, Stavanger► Never.no, Oslo► Secrets to Sports, Asker► Nettfokus, Lundesæter► Nimsoft, Oslo
www.nr.no
Media Stream Media feed
TV2 Content Production
Streaming server
Nextgentel Lyse
S2S Content P.
MOVIS – ScenarioUser agent Server agentNetwork agent
...
www.nr.no
The Challenge
► Content Providers distributes content to▪ End-users as multimedia streams▪ ISPs for redistribution
► The ISPs stream to end-users from own servers
► In case of degradations of QoS in delivery:▪ the end-user gets reduced visual quality.▪ Communicate problem before user reacts.▪ Help content providers / ISPs to identify problems.▪ Logs for quality assurance.
www.nr.no
Media Stream Media feed
Content Production Streaming server Network
MOVIS – Delivery ChainUser agent Server agent
•Streaming server•Encoding params•Media format
•Network •Decoding•User equipment
•Perceived QoS•Network•Encoding
•Technical QoS•Server dependent•Encoding dependent
•Technical quality •of original media feed
•Original quality
•Measure technical QoS•Mapping to potentially
perceived QoSwithout coding
•Measure technical QoS•Mapping to Perceived QoS
•Measure encoding QoS•Mapping to potentially
perceived QoSwithout network
•Define reference for max. available QoS
Network agent
End-to-End Measurement
www.nr.no
Streaming Content Distribution
Provider ...
Internet
Encoder
Encoder
Databases
Video File Server
Live Distrib. Server
Camera
Camera
AV Matrix
Video Source
...
Provider 1
Default Server
Provider 2
Provider 3
ISP Net
Encoders
www.nr.no
Infrastructure at ISP
Streaming Server
Cont Provider Default Server
File Server
Internet
Content Provider
Router /Switch
Router /Switch
Router /Switch
DSLAM
www.nr.no
End user equipment scenarios
Access point Router
WLAN
Viewing Characteristics
Digital TV
One PC
Home Network
WLAN
...
www.nr.no
Considerations for MOVIS
► Online measurements of end-user QoS for streamed content► Emphasis on end-to-end performance► Measure / collect perceived and calculated QoS► Report QoS values back to content providers and ISPs► Metrics, protocols, architectural issues► Measurements in all parts of delivery-chain
► End user is in mass market► End user interaction► End user equipment configuration beyond control of providers► Internal structures of ISPs must not be revealed
www.nr.no
How to solve problem?
► Research▪ Metrics for measuring video quality▪ Subjective / Objective assessment of video quality▪ Relations between technical and perceived QoS
► Development▪ Agent-based system to report perceived video quality
to ISP and content provider.▪ Interfacing with systems in use
► Evaluations▪ Field trial, user studies
www.nr.no
Test Methods
Original Content Encoder StreamingServer
TransmissionNetwork Decoder Processed Content
Feature Comparison Impairment Par.Feature Extraction
Feature Extraction
Original Content Encoder StreamingServer
TransmissionNetwork Decoder Processed Content
Monitoring System+ Model Impairment Par.
Original Content Encoder StreamingServer
TransmissionNetwork Decoder Processed Content
MeasurementSystem Obj.Quality Rating
Picture Comparison Test method
Feature Extraction Test Method
Single Ended Test method
www.nr.no
Characteristics of Test Methods
user – Subjective
picture – Directly
In-service
Real-time
Continuous
Objective – technical
Indirectly – signal
Out-of-service
Deferred time
Samples
vs
vs
vs
vs
vs
www.nr.no
QoS measurements / metrics
► Collecting QoS observations▪ Intrusive (controlled injection of content)
▪ Non-intrusive (passive observation)
► QoS metriccarefully specified quantity related to the performance and reliability of
the service that we would like to know the value of.
► Networking QoS metrics
► Perceived QoS▪ Laboratory▪ Real-time
www.nr.no
QoS measurements / metrics
► Perceived QoS metrics▪ Impact of networking characteristics▪ Impact of codec characteristics
► Assessment▪ Subjective assessment (end user evaluations)▪ Objective assessment (technically deducted)
► Methods▪ DVB: ETR 290, TR 101 290 (Measurement guidelines
for DVB systems), TR 101 291▪ BT.500▪ SAMVIQ – BT.700
www.nr.no
QoS measurements / metrics
► Networking QoS metrics▪ Connectivity, one-way delay, two-way delay,
throughput, loss, jitter, ...▪ Initiatives:
◦ IETF IPPM (IP Performance metrics)RFC 2330, RFC 2680, RFC 2681, IPPM, WOAMP, ...
◦ ITU-T G.107: Rating of transmission quality e2e.
► Nettfokus – mobile SLM▪ http://www.knowyourSLA.com/
► NimBUS▪ http://www.nimsoft.com/
www.nr.no
NIMSOFT
► Architecture built around message bus (NimBUS)
► All types of traffic / networks
► API libraries available
► Agent-based
www.nr.no
Firewall
Internet
enterprise networks
www.knowyoursla.com
Firewall
Master
Slave
Slave
Slave
Nettfokus / MobileSLM
► Management solution for SLM www.knowyoursla.com► Designed for multimedia traffic► Dedicated machines inject traffic► Agent-based
www.nr.no
APDEX
► Numerical Measure of end user satisfaction
► Ratings range from unacceptable to excellent
► Used for response times
► Three end-user categories▪ #Satisfied▪ #Tolerating▪ #Frustrated
www.nr.no
G.107 – E-model - VQM
► Need models for establishing relation between objective and subjective quality.
► G.107 / E-model is example for audio
► VQM for video▪ Document: NTIA-report 02-392 – Video Quality
Measurement Techniques by Pinson and Wolf. http://www.its.bldrdoc.gov/n3/video/documents.htm
▪ Developed by ITS / NTIA of the U.S. Dept. of Commerce
www.nr.no
G.107 / E-Model► Model for audio / IP-telephony► Uses impairment factors► Related standards:
▪ G.108▪ G.113▪ G.175▪ G.562▪ ...
www.nr.no
VQM – Video Quality Metric
► Objective measurement method
► How and what to measure
► Formulas to calculate VQM value
► Relation VQM ↔ subjective
www.nr.no
Encoder
Streaming server
Network
The MOVIS-factor MOriginal Content
Encoded Content
Cont Prov. Network
ISP Network
User Network
User Equipment
End User
MOVIS Ancilliary Channel
Viewing conditions (not part of MOVIS)
Streaming Server type, parameter settings, ...
Networking parameters, topology, type, ...
Type, parameters ...
Codec, parameter settings, encoder type, ...
MOVIS-factor M=
MO
-ME(Codec)
-MN(Codec)
-MU(Codec)
-MV(Cont)
+MA(...)
Advantage factor
-MS(Codec,Cont)
www.nr.no
Calculation of the MOVIS factor
M=M 0 I −M E C , I −M S C , P−M N C ,P−M U C ,P , E−M V I ,E M A...
The MOVIS Factor:
M=0∑ii mi∑
i , jij mi mj
Regression Formula using single influences, grouped into sub-factors
M=M 0 I ⋅M EC ,I ⋅M SC , P⋅M N C ,P⋅M U C ,P , E⋅M V I , E⋅M A...Alternative MOVIS Factor:
www.nr.no
Subjective VQ Assessment
► Test methods: SAMVIQ, DSCQS, DSIS, ...▪ Avoid contextual effects▪ Consider viewing distance, reference test room,
instructions for observers, training, ...▪ Suitability for multimedia▪ Absolute results▪ # of presentations of test material▪ Possibility to change vote
www.nr.no
SAMVIQ – ITU-R BT.700
► Standard for Video Quality Assessment in multimedia► SAMVIQ submitted as draft standard to ITU-R SG6 WP 6Q► Build on experiences from MUSHRA (audio quality mmt)► Uses hidden reference, low anchor, user knows total scale,
possibility to change vote, scale from 0 to 100.► High reliability, comparability w/ other test labs, absolute results► Use scenarios:
▪ Measure impact of different bit rates (MOVIS WP2)
▪ Comparison of codecs (MOVIS WP2)
▪ Minimum rate for specified quality (MOVIS WP2)
▪ Impact of network errors (MOVIS WP3)
www.nr.no
SAMVIQ: GUI used for sessions
www.nr.no
SAMVIQ: Structure of test sessions
Reference
ref
Algo. 3
D
HiddenReference
C
Algo. 1
B
Algo. 2
A
1. sequence
Reference
ref
Algo. 2
D
Algo. 1
C
Hidden Reference
B
Algo. 3
A
2. sequence
Reference
ref
Algo. 2
D
Algo. 3
C
Algo. 1
B
Hidden Reference
A
k. sequence
Example: Algo. 1: WM9, CIF,168kbpsAlgo. 2: WM9, CIF, 1032kbps
www.nr.no
Sequences used in assessment
Scenes:Skiing – bright Rugby – less detailsRainman – high frequenciesBarcelona - colourful
www.nr.no
Results: influence of content on quality
0
2 0
4 0
6 0
8 0
1 0 0
0 4 0 0 8 0 0 1 2 0 0
G lo b a l ( M e a n + C I ) S k i i n g ( M e a n + C I )R u g b y ( M e a n + C I )R a in m a n ( M e a n + C I )
B a r c e l o n a
(
M e a n
+
C I )
Windows Media 9
CIF formatall sequences
Total bit rate (kbps)
ExcellentG
oodFair
PoorBad
Mean Score
www.nr.no
Information access – server
► Server logs can give information▪ Which type of information do we need?▪ Can servers give this type of information?▪ RTP/RTCP exchange QoS data.▪ Proprietary protocols?▪ Passive measurements
► External boxes▪ Active measurements▪ Inject traffic / simulate server▪ needs response from client side.
www.nr.no
Information access – networking
► Juniper and Cisco major players
► Access of QoS values in routers▪ Which type of information do we need?▪ Can routers give this type of information?▪ Passive measurements
► External boxes▪ Active measurements▪ Inject traffic – needs response from client side.
www.nr.no
Information access – end user
► Router equipment of various brands, models, ...▪ Beyond the control of Content Provider /
► Media players logs can provide information▪ Send excerpts from logs to providers▪ Information through RTCP or proprietary protocols▪ Firewall issues !▪ Passive measurements
► Active measurements▪ Extra HW: Clean addon for user, but expensive.▪ Extra SW: maintenance and acceptance issues
www.nr.no
State of the Implementation
WebCollectorControl
PChealth
Central MetricRepository
setup