Quantifying Skype User Satisfaction
description
Transcript of Quantifying Skype User Satisfaction
Quantifying Skype User Satisfaction
Carol K. L. Wong
19 March, 2007
CSC7221
Skype a P2P Internet teleph
ony network > 2 million Skype dow
nloads ~ 85 millions users wo
rldwide
From Wikipedia.org
Skype’s Performance
Q: Is Skype providing a good enough voice phone service to the users?
Comparison of Proposed & Existing Methods
Speech quality measures
Proposed User Satisfaction Index
To quantify Speech quality User satisfaction
Built upon Subjective mean opinion score (MOS)
Call duration
Predictors Distortion of signals QoS factors: • the bit rate, • network latency, • network delay var
iations, • packet loss
Methodology
Collect Skype VoIP sessions and their network parameters
Analysis of Call Duration and propose an objective index, the User Satisfaction Index (USI), to quantify the level of user satisfaction
Validate USI by an independent set of metrics that quantify the interactivity and smoothness of a conversation.
Trace Collection
Collect Skype VoIP sessions and their network parameters. Present the network setup and filtering method
used in the traffic capture stage. Introduce the algorithm for extracting VoIP sess
ions from packet traces Strategy to sample path characteristics. Summarize the collected VoIP sessions.
Network Setup
Capturing Skype Traffic
Use 2-phase filtering to identify Skype VoIP sessions:
filter and store possible Skype traffic on the disk.
apply an off-line identification algorithm on the capture packet traces to extract actual Skype sessions.
Detect Possible Skype Traffic
Known properties of Skype clients:dynamic port number chosen randomly wh
en the application is installed and can be configured by users – “Skype port”
In the login process, submits HTTP requests to a well-known server, ui.skype.com
Heuristic to Detect Skype Hosts and their Skype Ports
treat sender for each HTTP request sent to ui.skype.com as a Skype host
choose the port number used most frequently for outgoing UDP packets sent from that host within the next 10s as the Skype port.
classify all peers that have bi-directional communication with the Skype port as Skype hosts.
maintained a table of identified Skype hosts and their respective Skype ports, and
recorded all traffic sent from or to these (host, port) pairs.
Identification of VoIP Sessions
regard An active flow as a valid VoIP session if The flow’s duration > 10s. The average packet rate is within a reasonable
range, (10, 100) pkt/s. The average packet size is within (30,300) byte
s. The EWMA of the packet size process must be
within (35, 500) bytes all the time.
Relayed Session
Merge a pair of flow into a relayed session if The flows’ start and finish time are close to
each other with errors < 30s; The ratio of their average packet rates < 1.5;
and Their packet arrival processes are positively
correlated with a coefficient > 0.5.
Path Characteristics Measurement
RTT and their jitters send out ICMP and traceroute-like probe packet
s to measure paths’ RTT while capturing Skype traffic
used
Collected VoIP Sessions
Category Calls Hosts Cens. TCP Duration Bit Rate (mean/std)
Avg. RTT (mean/std)
Direct 253 240 1 7.10% (6.43, 10.42) min
32.21 Kbps / 15.67 Kpbs
157.3 ms/ 269.0 ms
Relayed 209 369 5 9.10% (3.12,5.58) min
29.22 Kbps / 10.28 Kpbs
376.7 ms/ 292.1 ms
Total 462 570 6 8.00% (5.17,7.70) min
30.86 Kbps / 13.57 Kpbs
256.5 ms/ 300.0 ms
Methodology
Collect Skype VoIP sessions and their network parameters
Analysis of Call Duration and propose an objective index, USI, to quantify the level of user satisfaction
Validate USI by an independent set of metrics that quantify the interactivity and smoothness of a conversation.
Analysis of Call Duration
Develop a model to describe the relationship between call duration and QoS factors.
propose an objective index, the User Satisfaction Index (USI) to quantify the level of user satisfaction.
validate USI by voice interactivity measures.
Survival Analysis
With proper transformation, the relationships of session time and predictors can be described well by the Cox Proportional Hazards model (Cox Model) in survival analysis.
Survival Curves for Sessions with Different Bit Rate Levels
Survival Curves for Sessions with Different Bit Rate Levels
Group Median (min)
1 2
3 20
Bit rates Last > 40 min
< 25 Kbps 3%
> 35 Kbps 30%
The log-rank test strongly suggests that call duration varies with different levels of bit rates.
Relation of the bit rate with call duration
The trend of median duration shows a strong, consistent, positive, correlation with the bit rate.
Effect of Network Conditions
Network conditions are also considered to be one of the primary factors that affect voice quality.
the fluctuations in the data rate observed at the receiver should reflect network delay variations to some extent.
used – jitter to denote the standard deviation of the bit rate, and packet rate jitter, or pr.jitter, to denote the standard devi
ation of the packet rate.
Effect of Round-Trip Times
divided sessions into 3 equal-sized groups based on their RTTs, and compare their lifetime patterns with the estimated survival functions.
the 3 group differ significantly
Group Median duration of sessions (min)
RTTs > 270 ms 4
RTTs: 80 - 270 ms 5.2
RTTs < 80 ms 11
Effect of Jitter Jitter has a much higher correlation with call
duration than RTT.
Group Median session time (min)
Jitter > 2 Kbps 3
1 < Jitter < 2 Kbps 11
Jitter < 1 Kbps 21
These groups differs statistically
QoS related to Call Duration
most of the QoS factors they defined, including: the source rate, RTT, and jitter
are related to call duration.
Collinearity
Given that the bit rate & jitter are significantly correlated, true source of user dissatisfaction is unclear.
Use the Cox model and treat QoS factors, e.g. the bit rate, as risk factors or covariates; i.e. as variables that can cause failures. The hazard function of each session is decided
completely by a baseline hazard function and the risk factors related to that session.
Collinearity 7 factors - bit rate (br),packet rate (pr),jitter, pr.jitt
er, packet size (pktsize), and round trip time (rtt)
br pr jitter pr.jitter pktsize rtt
br * +++ ++ +++ -
pr +++ * - --- + --
jitter ++ - * +++ +++
pr.jitter --- +++ *
pktsize +++ + +++ *
rtt - -- *+/- : positive or negative correlation collinearity is computed by Kendall’s statistic (Pearson’s product moment statistic yields similar results)
Collinearity
the bit rate, packet rate, and packet size are strongly interrelated;
jitter and packet rate jitter are strongly interrelated.
the bit rate, jitter, and RTT are retained in the model
Cox Model
define the risk factors of a session as a risk vector Z
h(t|Z) = h0(t) exp(tZ) = h0(t)exp(pk=1kZk)
h(t|Z) - the hazard rate at time t for a session with risk vector Z;
h0(t) - the baseline hazard function computed during the regression process;
= (1,…, p)t - the coefficient vector that corresponds to the impact of risk factors.
Zp is the pth factor of the session
The Cox model 2 sessions with risk vectors Z and Z’, the
hazard ratio:
h(t|Z)/ h(t|Z’) = exp(pk=1kZk – kZ’k)
is time-independent constantHence, the validity of the model relies on t
he assumption – the hazard rates for any 2 sessions must be in proportion all the time.
Sampling of QoS Factors
In the regression modeling, we use a scalar value for each risk factor to capture user perceived quality.
Divide the original series s into sub-series of length w, from which network conditions are sampled.
Choose one of the min, average and max measures taken from sampled QoS factors having length [|s|/w] depending on their ability to describe the user perceived experience during a call.
Evaluation
evaluate all kinds of measures and window sizes by: fitting the extracted QoS factors into the Cox m
odel and comparing the model’s log-likelihood, i.e. an indi
cator of goodness-of-fit.
Finally, the max bit rate and min jitter are chosen, both sampled with a window of 30s.
Model Fitting
the Cox model assumes a linear relationship between the covariates and the hazard function
the impact of the covariates on the hazard functions with the following equation:
This corresponds to a Poisson regression model if h0(s) is known.
0
0 )()())(exp(][ dsshstIZfsE it
i
ti - the censoring status of session i, f(Z) – the estimated functional form of the covariate Z.
The influence of the bit rate is not proportional to its magnitude - scale transformation.
The RTT factor has an approximate linear impact.
Jitter factor
Verification Employ a more generalized Cox model that allo
ws time-dependent coefficients to check the proportional hazard assumption by hypothesis tests. After adjustment, none of covariates reject the linearity hypothesis at = 0.1, the transformed variables have an approximate linear impact on the hazard functions.
Use the Cox and Snell residuals ri (for session i) to assess the overall goodness-of-fit of the model. Except for a few sessions that have unusual call duration, most sessions fit the model very well.
Model Interpretation
Variable Coef, eCoef Std. Err. z P > |z|
br.log -2.15 0.12 0.13 -16.31 0.00e+00
jitter.log 1.55 4.7 0.09 16.43 0.00e+00
rtt 0.36 1.4 0.18 2.02 4.29e-02
define the factors’ relative weights as their contribution to the risk score, i.e., tZ.
- coeff
Relative Influence of Difference QoS for each session
The degrees of user dissatisfaction caused by the bit rate, jitter and round-trip time are 46%:53%:1%.
Conclusion
Possible to improve user satisfaction by fine tuning the bit rate used.
As the use of relaying does not seriously degrade user experience, higher round-trip times do not impact on users very much.
Jitters have much more impact on user perception.
The choice of relay node should focus more on network conditions, i.e., the level of congestion, rather than rely on network latency.
User Satisfaction Index (USI)
As the risk score tZ represents the levels of instantaneous hang up probability, it can be seen as a measure of user intolerance. Accordingly, define the USI of a session as its minus risk score:
USI = - tZ = 2.15xlog(bit rate) – 1.55xlog(jitter) – 0.36xRTT
where the bit rate, jitter, and RTTs are sampled using a 2-level sampling approach
The prediction is based on the median USI for each group. y-axis is logarithmic to make the short duration groups clearer.
Advantages of USI over Other Objective Sound Quality Measures
USI’s parameters are readily accessible: the 1st and 2nd moment of the packet counting process
can be obtained by simply counting the number and bytes of arrival packets
the round-trip times.Usually available in peer-to-peer applications for overlay network construction and path selection.
developed the USI based on passive measurement rather than subjective surveys, it can also capture sub-conscious reactions of participants, which may not be accessible through surveys.
Methodology
Collect Skype VoIP sessions and their network parameters
Analysis of Call Duration and propose an objective index, the User Satisfaction Index (USI), to quantify the level of user satisfaction
Validate USI by an independent set of metrics that quantify the interactivity and smoothness of a conversation.
Validation
Results of the validation tests using a set of independent measures derived from user interactivities show a strong correlation between the call durations and user interactivities. This suggests that the USI based on call duration is significantly representative of Skype user satisfaction.