Ulrich Speidel, 'Etuate Cocker, Firas Ghazzi, Nevil Brownlee Department of Computer Science, The...

COULD THE INTERNET HANG UP ON US? A LONGITUDINAL GLOBAL STUDY OF INTERNET PACKET AND DATA ARRIVAL PREDICTABILITYUlrich Speidel, 'Etuate Cocker, Firas Ghazzi, Nevil

BrownleeDepartment of Computer Science, The University of Auckland

Ulrich Speidel – [email protected]

AGENDA

VoIP and other real-time protocols Causes and implications of variability in packet

arrival Observing long-term trends in packet arrival

variability with a beacon network Why the Pacific Islands are a great test bed –

and why other parts of the world are important, too

Preliminary observations Measures of arrival quality: jitter and entropy


CONTRIBUTORS TO DATE Niko Rebenich, Stephen Neville, Aaron

Gulliver, University of Victoria, B.C., Canada Kashif Nisar, Zeeshan Aziz, Suhaidi Hassan,

Universiti Utara Malaysia Ming-Chui Dong, Vincent Wong, University of

Macau Raimund Eimann, Zurich, Switzerland (in a

private capacity) Bernhard Platter, Bernhard Ager, ETHZ

Zürich Hendrik Ferreira et al., University of

Johannesburg, South Africa AJ Han Vinck, University of Duisburg-Essen,

Germany Hirosuke Yamamoto, University of Tokyo Hidetoshi Yokoo, Gunma University Hiroyoshi Morita, University of Electro-

Communication ITS, University of Auckland Muriel Médard, Jason Cloud, Massachusetts

Institute of Technology Randy Bush (in a private capacity)

Ministry of Lands, Survey, and Natural Resources, Tonga

Connect ISP, Fiji Ministry of Infrastructure and Planning, Cook

Islands Oyster ISP (Telecom Cook Islands), Rarotonga,

Cook Islands Ministry of Fisheries, Government of Kiribati IT Department, Government of Tuvalu E.M. Jones Ltd., Tonga Ministry of Revenue, Tonga Solomon Telekom, Solomon Islands


INDUSTRY CONTRIBUTORS TO DATE

Ministry of Lands, Survey, and Natural Resources, Tonga

Connect ISP, Fiji Ministry of Infrastructure and Planning, Cook Islands Oyster ISP (Telecom Cook Islands), Rarotonga, Cook

Islands Ministry of Fisheries, Government of Kiribati IT Department, Government of Tuvalu E.M. Jones Ltd., Tonga Ministry of Revenue, Tonga Solomon Telekom, Solomon Islands


WHY ARE REAL-TIME PROTOCOLS IMPORTANT?

Voice over IP (VoIP) telephony / videoconferencing / remote call centre industry

Telemedicine (e.g. remote surgery / diagnostics / consultation)

Remote control in industrial applications

Internet of Things

Financial applications (rapid trading)


VOICE OVER IP AND OTHER REAL-TIME PROTOCOLS

blah

A/D Encoder

Internet

Decoder

Buffer

D/A

Packets (~50 per second)

blah

• analog voice signal is digitally encoded

• encoded byte stream is chopped into a packet train

• packets are transmitted via Internet to receiver

• receiver buffers packets to establishconstant-rate dataflow to decoder

• decoder recovers analog signal and replays it

Transmitter

Receiver


VOICE OVER IP AND OTHER REAL-TIME PROTOCOLS

Delay between signal generation and replay is undesirable – must be kept as small as possible

Replay delay is composed of three components:1. Physical latency of cables or satellite links along

packet paths (fixed for a given path)2. Time spent in router queues (variable)3. Size of buffer at receiver (governed by variation in

arrival times)


IRREGULAR PACKET ARRIVAL - CONSEQUENCES

Need to buffer more Real-time services such as VoIP, video

conferencing, and remote manipulation may become impractical

Downloads and streaming video/audio would still work

Significant economic impact Effects are most likely to be felt between

remote endpoints first


CAUSES OF VARIABILITY IN PACKET ARRIVAL

Length and number of router queues Grow with increase in traffic and network

mesh

Overflowing router queues cause packet drops

Load balancing at routers Could this cause problems?


LOAD BALANCING ROUTERS

Load balancing routers send packets to the same destination across different links

If packets from the same stream are load balanced, it causes them to take different paths and experience different latencies

In some cases, packets may overtake each other (out-of-order arrivals)


PACIFIC RIM SUBMARINE CABLES

Which way do packetstravel between Asia and NZ?

What are the possiblelatencies?

http://www.submarinecablemap.com/


GO FIGURE!>tracert 161.64.45.10

Tracing route to 161.64.45.10 over a maximum of 30 hops

1 1 ms 3 ms 3 ms tmk-net-207-gw.net.auckland.ac.nz [130.216.207.254] 2 2 ms 1 ms 1 ms cx-alpha-sxj-700.net.auckland.ac.nz [172.18.0.6] 3 3 ms <1 ms <1 ms dxj-260-1-to-cx-alpha.net.auckland.ac.nz [172.18.0.26] 4 2 ms 2 ms 2 ms br1-to-cx-alpha.net.auckland.ac.nz [130.216.246.41] 5 1 ms 1 ms 1 ms 203.167.201.41 6 4 ms 5 ms 5 ms unknown.telstraglobal.net [134.159.174.41] 7 29 ms 27 ms 31 ms i-0-0-1-1.sydo-core02.bx.telstraglobal.net [202.84.143.134] 8 29 ms 27 ms 27 ms i-0-3-2-0.sydo-core01.bi.telstraglobal.net [202.84.220.190] 9 212 ms 215 ms 215 ms i-0-5-2-0.hkth-core01.bx.telstraglobal.net [202.84.143.22] 10 141 ms 142 ms 141 ms i-3-0-0.hkth01.bi.telstraglobal.net [202.84.153.233] 11 155 ms 155 ms 155 ms tata-peer.hkth01.pr.telstraglobal.net [134.159.128.6] 12 144 ms 144 ms 144 ms 116.0.67.70 13 * 150 ms * z0l161.static.ctm.net [202.175.0.161] 14 160 ms 161 ms 160 ms cr1-191.macau.ctm.net [202.175.26.241] 15 161 ms 160 ms 160 ms ctm-int-umac.macau.ctm.net [202.175.0.58] 16 * * * Request timed out. …

Trace complete.

Traceroute to our beacon at Universiti Utara Malaysia:

>tracert 103.5.183.14

Tracing route to 103.5.183.14 over a maximum of 30 hops

1 4 ms 1 ms 1 ms tmk-net-207-gw.net.auckland.ac.nz [130.216.207.254] 2 2 ms 1 ms 2 ms cx-alpha-sxj-700.net.auckland.ac.nz [172.18.0.6] 3 <1 ms <1 ms <1 ms dxj-260-1-to-cx-alpha.net.auckland.ac.nz [172.18.0.26] 4 1 ms 1 ms <1 ms br1-to-cx-alpha.net.auckland.ac.nz [130.216.246.41] 5 1 ms 2 ms 1 ms 203.167.201.41 6 154 ms 166 ms 155 ms te7-3.ccr01.sjc05.atlas.cogentco.com [38.122.92.105] 7 155 ms 155 ms 155 ms te0-2-0-4.ccr21.sjc01.atlas.cogentco.com [154.54.84.53] 8 133 ms 137 ms 135 ms be2162.mpd21.lax01.atlas.cogentco.com [154.54.27.173] 9 133 ms 133 ms 135 ms be2021.ccr21.lax04.atlas.cogentco.com [154.54.87.250] 10 133 ms 132 ms 144 ms 38.104.210.82 11 239 ms 239 ms 240 ms ae-2.cr-gw-2-sin-pip.sg.globaltransit.net [124.158.224.17] 12 238 ms 238 ms 238 ms 124.158.224.91 13 239 ms 240 ms 248 ms 124.158.226.150 14 248 ms 247 ms 248 ms ae-1.cr-02-glsfb.ni.time.net.my [223.28.2.121] 15 244 ms 245 ms 245 ms xe-0-0-0-0.er-01-glsfb.ni.time.net.my [223.28.2.2] 16 245 ms 244 ms 247 ms xe-0-1.es-01-glsfb.ni.time.net.my [223.28.16.130] 17 345 ms 345 ms 345 ms 203.121.112.18 18 345 ms 344 ms 345 ms 211.25.227.212 …

Trace complete.

Traceroute to our beacon at the University of Macau:

via Sydney and Hong Kong

via San Jose (California) and Los Angeles

…and back across the Pacific!


WHICH EFFECT WILL WIN?

Technological progress or the forces of chaos it unleashes?

extra bandwidth new links

more powerfulrouters

more efficient codecs

more users

more traffic

more congestion

more malware

less predictable packetarrivals

more inefficient routing


CAN WE PREDICT A TREND?

Basic idea: Pick a sample application, e.g., VoIP Reality check: Applications don't stay stable, so need one

whose behaviour we can control

Repeatedly use the application over time between remote end points where deterioration is likely to be seen early

See whether you observe any long-term changes

Do this for multiple pairs of endpoints so you can ascertain whether a change is specific to a particular endpoint or pair of endpoints


HOW? THE BEACON NETWORK

Basic idea:

"Run a global network of beacon computers that act pair-wise as endpoints"

To date (December 2013), 28 beacons have been installed in Canada, Cook Islands, Fiji, Germany, Japan, Kiribati, Macau, Malaysia, New Zealand, Solomon Islands, South Africa, Switzerland, Tonga, Tuvalu, and the United States.

…and the network is growing! Additional beacons in Hawaii and Canada are under construction


THE BEACON NETWORK


BEACON SOFTWARE

The beacon software is a simple C program that runs one experiment, writes it logs and then shuts down

Scheduled via cron jobs & configured via command line parameters

Writes plain text logs

Requires sufficient privileges to run raw sockets (to determine TTL of received packets)


BEACON HARDWARE

PC or Mac platform with Ubuntu/BSD/OSX

Doesn't need to be overly fast – we use 600 MHz ALIX boards among others

Must have real-time clock, but GPS / atomic clock synchronisation is not required

Virtual machines are not a good idea – tend to spend too much time away from task at hand


BEACON UDP EXPERIMENTS

Beacon exchanges 10,000 UDP packets of 110 bytes with a partner beacon

Packets transmitted every 20 ms

Packets are timestamped and serial-numbered

At receiving end, packets are logged with arrival time stamp, serial number, arrival sequence number, and TTL observed

This experiment typically runs 3 times a day between selected beacon pairs


BEACON TCP EXPERIMENTS

Two beacons exchange a predefined set of data via TCP

Two modes available: VoIP (constant payload rate) and Download (maximum payload rate)

Receiving end logs data arrival at application layer plus TTL and size of packets involved in communication (from raw socket)


WHY HAVE BEACONS ALL OVER THE WORLD?

Want long-term global trend, not just local effects

Want a "developed world" baseline but also see what it is like in remote places on the fringe – many of our beacons run in the Pacific for that reason (need I mention Africa?)

Long paths generally are of interest – both in terms of latency and number of hops

A lot of international traffic passes through "hub regions" (North America, Europe, SE Asia). What effect do these regions have on traffic that passes through them?

Last but not least: We're looking for input (and your own experiments)!


BEACON CHALLENGES

Proper timing without proper timebase

Precise timing on multitasking computers

Getting two beacons to communicate (firewalls, ports, policies/politics)

Scheduling

Data volume backhaul (each beacon contributes ~3MB / experiment / day)

Data processing & presentation (currently UDP only)

Coordination with many partners, each with a unique network setup

Making it longitudinal


INITIAL OBSERVATIONS: PACKET LOSS


INITIAL OBSERVATIONS: LOSS VS OUT-OF-ORDER


PACKET TRAIN QUALITY

Subjective approaches, e.g., Mean Opinion Score (MOS) – reliably replicable only with very large sample

Objective approaches, e.g., jitter measurements.

But: jitter can be random (=problem) or predictable (=less of a problem)

How can we tell the difference?


NON-RANDOM JITTER

Consider the following simplified scenario:

Assume uncongested routers but differences in latency, blue router load balances

Have two paths/latencies -> high jitter, but not really queue-induced

Jitter ignores systematic patterns in arrivals

A good entropy estimator (i.e., not Shannon) can detect the difference


COMPLEMENT: ENTROPY (RATE)

Entropy: "measure of disorder in a system". Usually given in the form of an entropy rate rather than absolute entropy

Entropy rate: number of bits of actual information required to represent an object divided by number of bits actually used to represent it.

Dimensionless ("bits/bit")

Examples: Shannon entropy (well known). Better: Lempel-Ziv compression ratios, T-entropy, …

Predictable arrivals or arrival patterns: low entropy

"Really unpredictable arrivals" = high entropy


ENTROPY HOW-TO

Map inter-arrival times of successive UDP packets to symbol bins, e.g.: t < 17 ms: "A" 17 ms < t < 19 ms: "B" 19 ms < t < 21 ms: "C"

… Form string from these symbols: "CCCBDCACF…"

Determine entropy rate for string (e.g., as Lempel-Ziv compression ratio or T-entropy)

"Perfect" string will be "CCCCCCC…" – highly compressible, low entropy

Chaotic arrivals generate many new pattern combinations: harder to compress, higher entropy


ENTROPY VS. JITTER


ENTROPY VS. JITTER

0.001 0.01 0.1 1 10 1000

0.2

0.4

0.6

0.8

1

1.2

1.4

Jitter [s]

T-e

ntr

opy 5

-bin

[bit

s/s

ym

bol]

"Direction of worry"


WHERE IS ENTROPY INTRODUCED?


PICKING A TREND?


CHANGING TTLS – PROOF OF PATH INSTABILITY


WHAT ABOUT TCP?

Not ideal for real-time applications due to handshake delays

Still very widely used for real-time apps as UDP is often blocked at firewalls (because it's historically been used infrequently)

E.g., Skype fails over to TCP if UDP can't get through


TCP IN STREAMING APPLICATION

Time

Data[bytes]

Cumulative amount of bytes received

Bytes needed for continuous rate immediate replay

Bytes needed for continuous rate buffered replay (no underruns)

Actual replay with "boredom wheel"

minimum buffer period

minimum buffersize

Replay buffer underruns

Goal: Avoid buffer underruns with minimum buffer period / size


TCP MINIMUM BUFFER TIME TRENDS?


CONCLUSIONS

Real-time traffic and best-effort protocols are uneasy companions

We propose to use a large beacon network to measure arrival shape of packet trains

Our beacons already see interesting effects

Effects are strongly path-specific and sometimes not easily explained

Jitter and entropy are both useful for arrival shape evaluation

A lot of work remains to be done!

Ask us if you're interested in hosting a beacon

https://iibex.auckland.ac.nz (https://130.216.197.6)

https://iibex.auckland.ac.nz/

Questions ?

?

? ?

??

?

?

?

https://iibex.auckland.ac.nz

Ulrich Speidel, 'Etuate Cocker, Firas Ghazzi, Nevil Brownlee Department of Computer Science, The...

Documents

Transcript of Ulrich Speidel, 'Etuate Cocker, Firas Ghazzi, Nevil Brownlee Department of Computer Science, The...