Automatic Test Packet Generation

Hongyi Zeng , Peyman Kazemian , Nick McKeownUniversity, Stanford, CA, USA

George Varghese UCSD, La Jolla

Microsoft Research, Mountain View, CA, USA

:// . / / /https github com eastzone atpg wiki

2

Debugging the netTroubleshooting the network is difficult.

3

Troubleshooting the network is difficult.

Survey of NANOG mailing list Data set: 61 responders:

23 medium size networks (<10K hosts), 12 large networks (< 100K hosts).

Frequency: 35% generate >100 tickets per month.

Downtime: 56% take over an half hour to resolve.

Current tools: Ping (86%), Traceroute (80%).

70% asked for better tools, automatic tests.

4

Automatic Test Packet GenerationFramework that test the liveness , congestion and performance.

Goal: automatically generate test packets to test the network, and pinpoint faults.

5

T

Using the header space analyses we can collect the entire packet range from the net.

ATPG use the header space framework.

Header Space Analyses

FIB’s , topologyEquivalent classes & packets

(𝑝𝑜𝑟𝑡¿¿1 ,𝑝𝑜𝑟𝑡2)¿

6

ATPG use the header space framework.Reminder :

Represent by port and header.

Define how packet field are modified.generate a list of output packets.

Transfer function T- models a network.

7

ordered list of rules the packet match so far as it travers the net.

Transfer function models the network by specifying which pairs of ports are connected.

applying the T and repeatedly till the packet reach her destination.

ATPG use the header space framework.

8

Test packet generation

Assume there is a set of terminals in the network can send and receive test packets.

ATPG use only available test terminals. ATPG use headers that test terminal is permitted to send.

Goal : generate a set of test packets to exercise every rule.

All pairs reachability

parser

Test packet generator

Test terminals

Tfunction

Hea

der s

pace

ana

lysi

s

Topology , FIB , ALC …

Test Pkt

DB

9

Test packet generation

10

Test packet Selection

Determine the set of packet headers that can be sent from each terminal to another.

Generate

all

pairs

reachability

tabl

e,

equival

ent

cl

asses.

Sa

mpl

e

and

co

mpressi

on.

Sent

peri

odicall

y

the

test .If

a

error is

detected

,

use

l

ocalizati

on

al

gorith

m.

T - number test terminals D - network diameter

R - average number of Rules in boxComplexitiy

11

Find all packet space

TT

T

12

: dst_ip = 10.0/16 Port 0 : dst_ip = 10.1/16 Port 1 : dst_ip = 10.2/16 Port 2

: dst_ip = 10.2/16 Port 2 : dst_ip = 10.1/16 Port 1

: dst_ip = 10.0/16 : tcp = 80 Port 0

: in_Port = 1 Port : 0,2 : in_Port = 0,2 Port : 1

A

C

B02

1

2 01

1

02

Header Ingress Port Egress Port Rule History

dst_ip = 10.0/16, tcp = 80

dst_ip = 10.2/16

𝑃 𝐴

𝑃 𝐵

𝑃1

𝑟 𝐴1,𝑟 𝐵3 ,𝑟 𝐵4

𝑟𝑐 1 ,𝑟 A 3link  AB

link  CB, link  BA

𝑃𝐶

𝑃2

13

A

C

B02

1

2 01

1

02

: dst_ip = 10.0/16 Port 0 : dst_ip = 10.1/16 Port 1 : dst_ip = 10.2/16 Port 2

: dst_ip = 10.2/16 Port 2 : dst_ip = 10.1/16 Port 1

: dst_ip = 10.0/16 : tcp = 80 Port 0

: in_Port = 1 Port : 0,2 : in_Port = 0,2 Port : 1

Header Ingress Port Egress Port Rule History

dst_ip = 10.0/16, tcp = 80 , link AB

dst_ip = 10.2/16 link CB , link BA

dst_ip = 10.2/16 , link AB

dst_ip = 10.1/16 , link BC

dst_ip = 10.2/16 link BC

dst_ip = 10.2/16, tcp = 80 , link BC

dst_ip = 10.1/16 link AC

14

Sampling

Pick at least one packet from every equivalent class.

You can peak randomly or peak all the headers from a certain class to check a fault for a specific header.

Header Ingress Port Egress Port Rule History

dst_ip = 10.0/16, tcp = 80 , link ABdst_ip = 10.2/16 link CB , link BA

dst_ip = 10.2/16 , link ABdst_ip = 10.1/16 , link BCdst_ip = 10.2/16 link BC

dst_ip = 10.2/16, tcp = 80 , link BCdst_ip = 10.1/16 link AC

15

Compression

Using the Min-Set Cover algorithm. The remaining test packet will be use later for fault

localization.

Header Ingress Port Egress Port Rule History

dst_ip = 10.0/16, tcp = 80 , link ABdst_ip = 10.2/16 link CB , link BA

dst_ip = 10.2/16 , link ABdst_ip = 10.1/16 , link BCdst_ip = 10.2/16 link BC

dst_ip = 10.2/16, tcp = 80 , link BCdst_ip = 10.1/16 link AC

16

Fault Localization

Success and failure depend on the nature of the rule forwarding / drop.

Divide the fault in two categories: Action fault - effect all the packets matching the rule.

Match fault - effect some of the packets matching the rule.

17

Find the set of suspect rules P is the set of all the rules traversed by success test packets. F is the set of all the rules traversed by failing test packets. F - P is the set of suspect rules.

Minimize the suspect rules set We will make use of our reserved packets.

Find the

suspect rule

s

Minimize the

suspect

rules set

Report the set

Fault Localization

FPHeader Ingress Port Egress Port Rule History Result

dst_ip = 10.0/16, tcp = 80 link AB passdst_ip = 10.2/16 link CB fail

dst_ip = 10.2/16 , link AB fail

18

False positive

There is a possibly for false positive , Rule that are left as suspect although it behave correctly.

Can be solve by adding more test terminals to the network.

19

All pairs reachability

parser

Test packet generator

Fault Localization

Test terminals

Tfunction

Hea

der s

pace

ana

lysi

s

Topology , FIB , ALC …

Test Pkt

DB

20

Use case for ATPG

Test functional correctness of a network.

Forward rule and Drop rule : if packet exercises the rule.

Link rule : if packet passing the link without header modification.

Test performance of the network. Congestion and available bandwidth:

Measure the latency , bandwidth.

21

Header B AR2

A Broadcast

R1

R3

R4

A drop

B Fwd

C Fwd

Header B

Header CAHeader C

Drop rule

A

C

B

22

Implementation ATPG written in python.

The data plane information Collected and parsed (Cisco IOS) .

The generator use Hassel header space analysis library to construct T functions.

All pairs reachability is computed using multiprocess parallel processing

After Min set cover executes the output stored in SQLite database.

The process uses JSON to communicate with the test agents.

23

In SDN, the controller could directly instruct the switch’s to send test packets and to forward test packet back to the controller.

Alternate implementation & SDN

SDN controller

24

Testing in Emulated Network

Replicated the Stanford backbone network in Mininet.

Using open vSwitch , real port configuration and real topology.

Forwarding Error Congestion

Available Bandwidth Priority

25

Testing in Emulated Network

26

Forwarding Error

Change forwarding rule to drop rule in boza.

bbra

s1 s2

goza coza bozapoza

s4 s5

27

bbra

s1 s2

goza coza bozapoza

s4

Forwarding Error𝑅1❑

𝑏𝑜𝑧𝑎 :dst ip=172 .20 .10 .32 /27→𝑑𝑟𝑜𝑝

s5

28

Congestion

bbra

s1

rozayozapoza

s2

pozb

29

bbra

s1

rozayozapoza

s2

pozb

Ingress Port Egress Port Rule History

poza yoza Poza-s1,s1-bbra,bbra-yoza

roza yoza Roza-s2,s2-bbra,bbra-yoza

pzob yoza Pozb-s1,s1-bbra,bbra-yoza

pzob roza Pozb-s1,s1-bbra,bbra-roza

30

Production Network Deployment

Deployed an experimental ATPG system in Stanford University.

For test terminal they use 53 Wi-Fi access.

Test agent download the test packets from a central web server every ten minutes , and conducts a ping test every 10 seconds.

31

Production Network Deployment

During the experiment , a major outage occurred ,ATPG has captured the problem.

32

EvaluationEvaluated the system on Stanford backbone.

testing all rules 10 times per second needs < 1% of link overhead.0 20 40 60 80 100 120

0

100000

200000

300000

400000

500000

600000

700000

PacketsLinear (Packets)

0 20 40 60 80 100 1200

500

1000

1500

2000

2500

3000

3500

4000

4500

Test PacketsLinear (Test Packets)comp timeLinear (comp time)

33

Limitation

•internal state can be changed.•random to route packets.

Dynamic , Non deterministic

boxes

•ATPG uses sampling to collect the forwarding state.

Sampling time

•backup rules changes may be occurred without been notice.

Invisible rule

34

Conclusion

• ATPG tests liveness and performance.• < 1% of link overhead.• Works in real networks.

Questions?