The Art OfSystem and Solution Testing

Example• A software developer/tester convention was being held. On the train to the convention, there were a

bunch of developer majors and a bunch of tester majors. Each of the developer majors had his/her train ticket. The group of testers had only ONE ticket for all of them. The developer majors started laughing and snickering. Then, one of the software testers said, "here comes the conductor" and then all of the testers went into the bathroom. The developer majors were puzzled. The conductor came aboard and said "tickets please" and got tickets from all the developer majors. He then went to the bathroom and knocked on the door and said "ticket please" and the testers stuck the ticket under the door. The conductor took it and then the testers came out of the bathroom a few minutes later. The developer majors felt really stupid. So, on the way back from the convention, the group of developer majors had one ticket for the group. They started snickering at the testers, for the whole group had no tickets amongst them. Then, the tester lookout said "Conductor coming!" All the testers went to one bathroom. All the developer majors went to another bathroom. Then, before the conductor came on board, one of the testers left the bathroom, knocked on the other bathroom, and said "ticket please." Lesson learned: Any test that passed in unit testing can fail in system testing.

Goal• Find bugs that can not be fund by feature

level testing but will most likely be found on customer’s Usage.

• If no time, can even only do system level and solution testing.

• Development is a science, testing is an art.

What Is a System Level Testing• Company N has a testbed called “Mother of

All Testbed (MOAT). It is the superset of almost all Company N’s product features.

• To fully execute a cycle, it takes 2 weeks.

• It has just one test case. Compare to feature level testing where you have thousands of test cases

What Is a Solution Level Testing• Company C has a dedicated solution testing

team

• Each solution is an end-to-end application and has a code name– VoIP end-to-end solution

– Security end-to-end solution

– MPLS solution

– Cable/DSL solution.

Difference?• Solution is customer oriented. Not just one

company’s product. • System testing is product oriented. • Feature testing is code oriented. (need a

functionality Specification as input)• System/Solution: 30% strategy, 70%

experience• Feature:60% strategy, 40% experience

What makes a System/Solution tester

• System/Solution testing is to test the “Customer”. Input

TesterOutput

User Manual

Customer Secenrios.

Experience

Test Plan/Test Cases

System Level Defects

System Tester’s 360 Degree Advantage

• They interface with Customer (understand customer)

• They interface with SE, TME, Product manager (understand marketing)

• They interface with developer (understand development)

• They interface with feature tester. (understand testing)

Trademark of System and Solution Testing

• Race condition

• Memory corruption

• Algorithm performance (routing protocol convergence time for both uni-cast and multicast)

• How system perform in low memory and high CPU (timers will be stretched out)

Typical Bugs Found on System Testing• Crash: Infrastructure design issue

• Crash: Race condition: Malloc memory twice, crash only happens when 1st one success and 2nd one fail.

• Solution : Interoperability with 3rd party products

System Testing is a Time Capsule• 99.9999% up time = 10 seconds down time

in 3 years

• What your customers will experience in 3 years?

• Emulate 3 years in 2 weeks

• That is why lots of stress testing

System Testing is a Time Capsule

正常 异常 1异常 2 异常 3 异常 4 异常 5 异常 6

1-2年

2周

Ideas on Building Your System Cases

• Defined the baseline– Background traffic.

– # of clients

– Complexity of the configuration

• Identify scalability (performance) index.

• Get the typical customer scenarios and topologies.

• Design test cases.

Build Your Background Traffic• Background traffic need to drive CPU and

memory to 30%-40%.

• Need to be diverse among typical applications

• Need to be measurable on – Latency

– Transit loss.

• IMIX traffic may be considered

• Consider RFC2544 testing

No Performance Index, No Testing• Performance index means scalability

• Size of the routing tables

• Number of VPN tunnels

• Voice call per second

• Routing protocol convergence time

• Traffic throughput.

• Peer count (P2P)

• Connection Rate (P2P)

• Concurrent connections

QC Meets Customer• The product is a solution. If the problem isn't solved,

the productdoesn't work.• The best test case can ever be designed on your

product is in the customer’s real network.

• The complexity of your test cases should be equal to your most valuable customers.

• Customers like QC visit from vendor company very much. You will be surprised to see how much information customers like to share with you.

• Get your topology from your customer.

Define Your Scenarios• This is the way customer use it.

• Different location, different area (signal coverage)

• Barrtery

• Different combination way of using our product– GPRS browsing with MP3 in the background while

have an unidentified call

• Different speed for mobile devices

20/80 Rule in the consumer market• Satisfied customers seldom speak up

• Unhappy customers will complain

• 1% chance of reproducible rate is ok in the network equipment market, but not ok in the consumer market

• Higher testing standard in the consumer market

Simulation Is Your Friends• Customer level’s scalability can be achieved

in our lab by simulator

• Traffic simulator (commercial testing equipment are expensive).

• Client simulator

• Protocol Simulator.

• Device Simulator.

Simulation Is Your Friends• A 32 timer reverse Bug, happen once in

every 2 years, how you test that?

Steps to Design a System Level Testing

• One liner for your testing goals.

• List the features you want to cover.

• Define your features performance index (metrics)

• Design your topology– Customer feedback

– Performance index requirement

• Pick your test simulator (budget conscious)

• Design your scenarios.

• Define your pass/fail criteria.

Give a Detail Example on DHCP server testing

• Definition: Carrier grade high availability DHCP server System level testing.

• Goals: testing system instability especially during the failover.

• Performance index:

– IP address release per second.

– Size of the database.

– Size of the log.

– Command and Control traffic (admin traffic)

– DNS traffic.

• Customer’s topology (simple)

• Simulator: DHCP client simulator

Give a Detail Example on DHCP server testing

• Scenarios: 1 Primary down, then come back after secondary switch over

2 Primary down, then come back right away before the switch over.

3 Primary down, then come back after switch over, the down again.

4 Primary keep flapping – up, down, up, down for a while (both before and after the switch over).

5 Secondary down, then come back.

6 Secondary keep flapping – up, down, up, down for a while (both before and after the switch over).

7 Both primary and secondary keep flapping for a while.

8 Both shut down then come back the same time.

9 Both shut down, the secondary come back first.

10 Both shut down, the primary come back first.

Get the Pass/Fail Criteria. • DHCP servers can response DHCP request. • No server performance down grade, the

DHCP can still response to a high rate of DHCP request during the failover and recovery

• The failover relationship between primary and secondary, and the failover state is working as expected.

• Database sync correctly. And the data integrity is fine.

Overlap your System Test Cases• Unlike feature testing, the impact among

features are welcomed

• Overlap your test cases one on top of another increase the feature interaction.

• Carefully record your execution steps. (reduce non-reproducible bugs)

Example: Firewall Instability Testing• Definition: Testing a firewall system’s instability

• Goals: Discover critical issues when firewall is under severe network conditions

• Performance index:

– # of current sessions

– # of VPN tunnels

– # of content filtering rules.

– # of policies

– # of the NAT

• Customer’s topology (simple)

• Simulator: DHCP client simulator

Example: Firewall Instability Testing• Customer’s topology

Example: Firewall Instability Testing• Baseline traffic (pre-conditions)

– Valid traffic• L3 traffic

• L7 traffic (HTTP/FTP, voice, rich application)

– Invalid traffic (noise)• To the box

• Through the box

• Trust and un-trust zone

• Malicious (hacker traffic) and just invalid

Example: Firewall Instability Testing• Pass and fail criteria

– No crash

– No severe performance downgrade on • Packet loss

• Long latency

– Major functionality working on each of the performance index.

Example: Firewall Instability Testing• Customer Secenrios

– HA

– VPN

– Routing

– QoS

– DPI

14 Steps Firewall TestingPhase1: start valid traffics

Phase2: layer3 and layer4 attacks

Phase3: Inject firewall attack traffic

Phase4: Inject invalid frame

Phase5: Drop In

Phase6: management attacks

Phase7: standard security testing

Phase8: Internet applications

Phase9: IP fragmentation, ICMP

Phase10: HA function

Phase11: VPN function

Phase12: dynamic routing function

Phase13: QoS

Phase14: contents filter

System and Solution Automation• System/Solution automation is to make your

life easy

• End-to-end full automation is hard, try to do semi-automation first.