Slide 1

High Volume Automated Testing for Software Components Harry RobinsonDoug Szabo Microsoft

Slide 2

High Volume Test Automation The essence of HVTA techniques is automated execution and evaluation of large numbers of tests, for the purpose of exposing functional errors that are otherwise hard to find. Experiments with High Volume Test Automation, 2004 McGee & Kaner

Slide 3

Subtle Bugs Only weak bugs have a logic to them and are amenable to exposure by strictly logical means. Subtle bugs have no definable pattern - they are wild cards. Software Testing Techniques 2 nd Edition, 1990 Beizer

Slide 4

Bounded Exhaustive Testing Bounded exhaustive testing (BET) is a verification technique in which software is automatically tested for all valid inputs up to specified size bounds. Software assurance by bounded exhaustive testing, 2005 Coppit et al

Slide 5

Typical Biases Developers happy paths Testers a nose for bugs

Slide 6

CodeProject: Sorting Algorithms in C#

Slide 7

The Sorting Algorithms

Slide 8

Popular

Slide 9

Recommended

Slide 10

and it looks cool!

Slide 11

but dark clouds appear on the horizon

Slide 12

...

Slide 13

Reasonable Unit Tests

Slide 14

Even Some Randomized Unit Tests

Slide 15

Pre-Oracled Data 012345 250314 012345 shuffle sort compare

Slide 16

The Loop

Slide 17

Shearsort demo #1

Slide 18

Shearsort demo #2

Slide 19

Shearsort bug

Slide 20

Shearsort bug - fixed

Slide 21

Heapsort bug

Slide 22

Heapsort bug - fixed

Slide 23

OddEvenTransport bug

Slide 24

OddEvenTransport bug - fixed

Slide 25

Distribution of Buggy Array Sizes

Slide 26

Bug is not detected at array size 1000

Slide 27

Number of Random Arrays to Detect a Bug

Slide 28

Generated Unit Tests