1

Using Control Charts for Detecting and Understanding Performance

Regressions in Large Software

Thanh NguyenSupervisor: Ahmed E. Hassan

Queen’s University, Kingston, Canada

2

Motivation

3

Performance is very important aspect in modern software engineering

LocalSingle user

Asynchronous interaction

Before Now

NetworkMassive multi-usersSynchronous interaction

Facebook50%

Canadian

190

4

Performance testing is very important to ensure software performance

Local

Single user

Asynchronous interaction

Network

Massive multi-users

Synchronous interaction

Higher latencyRequire fast servers

Require central storage

Need states synchronization

Need guarantee response timeNeed redundancy

5

I focus on one kind of performance testing: regression load test

Version 1 Version 1.1

Baseline Target

6

How to detect performance regression?

Applying load

Version 1.1

Version 1 CPU %, Memory usage

CPU %, Memory usage

Detect regression

Challenge in Performance Regression Testing

7

Layer 1 – Agent 1

Layer 1 – Agent 2

Layer 2 – Agent 1

Layer 2 – Agent 2

Layer 2 – Agent 3

Layer 2 – Agent 4

Layer 3 – Agent 1

Layer 4 – Agent 1

56 counters x 8 agents = 448 counters

56 counters x 2 agents = 112 counters

Layer 1 Layer 2

Lots of data

Not enough automation Limited research

8

Approach

9

Data mining -> R&R

Reduce Relate

10

Performance regression detection is similar to monitoring and forecasting production servers load.

11

Disk Subsystem Capacity Management [Trubin and Merritt 2003]

12

Proposed approach to use control charts to find performance regression

BaselinePerformance counters

TargetPerformance counters

Determine the LCL, CL, UCL

0 5 10 15 20 25710

720

730

740

750

760

770

780

Performance counter

13

Using control charts to verify load test results

BaselinePerformance counters

TargetPerformance counters

Determine the LCL, CL, UCL

0 5 10 15 20 25710

720

730

740

750

760

770

780

Performance counter

Violation ratio

Reduce

14

BaselinePerformance counters

TargetPerformance counters

TargetPerformance counters

0 5 10 15 20 25710720730740750760770780

Performance counter

BaselinePerformance counters

0 5 10 15 20 25680

700

720

740

760

780

Performance counter

Low violation ratio

High violation ratio

We can use violation ratio to detect regression Relate

15

Progress

16

My three research questions

Is there a performance regression?Fairly confident that it works [ICPE 12]

Where does performance regression occur? Need more validation [APSEC 11]

What causes the regression? In progress

17

Experiment set up

BaselinePerformance counters

TargetPerformance counters

TargetPerformance counters

Number of out-of-control counter is smallAverage violation ratio should be low

18

Experiment set up

BaselinePerformance counters

TargetPerformance counters

TargetPerformance counters

Number of out-of-control counter is largeAverage violation ratio should be high

19

Bad Query

Limit

Extra p

rint

Extra c

onnection

Missing k

ey index

Missing t

ext index

02468

1012141618

NormalProblem

Series3

Number of out-of-control counters

20

Bad Query

Limit

Extra p

rint

Extra c

onnection

Missing k

ey index

Missing t

ext index

0.00%10.00%20.00%30.00%40.00%50.00%60.00%70.00%80.00%90.00%

100.00%

NormalProblem

Series3

Average violation ratio

21

Where does performance regression occur?

Backend

Other backend

Front End

22

Backend 50% stress

B B B B B O0%

10%20%30%40%50%60%70%80%90%

100%

Violation ratio

23

Frontend 50% stress

F F F F F F O0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Violation ratio

24

Obstacles

25

Obstacles #1: Inputs are unstable

1 2 3 4 5 60

5

10

15

20

25

30

35

40

45

Version 1.0Version 1.1

Time

CPU

%

Is there a performance regression?

26

It is very difficult to maintain stable input across test runs

Applying load

Version 1.1

Version 1 CPU %, Memory usage

CPU %, Memory usage

Detect regression

Randomization CacheWarm up

Background tasks

27

Solution #1: Scale the counter according to the input

• Step 1: Determine α and β

• Step 2:CPU% Request/s

May not work?

28

Solution #1: Example of the effectiveness of scaling

29

Obstacles #2: Multiple inputs

10 20 30 40 50 60 70 80 90 1000

5

10

15

20

25

30

35

40

45Density plot of two test runs

Version 1.0Version 1.1

CPU Usage

Dens

ity %

IF … THEN…ELSE…

30

Solution #2: Isolating the counters

10 20 30 40 50 60 70 80 90 1000

5

10

15

20

25

30

35

40

45Density plot of two test runs

Version 1.0Version 1.1

CPU Usage

Dens

ity % Local minima

31