1

SLA For Testing –

Approach to Quantify

Quality

Thomas Rumi

Imbus AG

2© 2005 Slide 2

Thomas Rumi, imbus AG

EuroStar 2005, Copenhagen

SLA for Testing

© 2005 Slide 3

About imbus

• Founded in 1992 in

Germany

• Specialist for Software

Quality Assurance and

Test

• ~100 employees at three

locations

• Headquarter in Möhrendorf

(near Nuremberg)

• Munich

• Hofheim

(near Frankfurt/Main)

© 2005 Slide 4

Services and Solutions

Consulting

• SW QA

• SW Test Process

Softwaretest

Services

• Test

Outsourcing

• SLA for Testing

Training

• ISTQB

Certified Tester

• Trainings

worldwide

Products &

Solutions

• imbus TestBench

© 2005 Slide 5

SLA for Testing - Agenda

Service Level Agreement (SLA)

Examples from other domains

Characteristics of SLAs

SLA for Testing

Challenges

Approach

Example - How to find KPIs for testing?

Summary and Outlook

© 2005 Slide 6

Example - SLAs in a DSL offer

DSL offer from Nefkom (http://www.maxidsl.de)

Service Level name

Optional conditions

Key Performance

Indicators (KPIs)

Price

© 2005 Slide 7

Example - SLAs in Help desk services

End users

1st le

ve

l su

pp

ort

2nd level support

Expert

Phone

Web portal

E-mail

Expert

Expert

Key performance

indicators

(KPIs)

• Service hours

• Timeframe for trouble ticket

confirmation

• 2nd level support within guaranteed

time

• Customer Satisfaction Surveys

© 2005 Slide 8

Characteristics of SLAs

Contract between a customer and a service

provider

Formalized service quality by service

levels

Service level definition based on clearly

defined and measurable KPIs

Guarantee of the service level

Monitoring of service level based on KPIs

Direct mapping of service level and price

© 2005 Slide 9

SLA Process

Service Level Requirements

Service Level Agreement

(Quality guarantees and prices)

Service execution

(including SLA monitoring)

Payment with bonus and remedies

SLA assessments /

SLA improvement process

Customer Service

Provider

© 2005 Slide 10

Benefits of SLAs

Get the appropriate level

of quality

Prevent too much or too

little service

Guarantee that service

quality is good enough

Optimal quality / cost ratio

Get satisfied customers

Visualize service ability

and quality

Be comparable against

competitors

Optimize resource

planning

Customer Service

Provider

© 2005 Slide 11

SLA for Testing - Agenda

Service Level Agreement (SLA)

Examples from other domains

Characteristics of SLAs

SLA for Testing

Challenges

Approach

Example - How to find KPIs for testing?

Summary and Outlook

© 2005 Slide 12

SLA for Testing - Roles

Product

Development

Test

Outsourcing

Provider

Service

ProviderCustomer

© 2005 Slide 13

SLA for Testing - Challenges

Find clearly defined and

measurable KPIs

Develop new Service Levels

Restrict additional monitoring

overhead

SLAs are new on the testing

market

Adaptable Service Level for

testing currently not available

© 2005 Slide 14

SLA for Testing - Approach

Step 1: Service Level Requirements (SLR)

Step 2: Identify the KPIs

Step 3: Define Service Levels

Step 4: Assign prices to Service Levels

Step 5: Improve and maintain Service Levels

SLA Framework

© 2005 Slide 15

Costs

Productivity

Quality

Step 1: SLR - Example: Test Management

Minimal costs

Fast execution

Run the right tests

Quick startup of tests

Be flexible on release delays

Provide clear test status report

Appropriate reporting intervals

© 2005 Slide 16

Step 1: SLR - Example: Bug Reporting

Quality

Report only essential bugs

Prevent misleading or

duplicated bugs

Precise and clear bug reports

Bug reproducible by

development team

Point to root cause

© 2005 Slide 17

Step 2: Identify the KPIs - Example: Bug Reporting

Measure test

effectiveness

DDP

Bug rates against

prev. projects

Measure the ratio of

bad bug report

Control QA activities

for bug reports

Review

Reproduce bug

Pre-analyze bug

Evaluate feedback for

bug report from

Product

management

Development

# (Duplicates + Closed Bugs)

# (All found Bugs)=

© 2005 Slide 18

Step 3: Define Service Levels -

Example: Bug Reporting and Reporting Intervals

Service Levels

Category Basic Advanced Expert

QA activities for bug reports

Critical / Serious Reproduce

Reproduce

Review

Reproduce

Review

Root cause

analysis

Medium

Reproduction

not guaranteed Reproduce

Reproduce

Review

Low No bug report

Reproduction

not guaranteed Reproduce

Reporting interval

Test status reports monthly weekly daily

Escalation reports weekly weekly immediately

© 2005 Slide 19

SLA for Testing - Agenda

Service Level Agreement (SLA)

Examples from other domains

Characteristics of SLAs

SLA for Testing

Challenges

Approach

Example - How to find KPIs for testing?

Summary and Outlook

© 2005 Slide 20

Summary

SLA are useful for testing

service, but testing services

are complex

Main challenge is to identify

essential KPIs

imbus has launched test

outsourcing services with

SLAs

First customer project with

SLAs is up and running

© 2005 Slide 21

Outlook

Our strategy - SLA for

Testing

Extend SLA offer for all

test services

Improve current KPI

approach within SLA

improvement process

© 2005 Slide 22

Interested in further information?

WWW: http://www.imbus.de/engl/testlab

E-Mail: Thomas.Rumi@imbus.de

Space shuttle Atlantis landing at Edwards Air Force Base, CA. (32K) Image: NASA