Thomas Axen - Lean Kaizen Applied To Software Testing - EuroSTAR 2010
-
Upload
eurostar-software-testing-conference -
Category
Technology
-
view
171 -
download
0
Transcript of Thomas Axen - Lean Kaizen Applied To Software Testing - EuroSTAR 2010
Lean Kaizen Applied To Software Testing
Thomas Axen
Agenda
Short about ATP - Test Organization
Why Test Kaizen - Blitz and Relationship to our test improvement
areas
Test Kaizen - Blitz seen from the theory to practices
The Test Kaizen - Blitz process in ATP
Test improvement overall view
Summary
Questions
4
5
The purpose of this presentation
To share my experiences with others
We hope that you can use some of this information in your
daily work
We hope that you come up with some good questions for me,
that I can use in my daily work
6
Short about ATP - General
Employees in ATP
- 800 FTE at Hillerød Campus
- 1000 FTE in 5 call centers
Employees in IT
- 210 FTE
- 80 Consultants
Employees in Test
- 27 FTE
- 23 Consultants
Customers– ATP livslang Pension (ATP)
– Arbejdsgivernes Elevrefusion (AER)
– Arbejdsmarkedets Erhvervssygdomssikring (AES)
– Barsel.dk
– DA-Barsel
– FerieKonto (FK)
– Finansieringsbidrag (FIB)
– Lønmodtagernes Dyrtidsfond (LD)
– Lønmodtagernes Garantifond (LG)
– Supplerende arbejdsmarkedspension for førtidspensionister
(SUPP)
– Den Særlige Pensionsopsparing (SP)
– Finansieringsbidrag (FIB)
– Skattenedslag for seniorer (SFS)
– Pension Danmark (PD)
– Objektiv sagsbehandling – New customers …
- Folkepensionen
- Udbetaling og indtægtsregulering af fortidspension
- Boligstøtte
- Barselsdagspenge
- Forskudsvis udbetaling af børnebidrag
7
Short about ATP - Test Organization
The test resources (50 FTE)
- 3 Test chiefs
- 6 Test Managers (RTM, TI and TM)
- 9 Test Coordinators
- 3 Test Specialists (automated test and test data)
- 6 IT-Testers and students workers (ITU and DTU)
- 23 Test consultants
Responsibility areas
- Test in projects and Sustained Engineering
- Test methods and tools (QC, QTP, ATSA, TDM)
Why Test Kaizen - Blitz
We have used Kaizen on other areas with great success
Top management – see it as a way of ”tuning” the organization
A god way of interacting with the stakeholders, to understand their
needs and attitude towards test
An new positive way of identifying improvements areas on test
All the testers can impact/influence on what the improvement
areas are and what we shall prioritize to work on
It verifies if we are working on the right improvement areas
8
Relationship to our test improvements areas
9
InspirationProcess
improvementImplementation
Test organisation
Improvements
Governance
Prioritizing
Execution
Improvement inspiration
Feedback
TMM assessment
Kaizen - Blitz
Guru
Network
Best Practice
Management decision
Strategy
Project
Sustained Engineering
Test Kaizen - Blitz seen from the theory to practices
Lean thinking:
Elimination of waste
Flow (production processes are made continuous)
Three central concepts:
Waste is ’everything the customer won’t pay for’
Flow implies elimination of batches and queues
Production is ’mass production’
10
…but Isn’t All Testing Waste?
The answer is yes in the context of software production:
11
Customers will not pay you for making errors.
Consequently customer will not pay you for finding and
correcting the errors you make.
But we are not looking at test in the context of software
production. We are looking at test in the context of a customer
checking the quality of software received from a vendor.
In this context test is not waste.
How Do We Identify Waste in Testing?
12
ATP chose Kaizen – Blitz from the Lean Toolbox.
Kaizen means ’continuous improvements’ and is “by the book”:
Dealing with a limited and thus manageable part of a production process (comparable to the work of 5 to 6 persons).
Being performed quickly but having ambitious goals for removing waste from the area under consideration.
Involving the 5 to 6 persons, who operate the process and are familiar with it.
Looking for improvements, which the persons involved are themselves in command of.
Being based on facts and measurements as far as possible.
Being repeated at regular intervals.
ATP chose a pragmatic approach to Kaizen
13
Dealing with a limited and thus manageable part of a production process. The
areas chosen were somewhat big, since it included both Domain, Integration,
and User Test.
Being performed quickly but having ambitious goals for removing waste from
the area under consideration. 3 short workshops over 6 weeks instead of 3
intensive days.
Involving exactly the 5 to 6 persons, who operate the process and are familiar
with it.
Looking for improvements, which the persons involved are themselves in
command of. Several useful improvements were “owned by other parties”.
Being based on facts and measurements as far as possible. Neither time nor
money for measurement on a large scale. (Common sense)
Being repeated at regular intervals. One time must be the first time.
Next time
Next time
The Test Kaizen - Blitz process in ATP
1 Step - Define the reason for the investment in kaizen
- Business case with focus on cost/benefit and scope.
- Communication to sponsor ongoing on status and progress
2 Step - ATP Test Kaizen kick-off
- Selecting the tester group that shall work with the kaizen process
- Short introduction to kaizen, and the plan (30 min.)
3 Step - Interviewing stakeholder and collecting facts
- The VoxPop (Voice of the Customer) input.
4 Step - Workshop, Testers as active players
- Present Voxpop and collected facts
- Kaizen scope and definition of waste in testing
- Value Stream (mapping all test activities and waste pr. Test phase)
5 Step - Workshop, RCA and classification
- Root Cause Analysis (Fishbone and selected defects analysis)
6 Step - Workshop, Consultation
- Classification and prioritization the improvement areas
- Finalize the proposal to management (the presentation)
7 Step - Summary (Presentation for managers and a execution plan)
14
3 Step - Interviewing stakeholder and collecting facts
Objective: To challenge the Kaizen team members attitude
towards their work.
15
Examples:
Far to many errors do not
surface before full production.
The Voice of the Customer:
- ”To little testing. To much test
administration. Spend more time testing!”
- ”I have never been on a project with so many test phases...”
- ...
4 Step - Workshop, Kaizen Scope and Definition of Waste
SIPOC - Scope of ATP Test Kaizen (Black outline in Kaizen scope)
Definition of waste in testing:
All activities not directly involved in finding errors or in
proving the absence of errors are considered waste.
All repetition of test etc. are considered waste.
16
ProgrammingComponent
test
Domain
test
Integrations
test
User
testOperation
4 Step - Workshop, Value Stream (mapping all test activities and waste pr. Test phase)
Objective: To get a clear picture of the time and resources spend on
testing and to identify activities where time and resources are
wasted.
What was done?
Detailed mapping of 3 subprocesses: Domain test,
Integration test and User test (3 value stream maps).
Identification of waste pr. subprocess.
Fordeling af kundeværdi og spil - Best case
0
50
100
150
200
250
300
350
400
1 2 3 4 5 6a 6b 6c 7 8a 8b 9b 9c 10 11
Aktivitet
Re
ss
ou
rce
forb
rug
/tim
er
NVA
BNVA
CVA
Example: Integration test
5 Step - Workshop, Root Cause Analysis
Objective: To prepare for elimination of waste by identifying the
underlying reasons.
What was done?
Systematic root-cause-analysis of all activities involving waste.
A. Testing is not finished within time and budget
I. To few people with proper business and domain knowledge in the individual test phases
a. Different (new) people are testing a given piece of software in each of the test phases.
1. To many handovers between phases.
6 Step - Workshop, classification and prioritization the improvement areas
What was done:
Brainstorm among participants on the basis of the Root
Cause Analysis.
Identification of 38 proposals
for elimination of waste.
Prioritisation of proposals:
’Resources vs. Effect’.
Anchoring:
Who is doing what?
Objective: To formulate operational proposals, leading to elimination
of waste in testing.
Low
Consider Don't
Hig
h
Do Consider
Low High
Resources needed for implementation
Eff
ect
on
waste
elim
inatio
n
6 Step - Workshop, Finalize the proposal to management
We identified 38 improvement areas
We grouped the improvement areas into 4 categories
- Go Fix – without any additional investments (9 areas)
- Investment – We need an investment based on a Business case and return of
investment analysis (3 areas)
- In progress – investment already in place project up an running (3 areas)
- Relationship to other areas (4 areas)
Rewriting of all the improvement areas so management can
understand the value of the investments
20
21
7 Step - Summary, “Go Fix” without any additional investments
Priority 1
- Tester should have more focus on progress, instead of roles
- Release management plan, must reflect dependencies so it is easy for the tester to plan
their test
- With critical issues, it should be possible to gather a team across the normal teams
Priority 2
- Post-mortem on found defect in production and projects (Root course analysis)
- Optimize the defect process
- General education plan (Consultants, PM, Business, Testers)
Priority 3
- Test cases must follow the BPT standard
- Improvement of the code documentation on defect and design changes, so test can
focus and minimizing the cost
- Quality assurance on deliverables in a structured way (Focus on static testing)
7 Step - Summary, “Investment” based on BC and ROI
Organization of the test pr. release
- All hours on test are used optimal – focus on planning, competences and the
team work together
Quality assurance of all software deliverables collected systematically
for reporting to management
- Systematic collection of data, for reporting to management so they can use it
for negotiations with vendors/teams on what the given quality should be for
each of them (Project and SE)
Release regressions test coverage
- Well documented release regressions test coverage pr. customer/solution,
so management can decide if we should “go live”
22
7 Step - Summary, “In progress”
Strategy for how to test letters
Test Data Management (Tools and reuse data)
More automated regression test (BPT, QTP)
23
7 Step - Summary, Relationship to other areas
Release management
- Overall plan, deliverables, dependencies, risk and issues.
- Requirements and change management
Deployment to test environments (36 applications using SOA)
- System test environments
- Integration test environments
Tester education with different focus
- Consultants, Business unit area, End users, Project management
Post-mortem on found defects In production and projects/Root cause
analysis
24
25
Test improvement overall view
Department vision/mission & overall goals- Be proactive (defects prevention, Innovation, think ahead)
- Agreements attitude (Clear, on time, expectation, follow-up)
- Well-known ABP (Stable, optimized, functional)
- Ready for changes (New things, flexible, developing and managed)
Test vision & overall goals- From known to respected in ATP
- Test in ATP should increase one level on TMM(i)
- Effective test with use of Best Practices and front-runners on selected areas
- Test process exists; it is always used and improved continually
- Competent/ professional test resources in the IT-organization
- Competent tests in the business-organization
Test leadership- Updating the A4 plans for the whole test area
- Updating detailed plans for each sub test areas
- Ongoing prioritization and follow-up on tests areas across applications
- Focus on improvements to minimize the effort
TMM – improvements (Level 3)- Continually optimizing/updating/support on test artifacts
- Continually optimizing all test processes
- Overview of all test tools and usages
- Extend the defect – root cause analysis
- More focus on non-functional requirement and test techniques
- More tester education to business, end users and consultants
- Focus on test data availability
- Use review for testing
- Metrics on test
Tester feedback loops- The best test case practice in ATP
- Optimize the usages of our test environments
- Optimizing of the review process on important artifacts
Test Kaizen – Blitz Improvements area- Quality assurance on deliverables in a structured way
- New test organization to handle releases
- Release regressions test coverage
- Tester should have more focus on progress, instead of roles
- Release management plan, must reflect dependencies so it is easy for
the tester to plan their test
- With critical issues, it should be possible to gather a team across the
normal teams, to analyse the issue
- Post-mortem on found defects in production and projects
- Optimize the defect process
- General education plan (Consultants, Business, End users, PM)
- Test cases must follow the BPT standard
- Improvement of the code documentation on defect and design
changes, so test can focus and minimizing the cost
- Strategy for how to test letters
- Test Data Management (Tools and reuse data)
- More automated regression test (BPT, QTP)
”Guru sanity check” Recommendations- All tester roles must be defined and communicated in ATP
- Test management meeting each 14 days
- PM gets a Tester education day, and understand each others roles
Governance projects- Extend the amount of automated regression test
- Optimizing of our Test Data Management area
- Stabilising of all reg. test cases for automation
Project managers improvements areas- Improve the test environment setup
- Test statistic (ex. defects in projects / production)
- Target groups on test artefacts (Who should read what)
- Move the test activities to the start of the project in the risk and QA
- Root cause analysis on project defects
Summary
We will definitely have a test Kaizen again, when we are ready
It is a good way of making sure that all stakeholders are being heard
Great way of getting a ”buy in” from all the testers, where the testers have an
influence on the improvements areas to execute on
It improves the image and awareness about Test in ATP
Remove activities that don’t add value
All relevant improvements areas are identified and prioritized
This is a “new angle” on ways to identify improvement areas
It also spins of improvement areas to other parts of the IT organization
Adjust the communication with respect, so the stakeholders can see the value of
each improvements area from their point of view.
26
27
Questions