Sogeti Webinar Effective Test Process Improvement 220709
-
Upload
sogeti-ireland -
Category
Technology
-
view
104 -
download
1
description
Transcript of Sogeti Webinar Effective Test Process Improvement 220709
© Sogeti Group 2009 Page 1
Sogeti Ireland
Effective Test Process Improvement
Fran O’Hara
22nd July 2009
© Sogeti Group 2009 Page 2
Contents
• Introduction
• TPI
• TMMi
• Process improvement challenges and sample solutions
• Summary
• Q&A
© Sogeti Group 2009 Page 3
3
SOGETI Overview: A Worldwide Leader in IT Services
• Sogeti Group� 20,000 professionals
� 200 locations in 14 countries
• Providing worldwide standards� Copyright TMap® and TPI® brands
� Dedicated R&D on new developments
� Test thought leadership
Part of• Leader in consulting, IT &
outsourcing
• 88,000 staff
• Revenue €8.7 billion
• Sogeti:
• £1.6 billion revenues
• Testing revenues
£150M (2007), £190M
(2008)USA
France
Spain
Sweden
BeluxGermany
Switzerland
NL
DenmarkUKIreland
India
Test Centre of Excellence
provides for • Reduced cost (>25% over 2 yrs)
• Improved quality
• Improved flexibility
• Reduced time-to-test/market
• Reduced risk
• Increased capacity
© Sogeti Group 2009 Page 4
People Process Technology
Capability
Products/services
World ClassProcess focus improves
predictability of
performance (reduces
variation), as well as
performance itself
(difference between
desired and actual)
Test Process Improvement - learning to work smarter,
to test software better, cheaper, faster
© Sogeti Group 2009 Page 5
Triggers
• Efficiency/cost?
• Effectiveness?
• Growing Complexity?
• Different levels of Test Maturity?
• Time Pressure?
• Are we doing the right thing(s)?
Development
Agile
XMLGUI’s
Integration
Automation
Webserv
ices
CBDCOTS
Development
Agile
XMLGUI’s
Integration
Automation
Webserv
ices
CBDCOTS
Business
Time-to-market
Error free
Quality
Cheap
Reuse
User-friendlinessMaintainability
Standard.
GrowingComplexity
Business
Time-to-market
Error free
Quality
Cheap
Reuse
User-friendlinessMaintainability
Standard.
GrowingComplexity
Tester
© Sogeti Group 2009 Page 6
Key areas
Levels Test
Maturity
Matrix
Improvement suggestionsCheckpoints
The TPI® Model
© Sogeti Group 2009 Page 7
• Test strategy
• Life-cycle model
• Moment of involvement
• Estimating and planning
• Test specification techniques
• Static test techniques
• Metrics
• Test automation
• Test environment
• Office environment
• Commitment and motivation
• Testing functions and training
• Scope of methodology
• Communication
• Reporting
• Defect management
• Testware management
• Test process management
• Evaluation
• Low-level testing
20 Key Areas
© Sogeti Group 2009 Page 8
Key area / Scale 0 1 2 3 4 5 6 7 8 9 10 11 12 13
1 Test strategy A B C D
2 Life -cycle model A B
3 Moment of involvement A B C D
4 Estimating and planning A B
5 Test specification techniques
A B
6 Static test techniques A B
7 Metrics A B C D
8 Test automation A B C
9 Test environment A B C
10 Office environment A
11 Commitment and motivation
A B C
12 Test functions and training A B C
13 Scope of methodology A B C
14 Communication A B C
15 Reporting A B C D
16 Defect management A B C
17 Testware management A B C D
18 Test process management A B C
19 Evaluation A B
20 Low -level testing A B C
Current Situation (example)
Dependencies!
© Sogeti Group 2009 Page 9
Process Improvement ….
Key area / Scale 0 1 2 3 4 5 6 7 8 9 10 11 12 13
1 Test strategy A B C D
2 Life - cycle model A B
3 Moment of involvement A B C D
4 Estimating and planning A B
5 Test specification techniques
A B
6 Static test techniques A B
7 Metrics A B C D
8 Test automation A B C
9 Test environment A B C
10 Office environment A
11 Commitment and motivation
A B C
12 Test functions and training A B C
13 Scope of methodology A B C
14 Communication A B C
15 Reporting A B C D
16 Defect management A B C
17 Testware management A B C D
18 Test process management A B C
19 Evaluation A B
20 Low - level testing A B C
Controlled EfficientOpti
mizing
© Sogeti Group 2009 Page 10
Testing Maturity Model Integration (TMMi®)
• Currently under development by the TMMi Foundation� See www.tmmifoundation.org� Levels 2 & 3 published
• The TMMi® Foundation will provide:� A standard TMMi® Model that can be used in isolation or
in support of other process improvement models and is either staged or continuous.
� An independently managed data repository to support TMMi® assessment method accreditation, assessor and assessment certification/validation and validated assessment data and certificates.
� Assessment Method Accreditation/Audit Framework for TMMi® in accordance with ISO15504 and the process to certify commercial assessment methods against the standard model.
� Certification and training/examination process, procedures and standards for formal, public accreditation of Assessors and Lead Assessors and the on-going management.
© Sogeti Group 2009 Page 11
TMMi® Model
Test Organisation
Test Training Program
Test Life Cycle and Integration
Non-Functional Testing
Peer Reviews
3 Integration
1 Initial
Test Policy and Strategy
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
2 Managed
Test Measurement
Software Quality Evaluation
Advanced Peer Reviews
4 Management & Measurement
Defect Prevention
Test Process Optimisation
Quality Control
5 Optimisation
Process AreasLevel
© Sogeti Group 2009 Page 12
Problems with Process Improvement
Typical Issues with Process Improvement:
� Lack of management commitment
� Lack of prioritised goals to focus/prioritise
� Excessive emphasis on documenting and assessing process
� Difficulty in defining and deploying the process
� Inadequate buy-in from staff
� Lack of process architecture
Results from Sogeti ‘Worldwide survey on TPI’ :
� Lack of management commitment **
� Unrealistic expectations *� Unclear scope and target (i.e. goals) *� Unclear communication� Only 17% had measurable results
� Results unknown for too long a time
� Lack of change process� Change team without influence
� Involvement of people within the test process*� Top down approach (‘Thou shalt..’)
� 44% combined with other improvement projects
© Sogeti Group 2009 Page 13
Improvement Strategy - 1
Achieve sign-off on documented requirements and constraints and associated improvement (project) plan
Align and prioritise improvement efforts with business goals through a goal-problem tree
Focus on early and measurable results
Lack of management commitment
- Lack of management commitment**
Lack of prioritised goals to focus/prioritise
- Unrealistic expectations*
- Unclear scope and target (i.e. goals)*- Unclear communication- Only 17% had measurable results - NB
Solutions
Problems
© Sogeti Group 2009 Page 14
Problem Scenario - 1
0
10
20
30
40
50
60
70
80
1 2 3 4 5 6
Months
Reported Production Defects
# Severity 1 & 2 defects
Customer
satisfaction
rating dropped
20% in June
survey
compared to 6
months
previously.
Cost reduction
of 10% is also
a department
target
© Sogeti Group 2009 Page 15
Problem Scenario -2
Origin of Production Defects
User requirements
System requirements
Architecture
Detailed design
Code & Build
Integration
Data also
shows that
rework of
production
defects is
costing 40% of
development
budget
© Sogeti Group 2009 Page 16
• Perform formal peer reviews of System requirements� Involve system testers
• Use a risk-based approach to system test involving business users in risk workshops
• Use a test driven development approach to unit test (unit test cases developed and automated before code written)
• Design system tests as soon as system requirements become available
• Resource system test with some dedicated test professionals
• Perform formal planning of system test
• Measure DDP for peer reviews, unit test and system test
• Some ways to identify improvement actions:� Brainstorm process problems and vote
� Brainstorm improvement ideas/actions
� Model gap analysis – assessment results
Examples of test-related improvement actions
© Sogeti Group 2009 Page 17
Goal Tree & Cost/Benefit
Satisfied customers
0As aboveReduce number of production defects per month by 50% (based on equivalent appn. throughput)
Improve Customer satisfaction ratings by 20%
1m
(10% of IT budget)
200kImprove effectiveness of Peer reviews, Unit test, and System Test by 75% (Currently running at 50% DDP)
Reduce rework costs of production defects by 50%
Reduce IT Appn. Development costs by 10%
Est. Benefit
Est. Cost of required actions
Test Process Goals
Dev/Test Department Goal
Business Goal
© Sogeti Group 2009 Page 18
Goal Metric Scenario – Sample Results
Defect Detection Percentage per Release
0
20
40
60
80
100
1 2 3 4 5 6
Release number (or subsequent projects)
DDP
© Sogeti Group 2009 Page 19
Improvement Strategy - 2
Use a more agile approach to PI - Iterate in short improvement cycles (2-3 months) – how do you eat an elephant?
Focus on lightweight, rapid process modelling for early deployment (80/20 rule for any gap analysis)
Perform formal assessments to confirm process maturity results
Excessive emphasis on documenting and assessing process
- Results unknown for too long a time
Solutions
Problems
© Sogeti Group 2009 Page 20
Improvement Strategy - 3
Use a facilitated workshop approach to process modelling/improvement
Process owners involved in workshops, ‘change team’become facilitators
Goal–problem driven focus
Use a continuously improving well-defined change process
Difficulty in defining and deploying the process
�Lack of change process
�Change team without influence
Inadequate buy-in from staff
�Involvement of people within the test process*
�Top down approach (‘Thou shalt..’)
Solutions
Problems
© Sogeti Group 2009 Page 21
Example Modeling Session
•Picture of process-level modeling (Image2)
•session inserted here.
© Sogeti Group 2009 Page 22
Example “Mapped” Element
© Sogeti Group 2009 Page 23
When a standard framework is required –
sample process level template
Inputs Entry Exit Outputs
Activity Name Activity Purpose Activity Controls
Measurements
Costs, Efforts,
Time, Quality
RolesIssues
Planning,
Tailoring, …
Tasks
Action
Action
Action
Action
Artifact
Artifact
Artifact
Action
Condition
Condition
Condition
Condition
NameName
ArtifactArtifact
Artifact
Artifact
Level of Abstraction of tasks is
key to a common frameworkActivity Controls or ‘Work Aids’ provide options
for supporting templates, tools, procedures, etc.
Tailoring can also be used to highlight
key differences of approach required
© Sogeti Group 2009 Page 24
Sample Test Process Element
Define the Test Strategy
Metrics1. Planned versus actual effort
Roles1. Test Manager (T.1-T.7)
2. Test Designer (T.5)
3. Business Analyst (T.3, T.7)
4. System Expert (T.3-T.7)
5. End User (T.3, T.7)
Tailorings
Outputs1. (Risk Based) Test Strategy.
2. Prioritised Risk List.
Tasks1. Determine level of Requirements definition for the
project
2. Divide the System under Test into its components, if relevant. Prioritise component parts
3. Identify and analyse Product Risks, taking Project Requirements, Statement of Work, Prioritized Quality Characteristics and System Components into consideration
4. Identify and Define Test Levels.
5. Identify and Define Test Level Attributes - Methods and Test Techniques, exit criteria, coverage, automation approach, etc.
6. Document the Testing Strategy.
7. Review and sign-off the Testing Strategy.
Inputs1. Project Requirements
Document(s) [User, System, Functional/Technical]
2. Statement of Work
3. Test Project Scope
Exit Criteria1. Test-ware is placed under
Configuration Management control.
Controls / Work Aids1. Project Lifecycles with defined Test Stages
2. Test Methodology (identifying Test Methods & Test Techniques.)
3. Review Meeting Minutes Template
4. Risk List Template
5. Test Strategy Template
Entry Criteria1. Inputs are under
Configuration Management control
© Sogeti Group 2009 Page 25
Performevaluation
Formulateplan
Implementimprovement
actions
Obtainawareness
Executeassessment
Defineimprovementactions
Determine target, area of consideration
and approach
Process of change
© Sogeti Group 2009 Page 26
Suggestion: Use a performance-based ‘agile’
process improvement approach
Phase 1: Selected
goals and problems
for process
improvement
Phase 2:
Documented
baselines of
current practice
and performance
Phase 3: Process
documents to be
deployed
Phase 4: Revised
process and
performance
baselines
© Sogeti Group 2009 Page 27
• Clear scope and target 16%
• Clear communication 9%
• Results periodically visible 8%
• Champions in change team 8%
• Use of a change process 7%
• Other ... 5%
Biggest DO’s: • Management commitment 24%• Involvement of “own” people 23%
Do’s – from TPI® survey
© Sogeti Group 2009 Page 28
Summary – key suggestions
• Use an agile approach to test process improvement – incremental with 80/20 prioritisation� Driven by and linked to measurable business goals
� Addressing key staff process problems
• Involvement and ownership by test staff
• Lightweight, usable process definition (80/20 again!)
© Sogeti Group 2009 Page 29
Sogeti Support
• Capability building
� test process and how to improve it
� seminars, workshops and training
• Process assessments
� Model based (e.g. TPI®) and/or benchmarking
• Consulting/mentoring during process improvement
• www.sogeti.ie