© 2008 Hewlett-Packard Development Company, L.P.The information contained herein is subject to change without notice
Excellence Around the GlobeTest Factory Services
Your Name
Agenda
• Why care about quality?
• Professional testing in a factory?
• How do I to get there?
• Why HP?
• Who has done it?
Why care about quality?
Why care about quality?
• What is the impact of a project delay?−Project impact
• A four week delay for a 75 person project = $1.3M in cost− 75 people x 900/person/day x 20 days = $1.3M
−Business impact• In September 2004, HP struggled with SAP consolidation
project resulting in a three week delay for their supply-chain software.
Results: HP's ordering system became chaotic
• $400M loss in Q3 revenue
• Damaged reputation with customers
• Three executives were dismissed
• HP's shares dropped 15% on the news
If all that can happen is a project delay…
16%
27%
26%
28%
34%
29%
35%
31%
40%
28%
23%
51%
53%
46%
53%
33%
46%
49%
15%
18%
19%
0% 20% 40% 60% 80% 100%
1994
1996
1998
2000
2002
2004
2006
Succeeded Challenged Failed
Source: CHAOS Report 2007, Standish Group International, Inc.
Success of IT projects
8 04/11/23 HP CONFIDENTIAL
The gap between the specification...
And the user accceptance test
9 04/11/23 HP CONFIDENTIAL
The gap between improvements...
And the retest
10November 1, 2006 HP Confidential
Introduction of defects
Detection of defects
Source: NIST 2002 RTI Project 7007.011
• Majority of defects are introduced during the requirements and design phase!
• However majority of defects are actually detected during user acceptance testing and in production!
Understanding defects
Industry References: 3 B. Boehm and V. Basili, "Software Defect Reduction Top 10 List," IEEE Computer, IEEE Computer Society, Vol. 34, No. 1, January 2001, pp. 135-137.
This industry average is used as a baseline to calculate the cost savings
Assessing cost of defects
The HP Quality Methodology (STLC)
Deployed ApplicationDeployed
Application
Business Requiremen
t
Business Requiremen
tDesignDesign
Effortand
Cost
time
TraditionalCost
IdeaIdea ImplementImplement
Developer Test
ProductionRequirement Verification
Functional Test / E2E TestSystem Test User Acc. Test
OperationOperation
Test preparation
Planned Cost
Defect (ISTQB): A flaw in a component or system that can cause the component or system to fail to perform its required function, e.g. an incorrect statement or data definition. A defect, if encountered during execution, may cause a failure of the component or system.
Inserted defect: Defect created (inserted) in a phase
Contained defect: Defect found in the phase it originated
Introduced defect: Defect not found in the phase it originated
Detected defect: Finding of an Introduced defect
Defect detection rate & cost
Profesional testing in a factory?
Customer success roadmap defined
Level 0Quality
Agnostic
Level 1Quality
Initialising
Level 2Quality
Conscious
Level 3QualitySavvy
Level 4QualityExpert
Project-based technology, processes, & personnel
The majority of testing, if not all,
is still manual
No documented QA processes or
recognition of the need for them
Testing, if done at all, is manual
Centralised technology,
licenses, admin and support
QA process in place, but mainly
in individual projects
Centralized technology
Centralized service bureau
for QA personnel
Centralized methodology and best practices for
QA procedures
Centralized technology &
personnel
QA processes govern testing
and quality initiatives
QA is thought leader and has
enterprise influence
ProductUtility
ServiceUtility
Centre ofExcellence
Increasing Maturity
11.50 – 12.00 10m 1 of 2
Most organization
s
Introducing the HP Quality Model
Design Implement Unit Test
IT Supplier
Production
Business User
Business Analyst
Design
Requirements
Test Execution
Test Environment
Testing in the Lifecycle - today
Test definition
Test dataspecification
Test automation
Design Implement Unit Test
Production
Business User
Business Analyst
Design
Requirements
AcceptanceCriteria
IT Supplier
ManualTest
Test execution
Test environment
Quality Factory
Professional Testing
Roll Out
Test Factory Service Catalog (1)
18 April 11, 2023
Lifecycle Phase Service KPI
Idea • Impact analysis • Number of businesses Functions affected
• Function change rate
Analysis • Business Impact Analysis
• Test Effort calculation
• Test Case Change effort
• Regression Test effort
Design • Acceptance criteria / Test rule definition
• Test environment design
• Performance test scenario definition
• Number of issues found
• Number of issues accepted
• Functional coverage per business function
Test Factory Services Catalog (2)
19 April 11, 2023
Lifecycle Phase Service KPI
Implementation • Functional Test– Test environment
setup– Driver and Stub
development– Test data defintion– Test script
development / maintenance
– Test execution • Non Functional Test
– Test environment setup
– Driver and Stub development
– Test data defintion– Test script
development / maintenance
– Test execution
• Number of defects found
• Number of defects accepted
• Functional coverage per business function
• Number of concurrent users during performance test
• System response times
• Number of concurrent users during Load test
Test Factory Services Catalog (3)
20 April 11, 2023
Lifecycle Phase Service KPI
Deployment • Go Live test• AD hoc regression• Training
environment provisioning
• Functional coverage during ad hoc regression
Maintenance • Impact Analysis• Test Case
maintance• Regression Test
• Change Rate per function
• Number of regression test cases
• Function coverage during regression test
• Average system response time during performance / load test
How to get there?
Building a Test Factory
EnvisioningWorkshop
• Present HP as solution Partner• Agreement on milestones for engagement
• Solution Blueprint & Business Case
Development
Pilot installation
Tim
eTim
e
1-2 days1-2 days
10-20 days10-20 days
30-100 days30-100 days
>100 days>100 days
Build blueprint for first test factory
line
Set up first factory line and run one cycle
Application QualityRoll-out Service
Sample Timeline
Q308 Q408 Q109 Q209 Q309 Q409
Phase 1: Preparation
Phase 2: Setup the first line
Phase 3: Operate the first line
Phase 4: Evaluate the results and adjust
Phase 5: Setup the next lines
Providing real time status informationDashboards as Program “Cockpit”
Application Management Dashboard - EriDocApplication Management Dashboard - EriDoc
Demand / Project Status
Performance Test Status
Functional Test Status
Controlling the FactoryConsolidated top-down planning & bottom-up execution
Project proposals
Operational work
Budgeting/costing
Resource capacity analysis
Project scheduling
HP PPM
Center™
Visibility Controls Flexibility
Governance structures Business rules
Workflow Notifications EscalationsAudit trail
Issues, risks, scope changes
Testing issues (defects)Change management
Service desk or 3rd party integrations
Time tracking
Why HP?
• Improve performance• Reduce backlog• Ensure quality
27 04/11/23 HP CONFIDENTIAL
Simplify Standardize Optimize
Strategic quality
methodology
Market-leading
HP Software
Global delivery
capabilities
Quality
HP Quality Services tri-dimensional approach aligns business with IT
Simply with HP Quality Methodology
28 04/11/23 HP CONFIDENTIAL
Simplify Standardize Optimize
Strategic quality
methodology
Market-leading
HP Software
Global delivery
capabilities
Quality
• Supports the application lifecycle, requirement to deployment
• Integrates with every development methodology
• Supports reuse of existing artifacts
• Incorporates the business • Bridges the
communication gap between IT and business
Blueprint - Based on real world experience
Standardize with HP Software Tools - • Market leading product (60% +)• Requirements management, risk-
based testing, and cycle management
• First real-time link between business requirements and quality in a single enterprise platform.
• Delivers risk-based release decisions and end-to-end traceability.
• Introduces dimension of time and release cycles for agile quality
• Web 2.0 • Performance validation in a world of
AJAX and FLEX• Accelerated testing• SAP “TAO”• HP Quality Management
Ecosystem™
30 04/11/23 HP CONFIDENTIAL
Simplify Standardize Optimize
Strategic quality
methodology
Market-leading
HP Software
Global delivery
capabilities
Quality
31 04/11/23 HP CONFIDENTIAL
Optimize with HP Global Delivery• 400+ plus testing
professionals across GDAS Centers
• Skilled professionals: Test Managers, Test Architects, Test consultants, Test Leads, Manual Test Engineers, Automated Test Engineers and Performance Test engineers, etc…
• 150+ certified professionals - CSTE, CSTM, CSTP ISTQB, Brain bench. etc…
• Strong people retention & development practices
Simplify Standardize Optimize
Strategic quality
methodology
Market-leading
HP Software
Global delivery
capabilities
Quality
• Learning and Development Program (Grow@hp)−Certified Tester Foundation Level
−Certified Tester Advanced Level (Test Management)
−Certified Professional for Requirements Engineering
• Certification Program
Qualification
Who has done it?
Total Cost of Defect Detection Project Example I – Automative Industrie (Experience)
34
Test
ing
Addendum
Average Defect Number in comparable Projects (Experience)Standard Software (SAP/Siebel) /high Complexity due to Functionality / Interfaces
HP approach Classic approach
Cost $/Defect
Defect %
Defect absolute
CostDefect
%Defect
absoluteCost
Requirements $
139,00 65% 2500 $ 347.500,00 5 250 $ 34.750,00
Unit Testing Functional Testing
$ 977,00 28% 1500 $ 1.465.500,00 15 750 $ 732.750,00
Integration Testing 5% 200 $ 195.400,00
System Testing
UAT $
7.136,00 2% 50 $ 356.800,00 60 3000 $21.408.000,00
Production
$14.102,0
0 1% 40 $ 564.080,00 20 1000 $14.102.000,00 Defect Costs (total) $ 2.929.280,00 $36.277.500,00
Savings (total) $33.348.220,00
KMDInformation Technology Services
35 April 11, 2023
IT improvements
Business Outcomes
Objective Approach Results
• To remain one of Denmark’s top three IT service providers
• To assure customer satisfaction by improving operational quality
• 2004: Initiated IT lifecycle management plan
• Created a Quality Center of Excellence built on HP Quality Center software
• Now accommodates 1600+ users and 150 test projects
• Established an Operational Quality Center powered by HP Performance Center software and HP Business Availability Center software
• HP Universal CMDB integration for 3rd party help desk solution and all HP management software
• 96% customer satisfaction with testing services has led to referrals and new business
• 18% revenue growth: 2006
• Reduced time to repair from hours to minutes
• Reduced risk through − Functional and
performance testing and monitoring
− 99.5% - 100% availability
• Rapid root cause analysis saves ~$300k, or 2200 work hours, per year
• Able to offer new services to its customers
Technology for better business outcomes
Top Related