Test and Verification Solutions129 April 2009 Managing Outsourcing - Case Studies Presentation to...
-
Upload
julius-bradford -
Category
Documents
-
view
214 -
download
0
Transcript of Test and Verification Solutions129 April 2009 Managing Outsourcing - Case Studies Presentation to...
Test and Verification Solutions 129 April 2009
Managing Outsourcing - Case Studies
Presentation to TMF
April 2009
Mike Bartley, TVS
2Test and Verification Solutions 229 April 2009
What is “successful” software testing?
Outsourcing Success
Insource Onshore Near Shore Offshore
Budget L Schedule M Quality H
Onsite contractor
Offsite UK contractor
Offsite non-UK contractor (e.g. Ireland)
Offsite non-UK
contractor (e.g. India)
Minimise budget
Hit schedule
Appropriate quality
Low impact on minimising budget
Outsourcing Success
Insource Onshore Near Shore Offshore
Budget L L - M Schedule M M Quality H H
Outsourcing Success
Insource Onshore Near Shore Offshore
Budget L L - M M Schedule M M M Quality H H H
Outsourcing Success
Insource Onshore Near Shore Offshore
Budget L L - M M H Schedule M M M H Quality H H H H
Outsourcing Success
Insource Onshore Near Shore Offshore
Budget L L - M M H Schedule M M M H Quality H H H H Risk L L - M M H?
Outsourcing Success
Insource Onshore Near Shore Offshore
Budget Schedule Quality
3Test and Verification Solutions 329 April 2009
The main risks
• Supplier corporate governance– “$1 billion of the cash reported being on Satyam's books
was utterly "fictitious" ” Times Newspapers Online, 13th Jan `09
• Over-inflated expectations– “executives assume that labor arbitrage will yield
savings comparable to person-to-person comparison” META Group Inc, 16th Feb `09
• The usual project risks are even greater– “Project managers must recognize that offshore projects
will likely require even greater management of risk due to the unique challenges posed by geographical, cultural, and other differences”Ralph Kliem, "Managing the Risks Of Offshore IT Development Projects“, Information Systems Management, 2004, Vol. 21 Issue 3
4Test and Verification Solutions 429 April 2009
The main risks
• Problems of a distributed team– Communication– Knowledge management– Failure to deliver required deliverables
• Stability of requirements and specifications• Security• Transition and change risk “I often have to caution
them not to be swayed by the “economics” without understanding all of the
“costs” involved”Rajiv Dholakia, VP Product Development, Cenzic Inc.,
“Happy About Outsourcing, 2005
• Do you know your tolerance for risk?
• How well do you manage risks?
5Test and Verification Solutions 529 April 2009
Agenda
• Agenda
– Some background
– How to successfully manage software testing offshore• Setting up the relationship
• Managing the ongoing relationship
– Q&AMaximise the benefits
Minimise the risks
6Test and Verification Solutions 629 April 2009
Outsourcing – some facts!
• Market size and share– India’s share of the global
offshore outsourcing market for software and back-office services is 44%
• Preferred destinations– Trend to look for cities
rather than countries
Global Services’ Top Outsourcing Cities for IT and BPO
Top Outsourcing Cities
Bangalore India
Chennai India
Delhi NCR India
Dublin Ireland
Hyderabad India
Makati City The Philippines
Mumbai India
Pune India
Source: Global Services October 2008
USD billion 2005-06 2006-07 2007-08
I T Software and Services Exports
17.3 22.9 28-29
Source: Nasscom
7Test and Verification Solutions 729 April 2009
Why do companies outsource?
• Access to flexible resource– Ramp-up and ramp-down
• Time to market• Concentrate on core skills• Process improvements
– Outsource suppliers tend to have a strong focus on process
• Quality improvement– Access to specialist resources
• Cost– Reduction– Variabilisation!
Outsourcing allows a small company to grow
aggressively without taking on too much fixed cost
Outsourcing allows a large company to variabilise its fixed costs and potentially
reduce overall costs
8Test and Verification Solutions 829 April 2009
The outsourcing process
• Define the strategic goals• Decide what to outsource• Review your readiness to outsource• Select your preferred outsource model• Select your outsource partner• Agree the
– Master Services Agreement – Service Level Agreement
• Outsource execution– Management– Governance
9Test and Verification Solutions 929 April 2009
Effective outsourcing - Defined strategic goals
• Quality improvement– Independent test team– Get access to specialist skills– Getting access to higher quality staff
• Improve company focus• Reduce time to market
– Through improved company resource flexibility • Cost variablization• Cost reduction
• Should be agreed by the appropriate stakeholders– the software manager– the testing manager– the engineering manager– the chief financial officer– the outsourcing manager.
“Successful outsourcing activities are ones which are derived from strategic
objectives of the
companies involved”Bill Widmer, CEO g8solutions,
“Happy About Outsourcing, 2005
Having agreed strategic goals allows low level
decision making without continuous approval from higher level management
10Test and Verification Solutions 1029 April 2009
The outsourcing process
• Define the strategic goals• Decide what to outsource• Review your readiness to outsource• Select your preferred outsource model• Select your outsource partner• Agree the
– Master Services Agreement – Service Level Agreement
• Outsource execution– Management– Governance
11Test and Verification Solutions 1129 April 2009
Deciding what to outsource
Control
Input OutputAlgorithm
New interactive “pad and mouse” device for computer interaction by disabled users
• The “special sauce” is in the algorithm• Develop algorithm unit tests and user
acceptance tests in-house• Outsource IO test development and
running system tests
Silicon
Driver
Libraries
ApplicationsSilicon manufacturer delivering
applications to run on their chipsNew applications being developed
• The variability and specialist domain knowledge is at the “application level”
• But the lower levels have stable APIs• Driver and library testing offshored• Application testing offshored as they stabilise
12Test and Verification Solutions 1229 April 2009
Deciding what to outsource ctd.
Interactive e-learning package using iterative development
• The “special sauce” is in the domain knowledge and pedagogy
• Outsource the GUI testing and “platform” testing
Bespoke application development using V-model• Detailed component level
specifications• Component testing must be
independent and achieve high structural coverage targets
• Outsource the component testing – black-box first, then white-box
Important considerations• Intellectual property issues• Proximity to product development• Complexity of interface• Minimise learning curve/knowledge transfer• How stable are the requirements/spec/design?• Is there a standard available?• Dev process: V-model vs. Agile
13Test and Verification Solutions 1329 April 2009
What development process do you use?
The V model• A very formal
development process
• Specifications are (in theory) written at the various levels
• Allowing early test spec and dev
• This lends itself well to outsourced testing
14Test and Verification Solutions 1429 April 2009
What development process do you use?
• Do agile methods work with outsourcing?• Quality of communication
– Bandwidth– Latency – Reliability – Availability
• There are a few examples of multi-site agile projects– Practical issues of daily scrum meetings, sharing burn down charts,
information radiators, story cards, …
• & fewer examples of offshored testing in agile projects• But …
– Testing the output of each time-boxed iteration– Regression testing to ensure older functionality doesn’t rust
15Test and Verification Solutions 1529 April 2009
The outsourcing process
• Define the strategic goals• Decide what to outsource• Review your readiness to outsource• Select your preferred outsource model• Select your outsource partner• Agree the
– Master Services Agreement – Service Level Agreement
• Outsource execution– Management– Governance
16Test and Verification Solutions 1629 April 2009
Effective outsourcing - Readiness for Outsourcing
• The Capability Maturity Model – Level 1 - Ad hoc (Chaotic) – Level 2 - Repeatable – Level 3 - Defined – Level 4 - Managed – Level 5 - Optimizing
• Less formally– Requirements management– Change tracking and management– Project planning, monitoring and control– Configuration management– Build, test and release processes (and level of automation)– Validation and verification processes
• There is often a mismatch in process maturity between supplier & client!
There is a real risk that you will simply outsource your internal chaos and
get double the chaos
17Test and Verification Solutions 1729 April 2009
The outsourcing process
• Define the strategic goals• Decide what to outsource• Review your readiness to outsource• Select your preferred outsource model• Select your outsource partner• Agree the
– Master Services Agreement – Service Level Agreement
• Outsource execution– Management– Governance
18Test and Verification Solutions 1829 April 2009
Effective outsourcing - Outsource model selection
• Commercial model– Cost-plus pricing– Fee-for-service pricing (or T&M)– Fixed price– Shared-risk/reward pricing– Business outcome achievement
• Location– Onsite– Near-shore– Off-shore “Multisourcing: the disciplined provisioning
and blending of business and IT services from the optimal set of internal and external
providers in the pursuit of business goals”Linda Cohen and Allie Young, Gartner, INC., “Multisourcing”
19Test and Verification Solutions 1929 April 2009
The outsourcing process
• Define the strategic goals• Decide what to outsource• Review your readiness to outsource• Select your preferred outsource model• Select your outsource partner• Agree the
– Master Services Agreement – Service Level Agreement
• Outsource execution– Management– Governance
20Test and Verification Solutions 2029 April 2009
Effective outsourcing - Outsource partner selection
• Criteria– People (academic profile, training policy)– Retention– Process maturity– Quality Management System– Project and risk management– Corporate and financial profile– Technology and domain knowledge– Cost
• Process– Identify potential partners– Assess them against selection criteria to generate a short-list– Visit short-list to undertake a more detailed analysis– Have potential partners undertake a pilot project
Ensure you consider the costs of testers at various
levels of seniority and experience
21Test and Verification Solutions 2129 April 2009
The outsourcing process
• Define the strategic goals• Decide what to outsource• Review your readiness to outsource• Select your preferred outsource model• Select your outsource partner• Agree the
– Master Services Agreement – Service Level Agreement
• Outsource execution– Management– Governance
22Test and Verification Solutions 2229 April 2009
Effective outsourcing – MSA & SLA
• The MSA is a complex legal agreement
– Confidentiality, ownership of deliverables, assignment of rights, indemnification, warranties, liability, changes in personnel, subcontracting, termination, penalties for not meeting agreed service and delivery targets, etc
• The SLA should cover the service and support goals
– Targets for responsiveness and quality of deliverables– The metrics & remedies should the targets be missed – Fixed-price vs. T&M– Sharing in success– Setting behavioral expectations
I find it useful to separate the MSA and SLA. They have different objectives and the MSA should be
more stable than the SLA
23Test and Verification Solutions 2329 April 2009
The outsourcing process
• Define the strategic goals• Decide what to outsource• Review your readiness to outsource• Select your preferred outsource model• Select your outsource partner• Agree the
– Master Services Agreement – Service Level Agreement
• Outsource execution– Management– Governance
24Test and Verification Solutions 2429 April 2009
Outsourcing Execution: Management and Governance
• Management is about responsibility for specific decisions• Governance covers decision-making processes: who, how, when
• Good governance involves at least the following– Established standard decision-making procedures – Well-understood processes for handling exceptions – Decisions made quickly at the correct level against well-defined criteria– Decisions get recorded
• And good governance is helped by the following– having agreed strategic objectives– formal communication methods which record project status and decisions– controls & records to ensure governance procedures are followed– regular meeting of the stakeholder group
• Good management means– The usual project management practises PLUS
You need to give the outsourcer the “right” level of independence!
25Test and Verification Solutions 2529 April 2009
Good management
• Communication– Who can communicate with who?– How often?– Formal vs. informal communication– How does the communication get recorded?– What happens on miscommunication?
• Knowledge management– Getting up the initial learning curve– On-going knowledge management
• What is the process for asking a question?• What is the process for recording an answer?
Perform regular audits of the knowledge
database!
26Test and Verification Solutions 2629 April 2009
Good management ctd.
• Audit trails – What trails do you need?
• Decision making
• Execution history
• Timesheets
– You need to save this data as quickly and as cheaply as possible
• What about access time?
– How strict does your audit need to be?
• Well-defined incoming QA proceduresYes – QA of the QA!
27Test and Verification Solutions 2729 April 2009
Case Study – ClearSpeedBackground
• ClearSpeed’s product range includes chips, accelerator cards, and software
• All work with industry-standard x86-based systems
• 3 outsourcers• 2 fixed-price• 1 T&M team-
based
28Test and Verification Solutions 2829 April 2009
Case Study – ClearSpeedSome details
• Readiness– Mostly V-model, some iterative– Use of MRD– cfg mgt system supporting regular builds, automated tests, ftp– Software releases followed alpha, beta, GA
• Complex MSA• SLA
– Fixed-price – simple SOW and end-of-project reviews– T&M
• Predicted test execution completion dates• Daily and weekly status reports• Direct access to the defect tracking system• Audit trail of test results, decisions, changes to plans
We had a matrix of tests
& platforms with expected
execution times
We could access the results of any test ever run
29Test and Verification Solutions 2929 April 2009
Case Study – ClearSpeedMain results• Improvement in main KPIs (key performance
indicators)– Quality (Defect Containment Effectiveness)– The time-to-market– Plus reduced costs
• Other benefits– Able to access specialist resource (e.g. GUI testing)– Team size matched required level of testing within
financial constraints– Undertake one-off projects that would not have been
possible otherwise– The use of buffer resources– Use of multi-sourcing to help control price increases
• Problems– The quality on one of the fixed-price projects
deteriorated over time– Costs for lower qualified staff– Active use of the knowledge database– Metrics fatigue!
Improved time-to-market helped byEngineers able to work on rel “X+1” in parallel to release “X” being testedEngineers focused on their core competencyReduced context switches for engineersReduced test cycle time and fewer spins (esp. close to the release)
Improved quality came throughIndependence of the test functionImproved test processesSpecialist test skillsImproved audit trails of tests
Could we have gained the benefits by building an internal team?
30Test and Verification Solutions 3029 April 2009
In conclusion
• Minimise risk of offshoring – Understand your strategic objectives for offshoring
• Cost, schedule, quality, …– Careful partner selection
• Domain expertise; People; Retention; Process; Quality; Project and risk management; Company profile; Technology; Cost
– Process maturity• Internally and at the offshore supplier
– Commercial contract• T&M vs. fixed-price
– Select what to offshore and the development process– Execution and governance
• Expect an increase in your management overhead• It requires specialist management and testing expertise
– Prepare your incoming QA well in advance
31Test and Verification Solutions 3129 April 2009
In conclusionMaximise impact of offshoring on your success factors
– Budget• Offshore SW development resource costs depend on
– Location– Supplier – Experience– Volume
– Schedule• Resource flexibility
– Rapid ramp-up (domain knowledge!) and ramp-down– Buffer resource
• Partners with a large workforce and skills-base (?)– Test techniques and specialism– Vertical markets– Tools
– Quality• Independence of the test function• Use of specialist services in development and testing• Careful partner selection
32Test and Verification Solutions 3229 April 2009
The potential benefits
• Revenue growth without fixed cost growth• Improve the 3 critical success factors
– Budget, schedule, quality
• Independent V&V– Studies show independent V&V can have a significant
impact on quality
• Process improvement(?)• Employee motivation
– Outsource the boring bits!
33Test and Verification Solutions 3329 April 2009
Questions?
Dr Mike Bartley MBA – [email protected]
07796 307958
• Thank you for your time – any questions?