A white paper analysis from Orasi Software

Software Testing Center of Excellence Why and How Organizations are Bringing CoEs to Software Quality Management

2

A white paper evaluation from Orasi SoftwareSoftware Testing Center of Excellence

IntroductionTo bring innovative solutions and services to market more quickly, more efficiently, and more cost-effectively, IT-oriented organizations must ensure the performance and availability of key software applications.

Unfortunately, many companies struggle with testing, quality control, and the broader requirements of quality assurance.

Shorter lead times are accelerating the pace of application test projects. Non-standard operating procedures and tools can reduce an organization’s ability to detect and correct software defects. More importantly, less-than-optimum testing can reduce an organization’s ability to deliver quality software products to the marketplace.

For many application-driven organizations, the Testing Center of Excellence (TCoE) approach provides a logical and affordable solution.

It is true: companies can deploy a Testing CoE to improve overall immediate product quality, while reducing the costs associated with reworking software to correct critical defects. But a true TCoE goes an important step further: allowing organizations to get at the root cause of application faults, and to realize long-term benefits by addressing and correcting those defects earlier in the development cycle.

In this Orasi white paper analysis, we examine the trends behind the move to the Testing Center of Excellence model, and how this approach affects people, processes, and technologies in an application-oriented environment. The paper explores how a TCoE can help advance testing activities toward more mature, cost-effective, and best-of-class capabilities.

The authors then outline how to plan, build, and operate a successful TCoE, and discuss the benefits of working with experienced testing and automation partners.

Why Organizations Invest in a Testing CoEQuality is a concern for any organization that builds new applications or changes existing software. By ensuring application quality, organizations can reduce the risk of operational disruptions, accelerate time-to-market, and improve performance and product quality.

Quality is a given need: but how can organizations best pursue the goal of high-quality applications? For many enterprises, a Testing Center of Excellence can be the solution.

A TCoE allows organizations to enhance application quality and performance, while reducing costs, and to do so in a manner and pace that makes sense for their enterprise.

Organizations can use the TCoE approach to accelerate the development and release of applications into the marketplace. In addition, this approach can help reduce the number and severity of defects, and to support the creation of functionality that is better aligned with the needs of the business and the expectations of end users and customers.

The TCoE approach enables organizations to leverage best-of-class tools and techniques, and to provide single-source testing services across projects and lines of business.

Here are specific ways in which a TCoE can help meet those objectives:

• Quality assurance. A TCoE helps ensure application quality and performance meet development expectations, for in-house and outsourced applications.

• Integration. Staff members, best practices, and development and delivery resources are shared by all project teams, thus eliminating duplication and overall costs.

• Synergy. By collecting test processes, resources, and artifacts from across the enterprise, then standardizing and optimizing those assets, testing can be enhanced across all projects. These synergies spread knowledge, shorten learning curves, and build testing success.

• Alignment. A TCoE approach strengthens governance and allows IT priorities to be more closely linked to strategic business goals.

• Scalability. TCoE’s can begin on a small scale and using existing resources, then expand as it achieves positive results in the organizations. Many TCoE efforts are in fact self-funding.

The Center of Excellence models have proven to be workable and beneficial in a wide range of environments, including application testing. Before proceeding too far down the road towards a Testing CoE, however, organizations should first evaluate where they currently fit in the overall Software Quality Management environment.

Figure 1 offers a helpful “Testing Radar” grid on which organizations can map their current positioning – for test planning & design, requirements analysis, various types of testing, reporting, shared services, and other variables – on a zero-to-five maturity scale. The outer level five represents a more active, mature level of activity.

While higher maturity levels are obviously beneficial, it should be noted that not every organization can, or should, reach a maximum Level Five of testing maturity for every variable. One company, for example, may be fine with unit and performance testing at

Requirements

Reporting

DefectManagement

SharedServices& Work

Requests

TestEnvironments

Test Planning & Design

Key PerformanceIndicators

Unit Testing

IntegrationTesting

SystemTesting

UserAcceptance

Testing

Performance Testing 5

4

3

2

1

0

Optimization

Transformation

Initiation

Goal: Optimizestrategic assets

Goal: Quantitatively manage strategicassets to deliver business value

Minimize Organizational Impact

Achieve Operation Excellence

Goal: De�ne CoC framework and implement all the strategicassets, knowledge management

Gain C

ontrol a

nd Cre

dibili

ty

Figure 1 – Testing Radar grid

3

A white paper evaluation from Orasi SoftwareSoftware Testing Center of Excellence

Level Three while needing Level Five for integration testing. Other software development environments will require entirely different mixes of maturity.

Some critical aspects of a TCoE, such as automation, may be needed if an organization hopes to advance a broader effort towards a more mature SQM model.

As recommended elsewhere in this paper, executives should first understand their broad business and quality needs, as well as available resources and skill sets, and then formulate appropriate maturity levels for each of these testing-related variables. In many cases, organizations can leverage external expertise and assets to “stretch” their testing maturity without adding permanent costs or infrastructure.

Figure 2 depicts the classic Maturity Model for software quality management, which consists of a three-step progression through initiation, transformation, and optimization. Organizations seek to

achieve the end-state benefits of greater control and operational excellence, while minimizing cost, disruptions, and other organizational impacts

The Testing Center of Excellence ApproachWhen many people hear the term Testing Center of Excellence, they understandably associate the term with the specific requirements of quality control and test management. But a TCoE should more correctly be seen as an integral part of the broader software quality management environment.

Certainly, a TCoE is often employed as part of a test management program, typically supporting functional and non-functional testing services in an overall quality control program. Along with code reviews, defect management, and compliance reporting, a TCoE can also play an important role in the process improvement element of a broader quality assurance effort.

Figure 3 illustrates where a Testing Center of Excellence may fit within the scope of an enterprise-class SQM model. As an organization advances to a more mature state for most application quality activities, a Testing Center of Excellence can contribute to meeting the higher-level objectives of Software Quality Management.

In the quality control context, a TCoE focuses on specific production-oriented defects and corrections. In the larger quality assurance environment, a TCoE can help organizations understand and address higher-level issues related to process, root-causes, skills, toolsets, and quality-related governance.

By understanding how a TCoE can and should fit into their overall quality ecology, organizations can realize optimum benefit from this approach.

Testing CoE FrameworkOrasi recommends a holistic approach to the creation and operation of an enterprise-class Testing Center of Excellence. A successful TCoE should engage not just the quality assurance group, but should seek the support and involvement of other key stakeholders.

Requirements

Reporting

DefectManagement

SharedServices& Work

Requests

TestEnvironments

Test Planning & Design

Key PerformanceIndicators

Unit Testing

IntegrationTesting

SystemTesting

UserAcceptance

Testing

Performance Testing 5

4

3

2

1

0

Optimization

Transformation

Initiation

Goal: Optimizestrategic assets

Goal: Quantitatively manage strategicassets to deliver business value

Minimize Organizational Impact

Achieve Operation Excellence

Goal: De�ne CoC framework and implement all the strategicassets, knowledge management

Gain C

ontrol a

nd Cre

dibili

ty

Figure 2 – Software Quality Management maturity model

Figure 3 – The Testing CoE in the broader SQM environment

- Code reviews

- Document peer reviews

- Defect tracking & reporting

- Non-compliance reporting

- Requirements traceability tracking

- Test automation

- Testing Center of Competency (CoC)

- Testing Center of Competency (CoC)

- Test management, planning, CM and design

- Test environment development

- Test case / script development

- Test execution and reporting

- Defect, risk and traceability tracking

- Ongoing support

Evaluate for compliance to:

- Policies

- Processes

- Procedures

- Templates

- Guidelines

- Tools

Functional Testing/Validation- Smoke testing / sanity testing- Integration testing- System testing- Regression testing- User acceptance testing

Non-Functional Testing/Validation- Load and performance testing- Stress & volume testing- Compatibility & migration testing- Data conversion testing- Operational readiness testing

Software Quality Management (SQM)

Quality Assurance

Process Improvement Test Management Governance Testing Services

Quality Control

4

A white paper evaluation from Orasi SoftwareSoftware Testing Center of Excellence

As illustrated in Figure 4, a workable TCoE framework consists of four essential elements: Vision and Planning, Governance, Strategic Assets, and Strategic Enablers.

Vision & PlanningPlanning establishes the organization’s vision and expectations for the TCoE, and maps out the general pathway and specific benchmarks to be measured as the program moves toward testing maturity. This stage requires experienced leadership and a true end-to-end understanding of modern testing systems, the scope and use of a TCoE project, and the entire software development lifecycle.

That understanding begins with a solid grasp of the organization’s requirements, and the ability to match skills and resources to those needs. After specific testing requirements are identified and analyzed, test cases and automated processes are selected or developed to meet those needs. When those test cases are executed, the resulting reports will document any issues with the automated process, and potential defects with the software application itself.

A broad Organizational Design effort should examine the people, processes, and technologies needed to reach the desired end-state environment. A rigorous testing assessment may be used to evaluate current personnel and technical capabilities. The assessment should focus on manual test cases, with the objective of identifying opportunities to improve testing throughput and repeatability through the application of automated processes.

A TCoE roadmap should describe the personnel, process, and toolsets needed across various maturity timeframes – such as the critical start-up phase, perhaps a six-to-12 month window, and during full operational maturity. Planning and mapping should produce detailed metrics on projected TCoE costs, Key Performance Indicators (KPIs), and return on investment (ROI) expectations.

GovernanceWorld-class Software Quality Management requires a continuous, enterprise-wide commitment to quality assurance, process improvement, and governance. As shown in Figure 4, strong governance constantly evaluates organizational compliance with established policies, procedures, and guidelines, and the specific application of approved processes, templates, and tools.

At the Testing CoE level, good governance leverages best practices, KPIs, and other benchmarks to establish and measure testing effectiveness. Orasi recommends the initial use of a relatively small set of KPIs, a measured approach that allows the TCoE to understand and build on good analytic practices. These metrics should flow to a robust reporting system capable of providing both executive dashboard views and drill-down details on all aspects of the application testing program.

Proven workflows can also be incorporated into the entire process, helping to ensure that when requirements are complete they trigger the established analysis, test case, and execution cycle.

Finally, strong program management, with an emphasis on continual improvement, helps keep the “E” in TCoE over the long term.

Strategic AssetsWhat does it take to build and operate a world-class testing organization? As with many technical efforts, a TCoE requires skilled personnel, solid processes, and the right tools. It may help to examine each in detail.

People. Personnel, training, and knowledge are crucial elements in any successful testing program. A TCoE should identify, preserve and strengthen the skills and expertise needed to deliver testing services, a key benefit when IT personnel are replaced or retire.

For IT organizations that do not have – and do not want to build – large internal testing capabilities, external expertise may be required for the development of testing-oriented applications, test automation, and other technical or specialized requirements. To ensure the long-term success of a TCoE, organizations should promote knowledge sharing between employees and across their organizations.

Figure 4 – Testing Center of Excellence Framework

Vision

Organizational Design Roadmap ROI Metrics

Strategic Assets

Governance

Strategic Enablers

People Process Technology

Better Practices

IT Business Alignment

Workflow

Change Management

Program Management

IP & Thought Leadership

Dashboard & Reporting

Sponsorship

5

A white paper evaluation from Orasi SoftwareSoftware Testing Center of Excellence

Process. Robust processes should be well-established for the vast majority of enterprise-level testing requirements. By harnessing available thought leadership and best practices, organizations can reduce the time and money needed to launch and operate a successful TCoE. In some cases, internal personnel can be trained to apply new processes, while some organizations may prefer to acquire test automation or other specialized processes from external specialists. In any case, processes should be adapted to meet the specific testing, project framework, and cultural requirements of each organization.

Technology. Any true Center of Excellence must be built upon a solid foundation of standardized, world-class tools and systems. By deploying automation platforms and state-of-the-art infrastructure, organizations can eliminate many of the consistency, compatibility, and efficiency issues that plague previous-generation testing programs. Standardized, state-of-the-art technologies enable the TCoE to deliver testing services that are consistent and repeatable, cost-effective, and help accelerate time-to-market.

Strategic EnablersThe final strategic enablers for a Testing Center of Excellence are both important, and too-often overlooked.

Quality control and assurance are ideally long-term commitments and the technical requirements of testing often span business lines and organizational boundaries. To realize the full benefits of the TCoE approach, organizations must also address the need for broad stakeholder participation and executive sponsorship.

Adjustments in staffing, skills, and processes, and the introduction of automation and other technologies can be disruptive. A well-formulated change management program can reduce uncertainty and waste during the transition to a more mature TCoE model.

A testing program must be configured to support and protect the organization’s Intellectual Property holdings, and should of course be fully aligned with the company’s broader business objectives.

In smaller or growing companies, the CIO or some other executive may provide the strategic support for the effort, while other organizations will require broader and more complex participation. In any case, it is important that top leadership understands and communicates the need for world-class testing.

Given the technical complexities of modern application development, advanced software testing can pose significant challenges to organizations of all sizes and types. For this reason, many forward-looking companies partner with specialists who are experienced in the formulation and operation of a modern Testing Center of Excellence.

Advantages of the TCoE ApproachThere are a number of powerful reasons why organizations pursue the Center of Excellence model for application testing.

At the highest level, the TCoE approach allows companies to promote enterprise integration as a formal discipline. By doing so, they can naturally combine skills, processes, and resources into a single group, thereby reducing duplication across the enterprise, and optimizing the use and value of those quality-related assets.

The inherent efficiencies of the TCoE model allow application developers to accelerate project delivery times. Instead of

re-formulating test procedures on every project, TCoE teams can build on past success, pursue continual improvement, and emphasize sustainable testing methods. A TCoE provides the flexibility to quickly ramp-up or ramp-down a testing project.

A TCoE can focus on the development of staff specialists and internal expertise, and encourages the development of subject matter experts and domain-specific skills. Testing processes, artifacts, and techniques can be gathered throughout the organization, refined and standardized, and made available to the entire enterprise. Testing products, best practices, and staff are accessible to all project teams through a convenient and centralized source.

A TCoE can be particularly valuable in today’s dynamic business environment. With IT departments in a state of flux due to corporate mergers or other changes, a TCoE can promote robust documentation and knowledge-sharing to ensure that technical skills and testing expertise are retained by the organization. For organizations that outsource or offshore, a TCoE helps ensure that both in-house and outsourced apps meet quality and performance requirements.

A TCoE provides positive and measurable economic benefits. A TCoE drives greater visibility and demarcation into the true cost of testing services, allowing executives to make more informed quality-related decisions. In addition, a TCoE promotes economies of scale by applying testing and QA/QC resources across multiple projects – thus optimizing investment costs at the enterprise level and lowering per-test costs.

A TCoE improves Return on Testing Investment (ROI). A TCoE can drive economic payback, through the creation and reuse of organizational assets such as best-of-class testing tools, standardized processes, automated frameworks, reusable test cases, and quality-related analytics and reporting systems.

Finally, the TCoE approach can help elevate “quality” as part of an organization’s culture. A TCoE can help keep all stakeholders informed on quality-related topics, and ensure that IT applications are better aligned with overall business objectives. A successful and well-positioned TCoE demonstrates to internal staff, partners, and customers an organization’s commitment to software development and quality assurance.

TCoE Next StepsAs noted earlier, to ensure the success of a TCoE, it is best to get strong support from senior leadership and a committed practice leader, and do that early in the process. A TCoE should be staffed with personnel who are proactive, accountable, and strongly committed to understanding and meeting customer requirements. A good TCoE will be accountable and focused on continuous improvement.

Organizations should consider exactly what types of testing will be handled by the TCoE. The Center of Excellence model is ideally suited to handle requirements management, test planning and management, and functional, regression, performance, and security testing. As described here, the TCoE approach can deliver end-to-end capabilities, from requirements collection and analysis to test cases, automation, execution, and the reporting and evaluation of testing results.

6

A white paper evaluation from Orasi SoftwareSoftware Testing Center of Excellence

Learn More: www.orasi.com | sales@orasi.com | 678-819-5300

© 2014 Orasi Software, Inc. The information contained herein is subject to change without notice. Orasi is a trademark of Orasi Software, Inc. All other product and company names are trademarks or registered trademarks of their respective owners.

Creating a Testing Center of Excellence is a viable objective for small, medium-sized and large organizations: many organizations can start on a small scale by leveraging existing resources, then expand the TCoE as it proves its value.

Thanks to the benefits noted in this paper, many organizations find their TCoE can be self-funded. In fact, because many companies already have budgets for application testing, those funds can often be applied towards a TCoE-oriented testing project.

The findings of an initial testing effort can help identify and manage defects at the granular level, begin to trace faults back to a point of origin, and help organizations identity and address both specific issues and the broader quality-related environment.

This small-to-large approach – which Orasi calls “Quality Control to Quality Assurance” – allows organizations to meet specific testing project deadlines without additional overhead, while also implementing more comprehensive needs analysis, process reviews, automation, and analytics. Those outcomes represent the fundamental elements of a true Testing Center of Excellence, and contribute directly to enterprise-class Quality Assurance efforts.

ConclusionQuality Assurance, quality control, and application testing are crucially important for any organization that depends on the functionality and reliability of software systems. Yet because per-project efforts to improve quality typically have only short-term impacts, companies of all sizes and types have sought a broader and more efficient approach to application quality.

The Center of Excellence approach can benefit any organization that must test, manage, and assure the quality of software applications. The CoE approach has been used successfully by companies the

world over, for software testing and to better manage integration, service-oriented architectures, business intelligence, and other technology requirements.

By understanding the maturity of their current testing capabilities, and their strategic Quality Management/Assurance objectives, business leaders can plan, establish, and operate a successful Testing CoE. A successful TCoE effort must address the requirements of personnel, processes, and technologies. To manage the complexity of advanced application testing, many organizations now work with proven and experienced software testing partners.

Organizations can leverage the TCoE model to improve the efficiency of testing projects, to enhance software performance and reliability, and to drive application-related quality across their value chains.

About OrasiAn HP Software Platinum Partner and authorized Support partner, Orasi resells HP’s test management and automation solutions, is a leading provider of software testing services, and offers mobile testing, security, and cloud-based testing and monitoring solutions.

For over 11 years, Orasi has consistently helped customers successfully implement and integrate software testing environments to reduce the cost and risk of software failures. Orasi was the HP Software US Solution Partner of the Year in 2011 and 2013, and was Support Partner of the Year for 2009, 2011 and 2012. Orasi offers proven solutions for the installation, implementation, configuration, and use of Testing Centers of Excellence.