Presented To:

April 21, 2023

CONFIDENTIAL Copyright 2007, TopCoder, Inc.

Sean Campion [scamp]

27 June 2007 TopCoder Open, Las Vegas

TopCoder Component-Based Software Engineering (CBSE)

Quality Assurance (QA)

• CBSE Quality Assurance Specific Challenges

• scamp@home

• TopCoder Process from QA perspective

• Wrap-up

[ ] Agenda

CONFIDENTIAL Copyright 2007, TopCoder, Inc.

[ ] CBSE Quality Assurance Specific Challenges

CONFIDENTIAL Copyright 2007, TopCoder, Inc.

The following are common QA problems cited against the use of reusable components. I will discuss each issue, and what TopCoder does to address the issue.

[ ] CBSE Quality Assurance Specific Challenges

CONFIDENTIAL Copyright 2007, TopCoder, Inc.

Issue: Lack of access to commercial component source code.

TopCoder: We provide both the source code and all design documentation for every component that we produce.

[ ] CBSE Quality Assurance Specific Challenges

CONFIDENTIAL Copyright 2007, TopCoder, Inc.

Issue: Components are not readily testable.

TopCoder: All of our components are required to have unit tests of stress, failure, and accuracy.

[ ] CBSE Quality Assurance Specific Challenges

CONFIDENTIAL Copyright 2007, TopCoder, Inc.

Issue: Difficult to use and interpret tests from component vendors

TopCoder: Our documentation provides examples on how to run the tests; we also provide the test results in the form of logs.

[ ] CBSE Quality Assurance Specific Challenges

CONFIDENTIAL Copyright 2007, TopCoder, Inc.

Issue: Concern that components were not developed for a specific operating environment; and that developing a component test bed for all possible environments is too costly and time consuming.

TopCoder: We are developing scamp@home, a software system that will take advantage of participants diverse environments to test components.

[ ] scamp@home

CONFIDENTIAL Copyright 2007, TopCoder, Inc.

scamp@home- Distributes component tests to participating

computers.- Takes advantage of diverse environments,

hardware, character sets, and so on.- Massively parallel- Can be setup at customer sites to start validating

existing components during architecture phase on actual target environment.

- Customers can use to test their own components.- Ideal for regression-testing when component is

updated.

[ ] scamp@home

CONFIDENTIAL Copyright 2007, TopCoder, Inc.

Data WarehouseMetric Reporting and User DB

-End1

*

-End2

*

-End1

*

-End2

*

SCAMP@Home -- Macintosh (several versions : SCAMP@home

SCAMP@Home -- Windows(several versions : SCAMP@home

SCAMP@Home -- Linux(several versions) : SCAMP@home

SCAMP@Home -- PALM Mobile(several versions) : SCAMP@home

SCAMP@Home -- WindowsMobile(several versions) : SCAMP@home

SCAMP@Home -- PALM Mobile(several versions) : SCAMP@home

Application Server

Metric Capturer

Environment Tracker

Component DistributorSCAMP@home

CPUIdleIdentification

Component Downloader

Metric Upload

Metric Capturer

Environment Capturer

Cell Phone Synchronization

-End3

* -End4 *

<<ETL>>

<<JDBC>>

<<HTTPS>>

Test Case Execution User Registration

User Security and Administration

[ ] TopCoder Process from a QA Perspective

CONFIDENTIAL Copyright 2007, TopCoder, Inc.

Specification Phase- Mid and end-point peer reviews of the

application specification, conducted by 2 Project Managers and 1 Architect.

- High-level test cases, tied to application level functionality, developed.

- Customer review and sign-off on application specification.

[ ] TopCoder Process from a QA Perspective

CONFIDENTIAL Copyright 2007, TopCoder, Inc.

Architecture Phase- Mid and end-point peer reviews of the

architecture design, conducted by 1 Project Manager and 2 Architects.

- High-level test cases expanded into detailed test cases and scenarios.

- Customer review and sign-off on architecture design.

- Future: Existing components identified for reuse can be validated using scamp@home, including on the customer target systems if desired.

[ ] TopCoder Process from a QA Perspective

CONFIDENTIAL Copyright 2007, TopCoder, Inc.

Component Production Phase - Design- Design competitions; designers know they are

competing against each other.- Automated submission screening tool.- Manual screening phase.- Review Board.- Scorecard-based evaluation.- Final fixes.- Winning designer required to support the

development phase.

[ ] TopCoder Process from a QA Perspective

CONFIDENTIAL Copyright 2007, TopCoder, Inc.

Component Production Phase - Development- Development competitions; developers know

they are competing against each other.- Automated submission screening tool.- Manual screening phase.- Review Board.- Scorecard-based evaluation.- Final fixes.- Unit tests: stress, failure, accuracy- Coverage Tool: minimum of 85% line coverage

from unit tests.

[ ] TopCoder Process from a QA Perspective

CONFIDENTIAL Copyright 2007, TopCoder, Inc.

Assembly Phase- Assembly competitions; teams know they are

competing against each other.- Manual screening phase.- Review Board.- Scorecard-based evaluation.- Final fixes.

[ ] TopCoder Process from a QA Perspective

CONFIDENTIAL Copyright 2007, TopCoder, Inc.

Assembly Phase - Testing- Testing competitions; teams know they are

competing against each other.- Produces automated tests that will be run

against the assembly competition submissions.

[ ] TopCoder Process from a QA Perspective

CONFIDENTIAL Copyright 2007, TopCoder, Inc.

Certification Phase- Replicate the customer’s target environment on

a TC server.- Deploy and rigorously test the winning assembly

submission.- Run automated tests against the submission.- Run manual tests against the submission.- Early customer access to the TC testing

environment for immediate feedback.

[ ] TopCoder Process from a QA Perspective

CONFIDENTIAL Copyright 2007, TopCoder, Inc.

User Acceptance Testing Phase- Customer has option of performing their own

User Acceptance Testing.- TopCoder will directly support any UAT.

[ ] TopCoder Process from a QA Perspective

CONFIDENTIAL Copyright 2007, TopCoder, Inc.

Deployment Phase - TopCoder will support the deployed application

for 90 days.- Customer receives all source code, including

component source code, and all design documentation from all phases.

[ ] Metrics

CONFIDENTIAL Copyright 2007, TopCoder, Inc.

Here’s how TopCoder compares to industry standards:

Industry Defects per KLOC: 3.43

TopCoder Defects per KLOC: 2.10

[ ] Wrap-Up

CONFIDENTIAL Copyright 2007, TopCoder, Inc.

- Quality Assurance addressed at every stage of the process in one form or another.

- TopCoder continuously looking to improve QA through processes and tools.

- scamp@home addresses a primary concern of reusable components.

- Quality software distinguishes TopCoder from other software.

[ ] References

CONFIDENTIAL Copyright 2007, TopCoder, Inc.

- “Testing and Quality Assurance for Component-Based Software”, Jerry Zeyu Gao, H.-S. Jacob Tsao, & Ye Wu; Artech House, Boston, 2003.

- “Risk Management in Component-Based Development: A Separation of Concerns Perspective”, Awais Rashid, Gerald Kotonya, Computing Department, Lancaster University, UK.

- “The Trouble With Testing Components”, Elaine J. Weyuker, in “Component Based Software Engineering”, Chapter 28, Addison-Wesley, 2001.

- “CBS Testing Requirements and Test Case Process Documentation Revisited”, Nasib S. Gill, Pradeep Tomar, Department of Computer Science and Applications, Haryana India, March 2007. From ACM SIGSOFT Software Engineering Notes, Volume 32 Number 2.

- “The QSM Software Alamanac: Application Development Series”, QSM Inc., 2006.