Pres 1117

RCS1

Testing COTS-based Applications

Randall W. Rice, CSQA, CSTERice Consulting Solutions, LLC

Oklahoma City, OKwww.riceconsulting.com

STC 2003, Salt Lake City, UT

© 2003, Rice Consulting Solutions, LLC

RCS2

What is COTS?

• COTS stands for Commercial Off-the-shelf applications.

• Available from commercial vendors• Sometimes called “vendor” software

RCS3

Variations of COTS – COTS-based, GOTS

• COTS-based applications may be delivered by integrating two or more products.

• GOTS - Government Off-the-shelf applications– typically developed by the technical staff of the

government agency for which it is created. – It is sometimes developed by an external entity,

but with funding and specification from the agency.

RCS4

MOTS

• Modified or modifiable off-the-shelf, or military off-the-shelf

• Typically a COTS product whose source code can be modified.

• The product may be customized by the purchaser, by the vendor, or by another party to meet the requirements of the customer.

RCS5

MOTS• In the military context, MOTS refers to an off-the-

shelf product that is developed or customized by a commercial vendor to respond to specific military requirements.

• Because a MOTS product is adapted for a specific purpose, it can be purchased and used immediately.

• Since MOTS software specifications are written by external sources, government agencies are sometimes leery of these products, because they fear that future changes to the product will not be in their control.

RCS6

What is the Purpose of COTS?

• To reduce risks of internal software development

• To reduce the costs of internal software development

• To increase the speed and reliability of delivering applications

RCS7

What are the Challenges of COTS Applications?

• Selecting the right package• Dealing with vendor

licensing issues• Integrating with existing and

future applications• Testing from the external

perspective

RCS8

Key Point #1 – Buying the Right Product is Foundational.

No matter how much customization and testing is performed, if the COTS

product fails to meet your needs at the outset, the project will fail.

RCS9

What are the Challenges of Testing COTS Applications?

• Unknown structure• Unknown functional requirements• Requires an external, black-box

approach• Release schedules• Technology issues• Test tool issues

RCS10

What are the Risks of Implementing COTS Applications?

• Functional problems• Security issues• Compatibility issues• Integration and interoperability issues• Vendor issues• Procurement and licensing issues• Testing issues

RCS11

A COTS Testing Framework

Marketplace Qualified Adapted Assembled Evolving

Verify Needs &ResearchProducts

Verify Needs &ResearchProducts

Evaluate Products &Plan Tests

Evaluate Products &Plan Tests

Verify & Validate Functionality and Interfaces

Verify & Validate Functionality and Interfaces

Verify:•User needs•Technical needs•Mission needs

Research:•Candidate productsand vendors

Evaluate:•Products•Functional fit•Technical fit•Mission fit•Quality levels•Performance levels

Verify:•Integration plans•Test plans

Validate:•Implementation•Integration•Compatibility

Verify:•Test plans

Validate:•Integration•Interoperability•Middleware

Verify:•Test plans•Release and upgrade

plans

Validate:•Integration•Interoperability•Compatibility•Regression

Statement of Req’s.Evaluation CriteriaEval ScorecardsAcceptance Criteria

Graded ScorecardsTest objectivesTest Plans

Verified PlansTest Results

Verified PlansTest Results

Verified PlansRegression CasesTest Results

RCS12

Roles and Responsibilities for COTS Testing

• Vendors – to provide and support a quality product

• Customers – to select the right product and validate

• Senior Management – to guide the acquisition and testing efforts

• Users – to provide quality requirements and to participate in acceptance testing

RCS13

Roles and Responsibilities for COTS Testing

• In-house Developers – to assist in product integration

• Test Team Leadership – to plan tests and lead the testing process

• Testers – to understand tests and perform them correctly

• QA Analysts and Leaders – to gather data from the acquisition, integration and testing processes for process improvement

RCS14

Test Terminology• Verification

– All QC activities throughout the life cycle that ensure interim deliverables meet specific specifications.

• Validation– The “test phase” of the life cycle which

ensures that the end product (e.g..., software or system) meets user needs.

RCS15

Two Other Views of Quality• Producer View

– Did we build the system or application according to specifications?

– Does it work correctly?• Customer View

– Does the product work correctly for me?– Does the product meet my requirements?– Does it work in my environment?– Does it work with other things I need to use?– Is it easy to use?– Is it a good value?

RCS16

It’s possible for a product to meet producer specifications

and miss customer expectations.

RCS17

Key Point #2 – Testing COTS is a Black-box Process.

Requirements and specifications are great targets for testing, but for COTS products, testing becomes

more focused toward validating that the products meet operational or

mission needs.

RCS18

Specify(Verification)

Evaluate(Verification)

Acquire(Validate)

Implement(Verification)

How the COTS Testing Framework Fits Into the Overall COTS Lifecycle

RCS19



Acquire(Validate)


Addresses:User needsMission needsTechnology fit- Integration- Interoperability- CompatibilityConstraints- Environment- User- Project


RCS20



Acquire(Validate)


- Needs- Constraints- Acceptance Criteria


RCS21



Acquire(Validate)


Addresses:Product FeaturesProject RisksProduct RisksProduct Trade-OffsVendor InformationTest Planning



RCS22



Acquire(Validate)



- COTS Product- Test Strategy- Test Plan- Acceptance Criteria


RCS23



Acquire(Validate)


Addresses:Product SelectionVendor ContractingIntegrationTesting (Functional)Testing (Behavioral)




RCS24



Acquire(Validate)




- Tested COTS Product- Baseline- Test Evaluation


RCS25



Acquire(Validate)


Addresses:DeploymentTrainingSupportPost-Implementation

Reviews

- Tested COTS Product- Baseline- Test Evaluation




- Post-ImplementationEvaluation

- Functional Baseline

RCS26



Acquire(Validate)




- Tested COTS Product- Functional Baseline- Test Evaluation

- Post-ImplementationEvaluation

- Functional Baseline


RCS27

• The framework is:– Cyclical and repeatable

• New releases will have new or removed features• Test plans will change

– Evolving• Technical and user requirements will evolve.

– Integrated• Products must work with other products.• Test plans and environments must include multiple

applications and interfaces.– Dependent on multiple sources of involvement for

success• COTS testing is not just one team’s responsibility.

The Evolutionary Nature of the COTS Lifecycle and How it Impacts Testing

RCS28

Key Point #3 – Integration and Interoperability Testing Validates the Glue

that Holds Related Products Together.

• The vendor can’t test integration and operability for you.

• These kinds of tests are the customer’s responsibility.

• Failure to test integration and interoperability is to place the project at great risk of failure.

RCS29

Summary

Testing COTS-based applications is a

challenge, but with careful planning and

effective testing strategies tailored for

COTS, the project risks can be reduced.

RCS30

Bio - Randall W. Rice• Over 25 years experience in building and

testing information systems in a variety of industries and technical environments

• Certified Quality Analyst• Certified Software Test Engineer• Conference chair 1995 - 2002, QAI’s

annual software testing conference• Co-author with William E.Perry, Surviving

the Top Ten Challenges of Software Testing and forthcoming book, Testing Dirty Systems (2003).

RCS31

Contact Information

Randall W. Rice, CSQA, CSTERice Consulting Solutions, LLCP.O. Box 891284Oklahoma City, OK 73189Ph: 405-793-7449Fax: 405-793-7454Web site: www.riceconsulting.come-mail: [email protected]

Pres 1117

Documents

Transcript of Pres 1117