SAS03/FaultPrediction 1

NASA OSMA SAS ‘03

Software Requirements Analysis As Fault Predictor

Dolores R. Wallace

SRS TechnologiesSoftware Assurance Technology Center

http://satc.gsfc.nasa.govdwallac@pop300.nasa.gov

William M. WilsonSRS Technologies/SATC

wilsonw@pop300.nasa.gov

SAS03/FaultPrediction 2

Software Requirements Analysis As Fault Predictor

Presentation Outline1. Research Rationale & Objective

2. Requested Project Data

3. Planned Approach

4. Data Acquisition & Analysis Obstacles

5. Results & Findings

6. Conclusions

7. Recommendations

SAS03/FaultPrediction 3

Research Rationale & Objective

Rationale• The earlier in the lifecycle that software faults can be

found the less expensive it is to correct them.• The earliest opportunity to preclude software faults is

during the development of the system's requirements.

Objective• To determine if software requirements analysis results can

be used as a predictor of software faults.

SAS03/FaultPrediction 4

Requested Project Data

• Requirements Specification– Accessible in electronic media in MS Word format

for use with the ARM tool

– Conforming to NASA-STD-2100

– Specification statements identified by hierarchical numbers

SAS03/FaultPrediction 5

Requested Project Data – Cont.

• Verification Test Matrix– Requirements to Test or vice versa

- OR -• Traceability Matrix

– Requirements to Design Elements– Design Element To Software Component

- AND -

• Test Plans– Tests vs. Software Components.

SAS03/FaultPrediction 6

Requested Project Data – Cont.

• Failure Reports/Data - Preferably in DDTS format– Including: test id, module id, CSCI id, release/build id,

suspected cause of failure and resolution (e.g., actual cause of failure - what was fixed)

• Configuration Control Reports– Baseline item changed -e.g., Requirement, Design,

Code– Reason/Source of change – e.g., RID, Failure Report,

PR, etc.

SAS03/FaultPrediction 7

Planned Approach

1. Use ARM tool to analyze projects’ software requirements.

2. Collected projects’ failure data in a DDTS database.

3. Organize the data to represent same components for each project.

4. Conduct correlation analysis.

5. Study results of step 4 to prove or disprove hypothesis.

SAS03/FaultPrediction 8

ARM TOOL Reports

Identifies and Counts:• Size of Requirements Document

– Lines of Text

– Numbered Paragraphs

– Number of Specification Statements

– Number of Unique Specification Subjects

• Number of Specifications Containing– Weak, Optional and/or Incomplete Phrases.

– Compound and/or Complex Statements.

• Requirements Document Structure– Depth and Distribution of Specifications

SAS03/FaultPrediction 9

Data Acquisition & Analysis Obstacles

• Common Problems– No commonality or standardization of data or formats

across project.

– Current projects are reluctant to provide data to outside organizations.

– Data from completed projects is incomplete, not in a usable form or is not accessible.

SAS03/FaultPrediction 10

Data Acquisition & Analysis Obstacles Cont.

• Specific Problems– No project staff available to guide through the mass of

documents in the collection

– Documents locked on a WEB site

– Documents in “.pdf” format

– Variation of format and media within project

– Complete set of data for a specific Build/Release not available

– Data held by support contractor and released only on a “Project Need-To-Know” basis

SAS03/FaultPrediction 11

Results & Findings

• Data provided by four projects– Projects A and B

• Major subsystems of operational information processing system

– Projects C and D• Flight instrumentation packages

SAS03/FaultPrediction 12

Results & Findings Cont

Summary Of Project's Data

PROJECT Tot

al S

peci

fica

tion

s

Wea

k Sp

ecif

icat

ions

Inco

mpl

ete

Com

poun

d

Max

Dep

th

Spec

s at

Low

est

Lev

el

Tot

al P

robl

ems/

Cha

nge

Req

uest

s

Req

uire

men

ts

Des

ign

Cod

e

Tes

t

Doc

umen

tati

on

Enh

ance

men

t

Oth

er

Non

e

A 849 256 10 241 5 226 3066* 0 183 2013 9 22 0 267 572B 226 47 13 39 5 38 1132** C 356 75 0 100 6 117 323*** 12 34 91 3 5 69 102 7D # 121##

* System Failure Reports** Details Not Yet Available*** Software Change Requests# System Detailed Design Spec not Requirement## Unformatted Text Problem reports

SAS03/FaultPrediction 13

Conclusions

• Data from many more projects is required to improve the possibility of finding a number of data sets with sufficient commonality to support analysis.

• Analysis approach needs to be expanded to include determining if failures attributed to documentation and design flaws are in reality related to deficiencies in requirements.

SAS03/FaultPrediction 14

Recommendations

Advocate and support:

• The use of NASA-STD-2100 documentation standard

• The use of a format to report problems and failures that is compatible with DDTS

• The creation and use of centralized Center repositories for data from completed projects.

SAS03/FaultPrediction 15

Summary

• Lessons– Getting data requires strong, interested Civil

servant advocate– Analyzing data requires tedious effort, even

with automated tools

• Future – Analyze latest set of data– Attempt correlations for the 4 data sets