Automating Software Module Testing for FAA Certification
description
Transcript of Automating Software Module Testing for FAA Certification
Automating Software Module Automating Software Module Testing for FAA CertificationTesting for FAA Certification
Usha Santhanam
The Boeing Company
Automating Software Module Automating Software Module Testing for FAA CertificationTesting for FAA Certification
Challenges in code based software testing FAA certification testingTest Set Editor (TSE) tool set and processRegression testingBenefits of TSELimitations of TSESummary
Challenges in Code Based Challenges in Code Based Software TestingSoftware Testing
Software complexity and accelerated development schedules make avoiding defects difficult
Untested code in user environment Difficulty of accurately capturing and
analyzing the sequence of events that occur during program execution
FAA Certification TestingFAA Certification Testing
Structural coverage is often not achieved through requirements based testing alone
Testing each module in isolation provides the desired coverage
The three levels of structural coverage to deal with are statement coverage, decision coverage, and modified condition and decision coverage (MCDC)
Structural CoverageStructural Coverage
Statement coverage: execute each source level statement at least once
Decision coverage: take each execution branch at least once
MCDC: designed to demonstrate the independence of individual conditions that constitute a complex decision
TSE Tool Set and ProcessTSE Tool Set and Process
TSE is designed to achieve shorter turnaround time and cut cost during regression testing
TSE is domain-independent and removes the language-dependent test driver chores
TSE requires the test engineer to construct the test cases, but the rest of the testing process is automated
TSE Tool Set and ProcessTSE Tool Set and Process(cont.)(cont.)
Tasks of testing a module for structural coverage are– Identifying inputs and outputs of a module (or
collection of modules)– Selecting test cases and tagging them with the
requirements they address– Constructing a test driver and stubs for
subprograms called by the module under test
TSE Tool Set and Process TSE Tool Set and Process (cont.)(cont.)
– Compiling and running the tests– Analyzing coverage
TSE assists the tester with all these tasks except the selection of the test cases.
Example: Procedure Get_End Example: Procedure Get_End in Package Ioin Package Io
package Io is Debugging : Boolean := False; procedure Get_End;end Io;
with Text_Io;package body Io is K : constant := 1024; Max_Line_Length : constant :=
16*K; Line : String (1 .. Max_Line_Length);
procedure Error (Msg : String) is begin null; end Error;
procedure Get_Line is
begin null; end Get_Line;
procedure Get_End is begin if Line (1 .. 4) = ".end" then Get_Line; else Error ("Missing .END, assume
here."); end if; end Get_End;end Io;
TSE_Tools.tclTSE_Tools.tcl
Regression TestingRegression Testing
All test cases are self-checking A batch file is used to run the test files
through the TSE testing process A batch log file is used to check for
failures Failed tests are examined and justified or
updated
Benefits of TSEBenefits of TSE
Tests can be developed by engineers who are not necessarily expert programmers in Ada
TSE includes a tool to analyze the source code and identify inputs and outputs of a given module
Provides coverage information
Benefits of TSEBenefits of TSE(cont.)(cont.)
TSE tests are easy to maintain due to uniformity of test definition files
TSE tests are repeatable with no manual interaction necessary
TSE tests can be executed on host or target platform using different support packages
Productivity ImprovementsProductivity Improvements
Based on our experience we have seen a range of 1:5 to 1:10 productivity advantage.– Depends on the complexity of the module
Estimate of TSE Based Estimate of TSE Based TestingTesting
We have developed a tool that estimates the number of hours needed to test a module with TSE
Based on Statement count, number of If’s, Case’s, Loop’s, Exit’s, Logical operators, short circuit control forms, and stub calls
The Get_End procedure estimated testing time is 7 hours
Estimate of Manual TestingEstimate of Manual Testing
Estimated size of the get_end test driver is 90 SLOC– Including main driver and stubs
Programmer productivity for the test code development is 2 SLOC per hour (Hihm)
This yields the productivity improvement of 1:6.5 for get_end procedure
Limitations of TSELimitations of TSE
It is not possible to construct one generic module test to verify all instantiations– Handled on an instantiation by instantiation
basis
Tasking is difficult to test because tasks run in independent parallel threads– Each subprogram can be tested independently
regardless of the number of tasks sharing it
Limitations of TSELimitations of TSE(cont.)(cont.)
For modules using private type parameters or returning private type, construction of external test driver is awkward
– Embed the test driver in the body of the package that defines the limited private type
Nested subprogram testing not available– Requires the test driver to be nested – Must be tested in the context of enclosing subprogram
Summary of TSE BenefitsSummary of TSE Benefits
Minimizes the need for programming knowledge
Cuts testing effort several fold over manual approach
Minimizes cost of retesting Allows us to meet FAA/DO-178B
requirements in cost effective manner