Analysis of Testability of a Flight Software Product Line

Fraunhofer CESE

Dharmalingam GanesanMikael Lindvall

1

NASA Goddard Space Flight Center

David McComasMaureen BartholomewSteve SlegelBarbara Medina

Architecture-based Unit Testing of the Flight Software Product Line

© 2010 Fraunhofer USA, Inc. Center for Experimental Software Engineering

Background

• Unit testing deals with testing individual modules in isolation– Assuming the correctness of dependent

modules• Bugs found during unit testing are cheaper

to fix than– Integration testing– System testing– Field reported defects

2© 2010 Fraunhofer USA, Inc. Center for Experimental Software Engineering

Unit Testing Challenges in Practice• Isolating the module under test from other

modules• Isolating from special OS and hardware

features• Handling the variability or configuration

dimension• Developing flexible build process (e.g.,

Makefiles) that facilitates unit testing


Research Questions

• What architectural design decisions facilitate or impede unit testing?

• How can we make use of the architecture to define the architecture of test code?

Approach: Case-study driven - NASA’s Core Flight Software Product Line (CFS)


Reusable Analysis Questions• Can modules be tested independently of other

modules they use?• Can modules be unit tested without access to

special hardware and/or OS?• How are variation points of each module being

handled during unit testing?• Is there a common look-and-feel in terms of the

way the modules are unit tested?• How easy it is to set-up a unit test? (additional questions are listed in the paper)


Source Code Makefiles

Test Suites

Makefiles of test suites

Data Extraction(Automated Parsing)

Call relation, Include relation,

variation points for test suites

Call relation, Include relation, variation points for source

code

Visualization, Query and Analysis

2

1

Technical Set-up for Analysis

6

Data extraction is automated using standard parsers

Data analysis and visualization are semi-automatic using tools: Relation Partition Algebra (RPA) query language SAVE-Light, SAVE Prefuse


CFS Application A CFS Application B CFS Application C …Core Flight Executive (cFE) Services Layer

Software Bus (SB)

File Service (FS) services

Table Services (TBL)Executive Services (ES)

Event Services (EVS)

OS Abstraction Layer cFE Board Support Package (BSP) Layer

Hardware Abstraction Layer

Real time OSBoard Support Package

(BSP)Device Drivers

Hardware

Product Line Structure of the CFS

7

See our SPLC 2009 paper for:

the heritage of the CFS the business goals of the CFS architectural analysis of the CFS implementation


Run-time Structure of the CFS

Inter-task Message Router (SW Bus)

SoftwareBus

CommandIngest

Telemetry Output

Memory

DwellChecksum

Memory

Manager

Some Mission Applications

cFE core Applications

Some CFS Applications

House-keeping

TimeServices

ExecutiveServices

EventServices

TableServices

FileServices* * * * * *

*

8

Applications communicate by publishing/subscribing messages to the busCore applications can also directly talk to each other using function calls

Example Variation Points: Maximum number of applications using the SW bus Maximum number of messages in the SW bus Maximum number of messages an application can receive


Mission Managed by cFE CFS Apps Status

CHIPS GSFC ✓ Successfully demonstrated

DSILCAS GSFC ✓ Integrated in FSW lab

GPM GSFC ✓ ✓ Integrated in FSW lab

LADEE ARC ✓ ✓ Integrated in FSW lab

LRO APL ✓ On orbit

MMS GSFC ✓ ✓ Integrated in FSW lab

Nanosat-6 AFRL/Cornell ✓ ✓ Integrated in FSW lab

RBSP APL ✓ Integrated in FSW lab

RLL APL ✓ Integrated in FSW lab

Solar Probe Plus

APL ✓ Integrated in FSW lab

Many missions depend on the quality of CFS/cFE and the reusable components it provides!

Missions using CFS/cFE


Executive Service (ES)

Software Bus (SB)

Timing Service (Time)

Table Service (TBL)File Service (FS)

Event Service (EVS)

cFE - Core

Dependencies of Core Modules

10

How to test each module independently of other modules?

Reverse engineered from the source code of the CFSBoxes denote modules (i.e. directories)Arrows denote code relations (e.g. include, call)


Unit Test Architecture of CFS• Core idea is to

– Define mocks/stubs for dependent modules– Systematically control return codes of

dependent modules• Mocks are independent of other mocks• Mocks have the same signature as the real

interface• Modules can be bound to mock modules

or to real modules– Facilitates incremental integration testing too!


How Return Code is Controlled?• UT_SetRtn_t is the key data structure:

– typedef struct {

uint32 count;

uint32 value;

} UT_SetRtn_t;

• Mocked functions are forced to return values based on the state of count

• For every mocked function, there is an instance of UT_SetRtn_t


Controlling the Return Code - Exampleextern UT_SetRtn_t SB_CreatePipeRtn;

// mock implementation of create pipe

int32 CFE_SB_CreatePipe (CFE_SB_PipeId_t *PipeIdPtr,

uint16 Depth,

char *PipeName) {

if (SB_CreatePipeRtn.count > 0) {

SB_CreatePipeRtn.count--;

if(SB_CreatePipeRtn.count == 0) {

return SB_CreatePipeRtn.value;

}

}

return CFE_SUCCESS;

}

13

void UT_SetRtnCode (UT_SetRtn_t *varPtr, int32 rtnVal, int32 cnt){ varPtr->value = rtnVal; varPtr->count = cnt;}

// forces CreatePipe to succeed in the first and fail in the second callUT_SetRtnCode (&SB_CreatePipeRtn, -1, 2);


Binding to Mock Modules

14

Executive Service (ES)

ut_evs_stubs ut_sb_stubs ut_time_stubs ut_tbl_stubs ut_fs_stubs

es_ut.c

ut_osapi_stubs

ut_bsp_stubs

The test suite for the Executive Service (ES)

Executive Service module uses stub implementations of dependent modules

The unit tests of ES control the return code of dependent modules to cover all paths


int32 CFE_ES_LoadLibrary(char *EntryPoint, char *LibName, …) { boolean LibSlotFound = FALSE; for ( i = 0; i < CFE_ES_MAX_LIBRARIES; i++ ) { if ( CFE_ES_Global.LibTable[i].RecordUsed == FALSE ) { LibSlotFound = TRUE; break; } } if(LibSlotFound == FALSE) return CFE_ES_ERR_LOAD_LIB; …;}

/* Test for loading more than max number of libraries */ for (j= 0; j < CFE_ES_MAX_LIBRARIES; j++) { CFE_ES_Global.LibTable[j].RecordUsed = TRUE; } Return = CFE_ES_LoadLibrary("EntryPoint","LibName“, …); UT_Report(Return == CFE_ES_ERR_LOAD_LIB, "CFE_ES_LoadLibrary“, ”No free library slots");

Testability and Module Secrets

15

CFE_ES_MAX_LIBRARIES is a variation point

CFE_ES_Global is an internal globalvariable of the ES module

How do we test for more than the max. no. of loaded libraries scenario?

Test code needs access to modules’ secrets!© 2010 Fraunhofer USA, Inc.

Center for Experimental Software Engineering

int32 CFE_SB_SendMsg(CFE_SB_Msg_t *MsgPtr) { /* check input parameter */if(MsgPtr == NULL){ CFE_EVS_SendEventWithAppID(“Send Err:Bad input argument”,…); return CFE_SB_BAD_ARGUMENT;}

MsgId = CFE_SB_GetMsgId(MsgPtr);/* validate the msgid in the message */if(CFE_SB_ValidateMsgId(MsgId) != CFE_SUCCESS) { CFE_EVS_SendEventWithAppID(“Send Err:Invalid MsgId”, …); return CFE_SB_BAD_ARGUMENT;} …}

void Test_SendMsg_NullPtr(void){ … ActRtn = CFE_SB_SendMsg(NULL); ExpRtn = CFE_SB_BAD_ARGUMENT; if(ActRtn != ExpRtn){ TestStat = CFE_FAIL; }

ExpRtn = 1; ActRtn = UT_GetNumEventsSent(); if(ActRtn != ExpRtn){ TestStat = CFE_FAIL; } …}

Same Return Code Impede Testing

Test code cannot easily verify which path was actually taken

Have to count the number of times log events (SendEvent)are used to justify the intended path

Test code

16© 2010 Fraunhofer USA, Inc.

Center for Experimental Software Engineering

linux/osapi.c rtems/osapi.c vxworks6/osapi.c Test/ut_osapi_stubs.cint32 OS_QueuePut(...){ ... sendTo(...); ...}

int32 OS_QueuePut(...){ ...rtems_message_queue_send(...); ...}

int32 OS_QueuePut(...){ ... msgQSend (...); ...}

int32 OS_QueuePut (...) { // Mock Implementation}

Software Bus (SB)

Testability, Abstract Interfaces and Alternative Implementations

17

Software Bus needs services of OS QueuesCFS needs to run on several OS typesOS APIs are often different among OSFor improved testability, variations among APIs have to be abstractedBenefits:

Mock light-weight implementations of APIs for unit testingUnit test code is free of variability issues to a large extent

options


Some Data

18

Core Module # of Functions Defined # Directly invoked in Unit Tests

SB 86 45

ES 117 68

EVS 33 12

Time 72 42

Table 60 41

File 11 11

Stub SB Stub ES Stub EVS Stub Time Stub TBL Stub FS

SB NA 11 3 1 0 1

ES 10 NA 4 3 1 4

EVS 8 10 NA 1 0 1

Time 9 8 2 NA 0

TBL 9 15 3 1 NA 3

FS 0 2 0 1 0 NA

Each public function has at least one dedicated test program

All internal functions are transitively tested

Only a few stubs are needed because of well-defined abstract interfaces!


Core Module # of Functions in Interface # Directly invoked in Unit Tests

SB 30 30ES 33 31EVS 7 7Time 24 24Table 14 13File 5 5

Conclusion• Programming to abstract interfaces is key

for improved testability• Internal details of modules have to be

open to facilitate unit testing– Apply architectural rules to prevent misusages

• Complete graph of dependencies do NOT imply poor design and low testability

• Treating testing as variation points help unit and incremental integration testing


Acknowledgement

• Lisa Montgomery of the NASA IV & V

• Sally Godfrey of the NASA GSFC

• All members of the CFS/cFE team


Analysis of Testability of a Flight Software Product Line

Technology

Transcript of Analysis of Testability of a Flight Software Product Line