Real Time Safety ERAU research conducted for the FAA under contract DTFA0301C00048 page 1 Assessment...

Real Time Safety

ERAU research conducted for the FAA under contract DTFA0301C00048

page 1

Assessment of Software Development Tools for Safety Critical Real-Time

SystemsFAA Contract DTFA0301C00048 Project Personnel:

Faculty: A.J.Kornecki J.Zalewski, N.Brixius

Students: J.P.Linardon, J.Labbe, L.Crawford, H.Lau, K.Hall,

D.Hearn, C.Sanoulliet, M.Milesi

Presented by: Andrew J. Kornecki

Department of Computer and Software Engineering

Embry Riddle Aeronautical University

Daytona Beach, FL 32114phone: (386) 226-6888 ; [email protected] ; http://faculty.erau.edu/korn

FY2005 Software/Complex Electronic Hardware Standardization Conference, Norfolk, VA, July 26-28, 2005

Real Time Safety


page 2

Outline

Project Background Research Approach Development Tool Assessment Development Tool Qualification Experiments Software Tool Forum Research Findings Conclusions

Real Time Safety


page 3

• E a c h C O T S t o o l c a p t u r e s a s i n g l e n a r r o w a s p e c t o f t h e s y s t e m d e s i g n / a r c h i t e c t u r e S y s t e m d e s i g n e r m u s t d e p e n d o n o t h e r t o o l s f o r a r c h i t e c t u r a l r e p r e s e n t a t i o n a n d i n t e g r a t i o n

C u s t o m w o r k a r o u n d s , p a t c h e s , a n d c o n v e r s i o n s n e e d e d a r o u n d e a c h C O T S t o o l , i n e a c h p r o j e c t

• S y s t e m i s c a p t u r e d i n d i f f e r e n t f o r m s i n d i f f e r e n t C O T S t o o l s M a n u a l t r a n s l a t i o n b e t w e e n t o o l s c a u s e s l o t o f d u p l i c a t e d e f f o r t

• P r o l i f e r a t i o n o f t o o l s a n d d e s i g n n o t a t i o n s L a c k o f s t a n d a r d d e s i g n / a r c h i t e c t u r e n o t a t i o n s l e a d s t o l e s s r e - u s e a c r o s s S B U s

P l a t f o r m d e p e n d e n c i e s g e t e m b e d d e d i n d e s i g n , r e d u c i n g p o r t a b i l i t y

M a t l a bS i m u l i n k

P r i c e

C o d e T e s t

S C A D E

E S C A P E( E P I C / D E O S )

V A L F A C

S t a t eM a t e

C o n f i g M g m tD O O R S

R e q u i r e m e n t s P r o d u c tP r e l i m i n a r y D e s i g n ,A n a l y s i s

D e t a i l e d D e s i g n C o d e ,T e s t

P l a t f o r mC o n f i g u r a t i o n , T e s t

F o r e S i g h t

U M L - B a s e d t o o l s ( e . g . R o s e )

I O G T( A R I N C 6 5 3 )

i S M A R T

C O R E A U T T

M a t l a bR T W

C u r r e n t T o o l s : I s s u e s , a n d L i m i t a t i o n s C u r r e n t T o o l s : I s s u e s , a n d L i m i t a t i o n s C u r r e n t T o o l s : I s s u e s , a n d L i m i t a t i o n s

With permission of Honeywell Software Solution Lab

Real Time Safety


page 4

Background: DO-178B Definitions Software tool: A computer program used to help

develop, test, analyse, produce or modify another program or its documentation

Software development tools: Tools whose output is part of airborne software and thus can introduce errors

References:o Section 12.2 of DO-178B o Chapter 9 in Order N8110.49 Software Approval

Guidelines NOTE: DO-178B is a “guidance” document only

focusing on software processes and objectives to comply with these processes

The development tools provide a transformation between the input and output artifacts

Real Time Safety


page 5

Background: Not Qualified vs. Qualified Development Tools Section 9.3 of Order N8110.49

DevelopmentActivity

(Using Tool) Verification

Activity

inputartifact

outputartifact

feedback

Can tool insert error

into airborne software?

Will the tool’s output NOT be verified (as specified in

DO-178B)?

Are processes of

DO-178B eliminated, reduced, or automated

by use of tool?

TOOL MUST BE QUALIFIED

NO QUALIFICATION

NECESSARY

NO

NO

NOYES

YES

YES

Real Time Safety


page 6

Background: Hypotheses Theory:

o Tools improve development process by automation and reduction of repetitive tasks

o Qualified development tools could help with the current certification process by reducing the verification burden of the intermediate software lifecycle artifacts

Practice:o Although the tool qualification is a well

established concept, only a handful of development tools were even attempted to be qualified

The reasons for this situation have been explored in the research

Real Time Safety


page 7

Background: Survey - Results

0

5

10

15

20

25

Follow-up on the e-mail to the FAA Software Certification mailing list (only 14 responses from over 500 individuals)

28 respondents representing airborne software (74%) and the FAA personnel (14%) at the FAA Software Conference, May 2002, Dallas/Ft.Worth,

count

0

2

4

6

8

10

12

14

Limited testing Extensive reviewand testing

Tool qualificationpackage provided

by vendor

Tool documentationreview without

testing

Evaluation based onbrochure

count

Real Time Safety


page 8

Background: Survey - Major Issues

Differing perspectives: applicant, certifying authority, tool vendor

Cost Obsolescence Tool functionality and

compatibility with existing development environment

Tool vendors typically not used to the level of effort required for a DO-178B compliance

False vendor claims (scaling, independence)

Difficult to identify features of a tool which could cause/contribute to errors in the target software

Inadequate documentation, training and understanding of development tool

Discouraging rigor of tool qualification and perception of qualification high cost

Business model prevents from investing in tool qualification

Need for re-qualification for each new certification project

Tool reliability as a measure is questionable (How to show it? Does it matter?)

Vendor support and alternate means for COTS tools

Real Time Safety


page 9

Outline


Real Time Safety


page 10

Research Approach: FAA Contract via AACE A number of software tool manufacturers claim their

tool will meet DO-178B criteria 2001 Aircraft Airworthiness Center of Excellence

(AACE) research program solicitation Contract DTFA0301C00048: “Assessment of

Development Tools for Safety Critical Real-Time Systems”

Three phase research activity (Jan 2002-June 2005)o Phase One: Baseline and Taxonomyo Phase Two: Experiment and Feedbacko Phase Three: Assessment and Guidelines

The purpose of the project has been to assess the safety, quality, economic, efficiency benefits, and advantages/disadvantages of software tool suites used to develop software

Real Time Safety


page 11

•Universities•Research Institutes•Industry

•FAA•IEEE•ISO•RTCA•…

Tool Vendors

Applicants -Safety-CriticalSoftwareDevelopers

rese

arch

resu

lts

information

information

regulations / standards

soft

war

e to

ols

Software Aspects of CertificationStakeholder Diagram

Approval

Certifying Authorities & DER’s

Real Time Safety


page 12

Research Approach: Objectives To review literature to establish a base for assessment of

software development tools To identify the development tools, their impact on the

software development lifecycle, and the qualification efforts To conduct industry surveys and analyze the

industry/government feedback to define tool evaluation criteria

To create a taxonomy and a set of criteria/guidelines for the tool selection and qualification

To assess effort and defects during a case study collecting experimental development process data

To analyze the data, edit the reports, and disseminate the results

To provide input for guidelines on tools selection and qualification

Real Time Safety


page 13

Research Approach: Problem Statement - Research Questions

Industry Viewo Qualification Process and Approacheso Safe Use of Toolso Future Trends

Qualificationo What, Why and How?o Barriers and Factors

Quality Assessmento Safety Critical Real-Time Specificso Mechanisms and Methods of Assessmento Tool Support Impact

Tool Evaluation Taxonomyo Tool Functionality and Categorieso Evaluation Criteria

Real Time Safety


page 14

Research Approach: Project Workflow

Tool Categorization(software lifecycle, industry practices, etc.)

Software Tool Reference(extensive list of tools currently being used by industry)

Tool Evaluation Taxonomy(evaluation methods & criteria)

Background Information(standards, qualificationhistory, etc.)

Industry Viewpoint(surveys, interviews, email correspondence, information from tool vendors, etc.)

Tools and PlatformPreparation

(workstation set-up, SW toolacquisition, SW tool installation)

Experiment Preparation

(product requirements design,tool assessment methodologyand mechanisms selection,

process design and associatedscripts write-up, )

Initial Experiments

(product development, processscripts execution, and data

collection)

Data Integration

(preliminary data verification andprocessing)

Data Analysis

(calculation, observation,inference, and conclusion)

Experiment Improvement

(Enhanced process scripts,simplified product requirements,

and questionnaires creation(Training/Support/ACG) )

Controlled Experiments

(product development, processscripts execution, and data

collection)

SW ProductQuality

Attributes andAssessment

MethodsResearch

(continuinginvestigation onpast and current

researchrelating to this

topic)

Identification of Tool Landscape

Real Time Safety


page 15

Outline


Real Time Safety


page 16

Assessment: Goals

To help providing sufficient information to support the qualification of a tool

To develop a set of criteria and methods to assess (measure) the quality of the tool: how well provides its function?

ToolDevelopment

ToolUse

Product Execution

Meta-evaluation Micro-evaluation

Macro-evaluation

TOOL PRODUCT

RESULTS

Two approaches:o Formal qualification-oriented

evaluation of functionalitieso Informal utilization-oriented

hands-on evaluation of the tool in operation

Real Time Safety


page 17

Tool category

Functional attributes

Non- Functional attributes

Technical attributes

Non-technical attributes

Evaluation methods

Concerns

Factors

Evaluation methods

Concerns

Factors

Evaluation methods

Concerns

Factors

Non-functional: Technical:

dependability, performance, security

Non-technical: support, cost, vendor viability, training

Assessment: Tool Evaluation Taxonomy

Source: TR CMU/SEI-95-TR-021 “Quality Attributes”, Barbacci, M. at al

Functional:o determinism, correctness, robustness, traceability,

standards conformance

Real Time Safety


page 18

Assessment: Functional Attributes

concerns factors methodsDeterminism Tools’ architecture

PredictabilityConversion and code generationSubset of language used in product

Architecture assessment (meta)Tool use (macro)Algorithm inspection (meta)Code inspection (macro)

Robustness Partition integrityBoundary conditionsArchitecture / Coupling

Code inspection (macro)Code testing (micro)Design review (meta)

Traceability Product’s architectureProduct’s code / coding rules

Model inspection (macro)Code inspection (macro)

Correctness Conversion and code generationSubset of language usedSyntax and formalityReal-time managementLanguage representation rules

Algorithm inspection (meta)Code inspection (macro)Formal techniques (macro)Timing evaluation (micro)Formal methods (meta)

Standards Conformance

DesignCodingBehavioral

Model inspection (macro)Code inspection (macro)System testing (micro)

Real Time Safety


page 19

RequirementsTool

StructuralDesignTool

FunctionalDesignTool

typically with code generator functionality

ImplementationTool

TestingTool

Target(with RTOS)

or/and

Tool Categories

AnalysisTool

e.g.: VxWorks

QNXOSE

IntegrityLynxOS

e.g.: CodeTestTestRT

VectorCastInsure++

Integrated Development Environment

(IDE)e.g.:

TornadoMulti

e.g.: RhapsodyRoseRTSTOODArtisan

e.g.: SCADEMatlab

BEACONSildex

e.g.: RapidRMATimeWiz

e.g.: ReqtifyDOORSSpecTRMDOME

Assessment: Tools Landscape

Real Time Safety


page 20

Assessment: Framework

Data required for evaluation (each category needs data items to collect criteria/metrics):o tool data

o model data

o source code data

softwarerequirements

design modeland

source code

TOOLUSE

developer/evaluator

Real Time Safety


page 21

Assessment: Data Items

Tool Model Code

Operational Profile Environment

Specification Functional

Characteristics Methodology and

Notation Primitives Support for Expressing

Timing and Safety Properties

Requirement Document

Design Document Test Scenarios and

Results

Software Requirement Document

Software Safety Properties

Size of Model (blocks, modules, interfaces)

Software Operational Scenarios

Expected Timing Characteristics

Expected Behavioral Model (states, sequences)

Partial vs. full code generation

Code properties (language, subsets)

Code parameters (LOC, quantitative measures)

Readability (names, indentation, comments)

Partitioning (separation of data, mutual exclusion)

Timing Executable size

(memory allocation)

Real Time Safety


page 22

Indicators(attributes, factors,criteria, metrics)

Measurement Methods(evaluation techniques)

Evaluationresults

Software DesignDescription (model)

Design Standards(s/w architecture, meeting DO-178B objectives)

SoftwareRequirementsSpecification

ToolData

ModelData

D evelopment P rocess

T ool Evaluation

Taxonomy V iew

B ehavioral V iew

P roject V iew

Qualification V iew

CodeData

Generated Code

Real Time Safety


page 23

Outline


Real Time Safety


page 24

Qualification: Objective

Establish assurance that the software requirements, as submitted to the tool and developer, are correctly and completely transformed into the generated source code

Qualification process involves the user to perform an audit of the tool, with an active cooperation of the tool vendor, who needs to show that:

o Tool requirements are properly documented

o Tool have appropriate configuration management

o Tool has been satisfactorily tested against its requirements

Procedures controlling the use of tools (environment, constraints, limitations, version control) are as important as tool itself

Real Time Safety


page 25

Qualification: Process

Tool qualification is an integral component of a specific certification program referenced within the Plan for Software Aspects of Certification (PSAC) and Software Accomplishment Summary (SAS) of the original certification project

For development tools the required documents are:

o Tool Qualification Plan (TQP)

o Tool Operational Requirements (TOR)

o Tool Accomplishment Summary (TAS) Other data, required for review, include Tool Configuration

Management Index, Tool Development Data, Tool Verification Records, Tool Quality Assurance Records, Tool Configuration Management Records, etc.

Real Time Safety


page 26

Outline


Real Time Safety


page 27

Experiments: Selected Tools

For detailed analysis we selected design tools, with a code generator functionality, identifying two groups of different modeling paradigm:

Structural/Discrete Models (UML-oriented):o Rose Real Time / Rational http://www.rational.com/ o Rhapsody / iLogix http://www.ilogix.com/ o Real Time Studio Professional / Artisan http://www.artisansw.com/ o STOOD / TNI-Valiosys http://www.tni-valiosys.com/

Functional/Continuous Models (Block-oriented):o SCADE / Esterel http://www.esterel-technologies.com/ o Sildex / TNI-Valiosys http://www.tni-valiosys.com/ o Simulink/RTW / Mathworks / RTW http://www.mathworks

.com/products/ o Tau SDL Suite / Telelogic http://www.telelogic.com

Real Time Safety


page 28

Experiments: Description

Process:o Preparation: project and tool familiarization

o Model development, code generation, and implementation

o Data collection

o Post-mortem System:

o Preliminary Experiment (4 tools, 4 developers): flight data collection with simple processing

(averaging, time-stamping) and displaying results on the terminal under user control

o Controlled Experiment (6 tools, 14 developers): hair-dryer simulator (training); microwave oven simulator (data collection)

Real Time Safety


page 29

Experiments: Criteria / Sources

{178B} RTCA Inc. (1992) Software Considerations in Aiborne Systems and Equipment Certification, Report RTCA/DO-178B, Washington, DC

{ISO} ISO/IEC 14102-1995 (1995) Information Technology – Guideline for the Evaluation and Selection of CASE Tools, ISO, Geneva, Switzerland, 15 November 1995

{FAA} U.S Department of Transportation, Federal Aviation Administration (1991) Software Quality Metrics, Report DOT/FAA CT-91/1, 1991

{VTT} Ihme T, Kumara P, Suihkonen K, Holsti N, Paakko M (1998) Developing Application Frameworks for Mission-Critical Software: Using Space Applications as an Example, Research Notes 1933, Technical Research Centre of Finland, Espoo, 1998.

{BSC} Wichmann B (1999) Guidance for the Adoption of Tools for Use in Safety Related Software Development, Report, British Computer Society, March 1999.

Real Time Safety


page 30

Experiments: Criteria Selection

Criteria survey 178B ISO FAA VTT BCS

Consistency *

Efficiency * *

Functionality * * *

Maintainability * *

Modifiability * *

Portability * *

Reliability * * * *

Robustness *

Traceability * *

Usability * *

Efficiency: code size

Functionality: questionnaire feedback

Traceability: manual tracking between model and code

Usability: developers effort (PSP)

Real Time Safety


page 31

Preliminary Experiment

target board

sensors

human-machine interface

aircraftsimulator (OpalRT)

Development tools:Matlab/Simulink/RTW, SCADE, Sildex, Artisan

RTOS (VxWorks)

IDE (Tornado)

Purpose: to establish a base for assessment of software development tools

Personal Software Process used to capture effort data

Process improvement

Traceability assessment matching: requirements design code

Real Time Safety


page 32

Example: Flight Data Acquisition Modeling

Operator

Data System

Operator_initialiaze-s/Identifies_Numbe-

rofParam

Operator_Sel-ects_ParamID

Operator_Identifie-s_Output'sAveragi-

ngFrequency

DataSystem_sends_P-

aramValue

System NotInitialized

System Initialized

System Initialized_Calculating/Printing Average

Control Unit

/

/operator enters number of parameters, selects frequency for calculation & parameter ID

/Data System sends parameter values& frequency reached

/operator ex its system

/Data System sends parameter values& frequency not reached

/Data System sends parameter values & frequency reached

/Data System sends parameter values& frequency not reached

::Control Unit

array parameters

array Frequencies

int numberOfParameters

char * timestamp ()

void set_paramID (in array parameterArray)

void set_num (in unsigned short param)

void set_avgFreq (in array frequencyArray)

double get_avgdata (in array paramArray, in int tempParamFreq, in int *counter, in double *total, in int i, in string names[], in string units[], in int index)

::Average Calculator

double calc_avg (in double *dTotal, in int *iCounter)

::I/O Unit

int numParam

typedef double valArray[maxParam]

typedef int array[maxParam]

array parameterArray

array frequenctArray

valArray valuesArray

int counter[3]

double total[3]

string names[45]

string units[45]

void input ()

void display (in char * *str)

void request_paramID (in array parameterArray)

unsigned short request_num ()

void request_avgFreq (in array frequenctArray, in array parameterArray)

void getPackets (in array paramArray, in array freqArray, in string names[], in int numParam, in string units[])

1

1

1

1

Operator_Identifies_Output'sAveragingFrequency

Description Operator Data Input Control Unit

Data Input requests averaging frequency request_avgFreqOperator enters the parameter ID enter_avgFreqControl Unit sets the parameter ID set_avgFreq

Real Time Safety


page 33

Preliminary Experiment: Usability

0

10

20

30

40

50

60

70

80

90

100

Tool A Tool B Tool C Tool D

Postmortem

MeasurementModel/Code

Preparation

Real Time Safety


page 34

Controlled Experiment

target board

human-machine interface

Development tools:SCADE, Sildex, Rhapsody, Artisan, Tau, STOODRTOS

(VxWorks)

IDE (Tornado)

Purpose: to collect data supporting assessment of software development tools

Personal Software Process used to capture effort data

Questionnaire used to collect qualitative data

Two-phase approacho Training model:

hair-dryer (4 requirements)o Experimental model:

microwave (10 requirements)

Real Time Safety


page 35

Requirements microwave oven:1. The oven shall allow user to set the cooking time in

minutes and seconds (from default 00:00 to 59:59). 2. The oven shall allow user to set the power level (in

the range from default 1 to 5).3. The start of cooking shall initiate on explicit user

request.4. When the cooking starts, the oven shall turn on the

light and the rotisserie motor for the specified time period.

5. When the cooking starts the oven shall cycle the microwave emitter on and off: the power level of 5 means that the emitter is on all the time, the power level of 1 means that the emitter is on only 1/5 th of the time.

6. The oven shall display the remaining time of the cooking and the power level.

7. When the time period expires, the audible sound shall be generated and the light, motor, and emitter shall be turned off.

8. The oven shall turn on the emitter and the motor only when the door is closed.

9. The oven shall turn on the light always when the door is open.

10. The oven shall allow the user to reset at any time (to the default values)

1. The system shall allow user to select motor speed (off, low, or high).

2. The system shall apply power to motor depending on the selected speed setting.

3. The system shall cycle the heater (30 seconds on and 30 seconds off) when in low- and high- speed modes.

4. The system display shall show the selected speed, heater status and the count down time when the heater is on.

hair-dryer:

Real Time Safety


page 36

Example: Microwave Modeling

Real Time Safety


page 37


Real Time Safety


page 38


active <<'C Application'>>class MicrowaveMicrowaveClasses {1/3}active <<'C Application'>>class MicrowaveMicrowaveClasses {1/3}

Oven

time : Integerpart d1 : Doorpart l1 : Lightpart e1 : Emitter

doDoorOpen()

doDoorClose()doReset()

Oven

time : Integerpart d1 : Doorpart l1 : Lightpart e1 : Emitter

doDoorOpen()

doDoorClose()doReset()

ovenInPortovenInPort

FromUserFromUser

ovenOutPortovenOutPort

ToUserToUser

Door

-doorOpen : Boolean

+open()

+close()

Door

-doorOpen : Boolean

+open()

+close()

d1

d1

Light

lightOn : Boolean

+turnOn()+turnOff()

Light

lightOn : Boolean

+turnOn()+turnOff()

l1

l1

Emitter

powerLevel : Integer

+ setPower( pl :Integer)

Emitter

powerLevel : Integer

+ setPower( pl :Integer)

e1

e1

Motor

motorOn : Boolean

+startMotor()+stopMotor()

Motor

motorOn : Boolean

+startMotor()+stopMotor()

m1

m1

UI

run ()initialize ()

userCommand : IntegerinputValue : Integer

UI

run ()initialize ()

userCommand : IntegerinputValue : Integer

'input''input'

ToUserToUser

'output''output'

FromUserFromUser

+ dl

+ dl

active <<'C Application'>>class

Microwave

MicrowaveCommunication {2/3}active <<'C Application'>>class

Microwave

MicrowaveCommunication {2/3}

o1:Oven

o1:Oven

ovenInPortovenInPort

FromUserFromUser

u1:UI

u1:UI'input''input'

ToUserToUser

'output''output'

FromUserFromUser

UserOven

UserOven

statemachine initializeEmitterStateChart {1/1}statemachine initializeEmitterStateChart {1/1}

EmitterReadyEmitterReady

powerLevel = 1;emitterOn = false;cycle = 0;

powerLevel = 1;emitterOn = false;cycle = 0;

emit()emit()

emitOnemitOn

truetrue falsefalse

ÊmitterStatus(true);emitterOn = true;ÊmitterStatus(true);emitterOn = true;

ÊmitterStatus(false);emitterOn = false;ÊmitterStatus(false);emitterOn = false;

toReadytoReady

emitterReset()emitterReset()

toReadytoReady

toReadytoReadytoReadytoReady

Real Time Safety


page 39


Off>

On>

Off>

On>

tm(1000)

unplug plugin

On>

Off>

On>

Off>

turnLightOn turnLightOff

On>

Idle> Off>

On>

Idle> Off>

tm(cycle_on_time)

turnOff

tm(cycle_off_time) start

turnOff

+light : int

+Light()+getLight():int+~Light()+setLight(int p_light):+startBehavior():OM+turnLightOn()+turnLightOff()

Light

+timer : int+countdown : int

+Timer()+getTimer():int+~Timer()+getCountdown():int+setCountdown(int p_countdow+setTimer(int p_timer):void+startBehavior():OMBoolean+breachZero()+enteredTime()+pause()+start()+clear()

Timer

-OPEN : int-CLOSED : int-OFF : int-IDLE : int-ON : int+powerLevel : int

+getItsLight():Light*+setItsLight(Light* p_Light):void+getItsTimer():Timer*+setItsTimer(Timer* p_Timer):v#cleanUpRelations():void+startBehavior():OMBoolean-getCLOSED():int-setCLOSED(int p_CLOSED):v-getIDLE():int-setIDLE(int p_IDLE):void-getOFF():int-setOFF(int p_OFF):void-getON():int-setON(int p_ON):void-getOPEN():int-setOPEN(int p_OPEN):void+getPowerLevel():int+setPowerLevel(int p_powerLe+getItsDoor():Door*+setItsDoor(Door* p_Door):void+getItsEmitter():Emitter*+setItsEmitter(Emitter* p_Emitt+getItsMotor():Motor*+setItsMotor(Motor* p_Motor):v+Microwave()+~Microwave()+plugin()+unplug()

Microwave

+motor : int+Motor()+getMotor():int+~Motor()+setMotor(int p_motor):void+startBehavior():OMBoolean+turnMotorOn()+turnMotorOff()

Motor

+door : int+Door()+getDoor():int+~Door()+setDoor(int p_door):void+startBehavior():OMBoolean+openDoor()+closeDoor()

Door

+emitter : int+cycle_on_time : int

+Emitter()+getEmitter():int+~Emitter()+setEmitter(int p_emitter):void+startBehavior():OMBoolean+getCycle_off_time():int+setCycle_off_time(int p_cycle_+getCycle_on_time():int+setCycle_on_time(int p_cycle_+start()+pause()+resume()+turnOff()

Emitter

itsTimer

itsEmitter

1

1

itsLight1

itsMotor

1

itsDoor

1

Idle> Off>

On>

Idle> Off>

On>

clear

enteredTime

breachZero/cout << "BEEP!!" << endl;

start

clear

pause

tm(1000)

Real Time Safety


page 40

Controlled Experiment: Efficiency

0

2000

4000

6000

8000

10000

12000

Tool L Tool M Tool N Tool O Tool P Tool Q

Real Time Safety


page 41

Controlled Experiment: Usability

Real Time Safety


page 42

Controlled Experiment: Functionality

0,00,5

1,0

1,5

2,0

2,5

3,03,5

4,0

Tutorial

User Manuals

Readability

Flexibility

AVERAGE

Real Time Safety


page 43

Controlled Experiment: Traceability

Small system size (only ten requirements) allowed developers to:o identify and manually trace all requirements to their

equivalent modeling components o identify and manually trace all modeling components

to their equivalent code componentso identify and manually trace all code components to

their equivalent modeling components For all tools:

the model and code components matched the experiment identified code components generated

as the run-time framework with no direct trace to the requirements

Real Time Safety


page 44

Experiments: Viewpoints

Software Engineering Viewpoint (Discrete Models):

o Interactive systems which use asynchronous languages based on interleaving tasks and operating systems principles (viewed as non-deterministic)

Control Engineering Viewpoint (Continuous Models):

o Reactive systems using event sequencing and logical time abstraction on common discrete time scale computing one step at the time (considered to be deterministic)Formal - by reducing the system to set of dynamic

equationsPractical - by modeling and solving differential

equations

Real Time Safety


page 45

Experiments: Determinism

Restrictive interpretation of determinism: the same input necessarily leads to exactly the same output

More accurate interpretation of determinism for tools: established the ability to determine correctness of the output from the tool

Approach:

o Construct a state machine using variation of build sequences (the states and transitions built in different order)

o Analyze the code generated from these variable builds

The experiments show that the generated code may have different structure, but provides the same behavior

Real Time Safety


page 46

Behavioral Experiment: Model

Create simple “producer-consumer” state machine Implement the model using various tools Test the behavior of generated code

Real Time Safety


page 47

Behavioral Experiment: Results

Tool Behavior

TOOL X Upon receiving and consuming EV1, the SUBCHART1 alternated between entering S2 and S3. The specific timing of the tool in resetting A to TRUE allowed the entry to state S2, even though it was not intention of the original design

TOOL Y With the guarded transitions between states in the SUBCHART1, transitions occurred between states in the SUBCHART2 only, and the events are generated, but EV1 and EV2 were never consumed. No transitions from S1 in SUBCHART1. No entry into S2 nor S3 ever occurs

TOOL Z With the guarded transitions in the SUBCHART1 the events EV1 and EV2 would occasionally be consumed. Several transitions would occur in SUBCHART2 before either EV1 or EV2 would be consumed. State S2 was never reached. Transitions were only between states S1 and S3

Real Time Safety


page 48

Experiments: Synchronicity

The synchronous approach enables to achieve determinism and is suited for time-triggered applications/functions

It is based on the abstract viewpoint that programs instantaneously and deterministically react to input events coming from their environment

The programs cyclically (in the loop):

o read inputs (events & values)

o compute the systems outputs and/or new states

o write the outputs Generally the loop is performed according to a basic

clock; some computations may be performed at a lower pace (for instance every two cycles)

The approach used in qualified tools (SCADE)

Real Time Safety


page 49

Experiment: Comments

Tools with code generators allow developers to focus on a higher level of abstraction rather than mundane coding

However …o Specific tool’s approach may constrain

development methodologyo Modern development tools require a long learning

curveo Inadequate documentation and tutorialso Possible tool malfunction due to workstation

environment o Unclear messages in code generation, compiling

and downloading

Real Time Safety


page 50

Outline


Real Time Safety


page 51

Software Tool Forum: ERAU Daytona Beach, FL

May 18/19, 2004: 150 participants representing 68 organizations (industry, government, academia): Ada Core Technologies, Airbus, Aircraft Braking Systems, ATL Software, Altair Avionics, Ametek Aerospace, Aonix, Artisan Software, Avidyne Corp, Avionics, Avionyx, Inc., Avista Inc., BAE Systems, Barco, Bechtel, Belcan, Boeing, CertTech LLC, Cessna Aircraft, CMC Electronics, Comm-Nav, Crane Aerospace & Electronics, CS Canada Inc., Directorate of Technical Airworthiness, DiSTI, Embedded Plus, ERAU, EMBRAER, Engenuity Technologies, ENSCO, Inc., Escher Technologies, Inc., FAA, FGCU, Garmin, GB Tech Inc,. GE Aircraft Engines, General Aviation Manufacturers, Goodrich Corp., Green Hills Software, Gulfstream Aerospace, Hamilton Sundstrand, High Integrity Solutions, Honeywell, IAI, Iowa State University, LDRA Technology Inc., Lockheed Martin, Metrowerks, MPC Products, NASA Langley, Polyspace, Pratt & Whitney, Real-Time Software Solutions, Rockwell – Collins, Rolls Royce, Safe Flight Instruments, Software Engineering Institute, Software Productivity Consortium, Teledyne Controls, MathWorks, MITRE Corp, Titan Systems, Transport Canada, TTTech, University of Minnesota, University of Paderborn, University of York, Vector Software, Verocel

Identification of issues to consider in the future actions and need for clarification/guidelines

Real Time Safety


page 52

Software Tool Forum: Priority List (1) Development tool qualification criteria need to be modified

o Criteria for tool qualification too stringento Differences between the tool and the target softwareo Guidance for COTS missingo Platform issueso Partial credit would be neededo Flexible rigor for tool qualificationo Code reviews issues

Criteria needs to be established to fit the Model-Based Development (MBD)o Lifecycle considerationo Definition of what is the “source code”o Fuzzy boundary between requirements and implementationo Structural coverage for modelso How to analyze the models?o Thorough analysis for the generated code?o Guidelines for model reviews

Real Time Safety


page 53

Software Tool Forum: Priority List (2)

Qualification criteria need to be developed that enable qualification to be carried from one program to anothero Definition of certification vs. qualificationo Reuse credit for the tool softwareo Formal qualification approval documento Tool upgrades and their impact on qualification statuso Specific list of documents required for qualification credit

Automatic Code Generator (ACG) usage and qualification require developing and documenting different approacheso Specific qualification process for ACG technologyo Is automatically generated code review practical?o Do we need to qualify ACG? (we do not do it for

compilers)

Real Time Safety


page 54

Outline


Real Time Safety


page 55

Research Findings: Industry View Basic Issues

o Reliance on verificationo Simplicity of in-house toolso Intellectual property issue for COTS

Qualification Processo Current guidelines restrictions o “Business as usual” vs. trust in advances of

software engineering technology Acceptance of Qualification Approaches

o What is “source code” in MDD paradigm?o Development tool “determinism”?o Need for tool testing and verification o Roadways: the simplicity of tool function,

separation of concerns, partitioning, use of model checking and formal evaluation

Real Time Safety


page 56

Research Findings: Industry View (cont) Safe Usage of Tools

o Education and training o Tool constraints and limitation of development toolo lifecycle with safety closely interfaced process o Strict tool version control o Control tool platforms and operating environments

Kinds of Development Toolso State of the art ahead of the regulations o Futuristic features vs. quality and applicabilityo Design properties assuranceo Easy interface

Tools Considered for Qualificationo Simple automatic transformation utilities (in

house) o The applicant holds intellectual property and datao COTS design tools with code generation capability

Real Time Safety


page 57

Research Findings: Qualification

Why do we need to Qualify Development Tools?o Reduction of development and certification efforto Focus on higher level of abstractiono Reuse

What Tools Were Attempted to be Qualified?o Few in-house tools qualified on certified projects

(CLARA, GALA, CTG, GPU, UTBT)o COTS qualified by European JAA: SCADE KCG

(Esterel Technologies) and VAPS CG (eNGENUITY)o Interest to qualify functional/block-oriented tools

(Mathworks, TNI-Software, Applied Dynamics, and National Instruments)

o Limited interest to qualify the structural/object-oriented tools (iLogix, TNI-Valiosys, IBM/Rational, and Artisan Software)

Real Time Safety


page 58

Research Findings: Qualification (cont)

How to Achieve Qualification for COTS Tools?o DO-178B is revision: stand-alone tool

qualification? o Version control, precise definition of operational

environment, constraints, and limitations o Availability of tool software development data

Development Tools Qualification Barrierso The state of regulations and guidelines o The business model, lack of incentives, cost o Lack of comprehensive tool data o Certification is the goal

Factors Regarding Tool Qualificationo Identification of metricso An independent qualification lab?o Development tool information disclosure? o FAA-sponsored database?

Real Time Safety


page 59

Research Findings: Quality Assessment Assessment of Quality for Safety/Real-Time

o Development tools are software artifacts o Software development strategies for safety o Target application vs. development tool software

Evaluation Mechanisms and Methodso Product perspective: functionality and quality of

service o Process perspective: following defined procedures

Evaluation Criteriao Tool selection o Standards compliance o Criteria adoption

Log-Term Supporto Tool vendors vs. regulated industry o Software upgrades, updates, and obsolescenceo Software industry instability

Real Time Safety


page 60

Research Findings: Evaluation Taxonomy Tool Functionalities (design tool / ACG)

o Graphical model representing system propertieso Validation, simulation and animationo Automatic translation and target interfaceo Document generation and the version control

Tool Categorizationo Reflection of the software lifecycle phaseso Focus on the composition and behavior o Quality of model and correctness of

transformation Categories/Function Vital for Development

o Model Driven Development paradigmo ACG interest: quality of the translation

Categories/Functions the Need to be Evaluated o Code generationo Complexity of modern toolso Qualification of narrow functionality of the tool

Real Time Safety


page 61

Outline


Real Time Safety


page 62

Conclusions: How to Show Correctness of Code Generation?

Generate test suites from specificationo Formal representation for

specification modelo Compare specification

behavior with generated code

o Automate, not eliminate, unit testing

The match of two outputs makes an argument on correctness of generated code

Specification/Model

Implementation

Output

Output

Specification Based Tests

Generate

{source: Whalen and Heimdahl}

Real Time Safety


page 63

Conclusions: COTS Tool Qualification Strategy

Use existing software lifecycle items:o Tool source code

o Tool design documentation (headers and comments)

o Configuration management and SQA policies Reverse engineer (reconstruct):

o Tool Software Requirements Specification

o Tool Software Development Plan

o Tool Software Verification Plan

o Tool Software Design Document

o Tool Traceability Matrix

o Tool Software Accomplishment Summary

o Tool Software Test Results

Real Time Safety


page 64

Conclusions: Challenges for Development Tools

How to Validate Models?o The models must satisfy the requirementso The properties used in analysis must hold o Model testing is of paramount importance

How to Validate Tools?o We will rely a lot on tools for model validation, can we trust

them?o Creative use of testing necessary

How to Verify and Validate Generated Code?o Can we trust that the translation was correct?o Test automation is critical o Includes output to analysis tools

How to Handle Tool Evolution?o Tools will evolve and change—re-qualificationo Models will not come in one language—translation

Real Time Safety


page 65

Conclusions: Final Remarks

New technologies and tool capabilities Model Driven Development paradigm Cost of tool qualification Partial certification credit How to handle tool evolution? Tool audition independent of application? Update DO-178B/ED-12B Develop a new guidance document

dedicated to software tools Define research tasks to address specific

tool issues

Real Time Safety


page 66

Publications Kornecki, A., Zalewski, J., (2005), “Experimental Evaluation of Software Development Tools for Safety Critical

Real-Time Systems” submitted to NASA Journal Innovations in Systems and Software Engineering, May, 2005

Kornecki, A., Zalewski, J., (2005), “Software Development Tool Qualification from the DO-178B Certification Perspective” submitted to Crosstalk: the Journal of Defense Software Engineering, May, 2005

Kornecki, A., Zalewski, J., (2005), “Process-Based Experiment for Design Tool Assessment in Real-Time Safety-Critical Software Development”, to be printed in proceedings of 29 th Annual IEEE/NASA Software Engineering Workshop, April 2005

Zalewski J.; Trawczynski D.; Sosnowski J.; Kornecki, A., (2005), “Safety Issues in Avionics and Automotive Databuses”, to be printed in proceedings of International Federation for Automatic Control IFAC Congress Praha, Czech Republic, July 2005

Kornecki, A., Hall, K., (2004), “Approaches to Assure Safety in Fly-by-wire Systems: Airbus vs. Boeing”, Proceeding of the Eight IASTED International Conference on Software Engineering and Applications (SEA 2004), ISBN: 0-88986-427-6, MIT, Cambridge, MA, November 2004, CD-edition

Kornecki, A., Zalewski, J., (2004), “Criteria for Software Tools Evaluation in the Development of Safety-Critical Real-Time Systems”, Proceeding of 28th PSAM7-ESREL’04 Conference, Berlin, Germany, June 2004, pp. 2364-2370

Kornecki, A., Erwin, J. (2004), “Characteristics of Safety Critical Software”, Proceedings of the 22 nd International System Safety Conference, System Safety Society, ISBN 0-9721385-4-4, Providence, RI, August 2004, CD-edition

Kornecki, A., Hall, K., Hearn, D., Lau, H., Zalewski, J., (2004), “Evaluation of Software Development Tools for High Assurance Safety Critical Systems”, fast Abstract in Proceedings of 8 th IEEE International Symposium on High Assurance Systems Engineering, HASE’04, ISSN 1530-2059, March 2004, pp. 273-274

Crawford, L., Erwin, J., Grimaldi, S., Mitra, S., Kornecki, A., Gluch, D., (2004), “A Study of Automatic Code Generation for Safety-Critical Software: Preliminary Report”, fast Abstract in Proceedings of 8 th IEEE International Symposium on High Assurance Systems Engineering, HASE’04, ISSN 1530-2059, March 2004, pp. 287-288

Kornecki, A., Zalewski, J., (2003), “Design Tool Assessment for Safety-Critical Software Development”, Proceeding of 28th IEEE/NASA Software Engineering Workshop, December 2003, pp. 105-113

Kornecki, A., Zalewski, J., (2003), “Assessment of Software Development Tools for Safety Critical Real Time Systems” Invited Paper in IFAC Workshop on Programmable Devices and Systems, Ostrava, Czech Republic, February 2003, pp. 2-7

Real Time Safety ERAU research conducted for the FAA under contract DTFA0301C00048 page 1 Assessment...

Documents

Transcript of Real Time Safety ERAU research conducted for the FAA under contract DTFA0301C00048 page 1 Assessment...