American Institute of Aeronautics and Astronautics

1

Challenges in Independent Verification and Validation of

Safety Critical Railway Signalling Systems

Sandeep Patalay1

CMC Americas, Inc., Pittsburgh, PA, 15220

A railway signalling system is safety critical system that controls the traffic which includes train routes,

shunting moves and the movements of all other railway vehicles in accordance with railway rules, regulations

and technological processes required for the operation of the railway system. The overall signalling system

consists of Microprocessor based Wayside controllers, On-board systems controlling the railway vehicle and

supervision systems to monitor the vehicle movements from a centralized location. The complex nature of

railway signalling rules and operational practices adopted by different railroads pose a difficult task for the

software development of these systems. The complex nature of the software poses an even more challenging

task during the Independent Verification and Validation of the system. The CENELEC set of standards is the

widely accepted as the governing standard for design, development and Independent Verification and

Validation (IV&V) of railway signalling systems. This paper describes the challenges faced during the

different phases of IV&V of safety critical railway signalling software which is unique compared to other

domains.

Nomenclature

IV&V = Independent Verification and Validation

ATP = Automatic Train Protection

On-Board = Embedded systems used on the Train

CENELEC = European standards for Railway Signalling

I. Introduction

IV&V is the most important phase of any safety critical system life cycle. The result of this phase decides the

final outcome of the project and decides whether the product is fit for use. The IV&V of safety critical software

for railway signalling applications is faced with many challenges due to the complexity of the systems and the

variations it has depending upon the geography and environment in which it needs to operate. This paper

particularly focuses on the experiences and challenges during different phases of the IV&V in a railway signalling

project. The following areas will be discussed:

1) Systematic Problems

2) Challenges during Software Analysis

3) Challenges during System Integration and Field Validation Testing

4) Challenges during Test Result Analysis

II. Systematic Problems during IV&V of Railway Signalling Software

The following systematic problems are experienced during the IV&V of safety critical software developed for

railway signalling applications:

5) Lack of formal methods in developing the control algorithms results in poor understanding of the system by

a test engineer.

6) Lack of domain knowledge in railway signalling systems creates a technological gap between the software

test engineers and the domain consultants. This leads to errors in software testing, which might lead to

unsafe failures being un-detected.

1 Senior Systems Engineer, Embedded Systems Group, CMC Americas, Inc., sandeep.patalay@cmc-americas.com.

T

American Institute of Aeronautics and Astronautics

2

7) Since the software and hardware is so complex, complete test of the system is not possible and most of the

faults are revealed at the field Installation stage or during normal working of the system in field.

8) The software is often changed for every geographical location and results in specific code for each location.

When the software structure is not in a generic form, it becomes difficult for the test engineer to develop

test cases for every possible scenario.

9) The lack of standardization in the railway working principles results in incomplete test cases as test

engineers are not well versed with all types of railroads.

10) Increase in the complexity of the software leads to difficulty in testing, since most of the railway systems

are sequential machines they are error prone and are very difficult to test.

III. Challenges during Software Analysis

The following section describes the challenges faced during Static and Dynamic analysis of safety critical

software developed for railway signalling applications:

1) Static Analysis of software is the analysis of the software code without actually executing it. The railway

signalling software particularly the vehicle braking algorithms are very complex and require the test

engineer to be well versed with the dynamics of the vehicle as well as good mathematical knowledge.

These algorithms require the test engineer to envisage all the possible states of the algorithms and then

create a formal model of the system. In many cases the lack of Test Engineer’s knowledge about these

algorithms results in insufficient test cases for the model and results in many of the errors revealing in the

latter part of the dynamic analysis.

2) Dynamic Analysis of software is the analysis of the software code by actually executing the software and

observing the executions. The dynamic analysis of the safety critical software is an important phase of the

Independent Verification and Validation of the system. The Test engineer should be well versed with the

domain of inputs and outputs of the system. In many cases the test engineer chooses the boundary values

based on the data range of the variable type. In the real world the boundary values depends on the actual

working environment of the system, for example the GPS signal boundary values received by the vehicle to

determine its position varies based on the geographical location of the railroad and is embedded in the

vehicle database. When the test engineer creates test cases for this part of the software it should be changed

based on the geographical location where the train is operating, instead the literal boundary value of the

variable type would pass the dynamic analysis and this error would only be revealed during field validation

tests.

3) Inexperienced test engineers just follow the rule book and often result in insufficient test scenarios. Testing

experience and intuition combined with knowledge and curiosity about the system under test may add some

uncategorized test cases to the designed test case set. Special values or combinations of values may be

error-prone. Some interesting test cases may be derived from inspection checklists.

4) Test engineers generally are not well versed with the concept of error seeding and do not try to measure the

effectiveness of their test cases. Some known error types should be inserted in the program, and the

program should be executed with the test cases under test conditions. If only some of the seeded errors are

found, the test case set is not adequate. The ratio of found seeded errors to the total number of seeded errors

is an estimate of the ratio of found real errors to total number errors. This gives a possibility of estimating

the number of remaining errors and thereby the remaining test effort.

5) Performance testing of the system in the lab environment is often inadequate, since the simulators are not

exactly replicating the field environment, this result in many errors being revealed during field validation

phase.

6) Test engineers often follow the concept of “Equivalence Classes and Input Partition Testing” to save time

and testing effort. This principle is often flawed due to the inexperience of the test engineer or insufficient

coverage of the data classes.

7) Lack of formal methods in developing software prevents the test engineer to take full advantage of the

“Structure Based Testing” concept where clearly defined states and modules are required for generating test

cases for complete coverage of the system.

American Institute of Aeronautics and Astronautics

3

IV. Challenges during System Integration and Field Validation Testing

The following section describes the challenges faced during System Integration testing of safety critical railway

signalling systems:

1) The System Integration testing should be ideally started after the unit/Module tests are successful passed,

but in reality due to the delayed nature of the railway signalling projects, the system integration tests are

carried out in parallel with the unit/module tests, this causes many problems to be revealed only at the

system level tests.

2) Many unexpected behaviors of the system are revealed at this stage since the software has not completely

gone through unit tests. This causes more delays in the system integration tests and changes in the

requirements are required since all the scenarios at the unit level have not been accounted.

3) The test engineers who have been devoted to find integration issues find themselves more involved in

sorting out the problems that should have been caught during unit tests.

4) The integration issues found during this phase often lead to design changes which are costly to fix and in-

turn increase the complexity of the system.

5) The Field Validation tests are executed in the actual field environment and many of these scenarios are not

accounted in the lab, so the same software has different behavior in the lab and field, this leads to confusion

and is hard for the developers to debug.

V. Challenges in Analysis of Test Results

The Railways signalling systems generally have large test scenarios to be performed in the field and lot of the

data collected during the tests require offline analysis, the below section describes the challenges and problems

associated with this phase.

1) In many railway projects, often the field engineers are recruited locally to ensure easy access to the test site,

often these field engineers are new to railway signalling and have little or no training to execute the tests.

2) The complex nature of the offline log analysis requires test analyst to be fully in synchronization with the

field engineer who executed the test, in many cases the lack of communication between the field and

offline test analysts result in falsely reporting the test as failed.

3) In many cases, due to some inherent errors in the test procedure, the log file analyst reports the test as failed

and goes through multiple cycles of test execution.

4) In On-board ATP log analysis, often it is required to check the braking profiles of the systems and requires

complete knowledge of the braking algorithms. In many cases the test engineer is not qualified to perform

this analysis and results in poorly reported analysis.

5) The lack of co-ordination between the test lead and the field technicians often result in incomplete tests and

later results in incomplete log analysis.

VI. Conclusion

Railway signalling is very specialized and unique area where high level of planning is required for all the phases

of the project lifecycle especially for the IV&V of safety critical software. Poor planning at the start of the project

usually result in cost overruns and delays. In our experience with railway signalling projects, generally limited

budgets and time is allocated to IV&V phase which in realty takes the majority of the project budget. If the IV&V

phase is planned well in advance and sufficient managerial responsibility is assigned specifically for this task, the

projects can be completed in time and with better results, which in turn makes the job of the safety assessor easy.

We suggest the following mitigation measures to ensure a successful IV&V of railways signalling systems:

1) Care should be taken to recruit test engineers who at least have basic knowledge of railway signalling and

associated systems.

2) In case the test engineers are new recruits, they should be put through rigorous training before being

assigned critical tasks such as writing test procedures and analyzing the test data logs.

3) Regular training sessions should be conducted for the test engineers in the project to impart in-depth

knowledge of the system.

American Institute of Aeronautics and Astronautics

4

4) Encourage test engineers to be innovative in their testing methods instead of just following the regular

patterns, this way more errors in the system are revealed which often get undetected with traditional test

methods.

5) Create an environment where test engineers regularly interact with the design team to share each ones

experiences and concerns

6) Create a dedicated managerial team to monitor all the test activities occurring a different sites and co-

ordinate them. Better co-ordination between the Lab and field test teams leads to better analysis of the

system.

7) Never follow the approach of parallel testing activities, for example, the system integration tests should

never be planned in parallel with the unit tests.

Acknowledgments

The author would like to express his gratitude to Stephen A. Jacklin from the NASA Ames Research Center for

his encouragement to take up this study and present my experiences with IV&V in railway signalling domain.

References

1S.Vinogradov, V.Okulevich, M.Gitsels, “Approaches to meet Certification Requirements for Mission-Critical Domains”,

Software Engineering Conference (Russia), 16th Nov. 2006 2Ulrich Haspel, Gunni S. Frederiksen., “The Automated Copenhagen Metro In The First Year Of Operation

- Experience And Outlook,” 9th International Conference On Automated People Movers 2-5 September 2003, Singapore 3K.K Bajpayee, “Emerging Trends in Signalling on Indian Railways” in IRSTE Conference, 2003 4Peter Wigger, “Experience with Safety Integrity Level (SIL) Allocation in Railway Applications”, WCRR 2001 – 25. – 29.

November 2001, Köln 5Dr. Hendrik Schäbe, “The Safety Philosophy Behind the CENELEC Railway Standards”, ESREL 2002, Lyon, March 19-21,

2002 6G.Biswas, S.Kumar,T.K.Ghoshal,V.Chandra, “Independent Verification and validation of Software with reference to UFSBI”,

presented at IRSTE Seminar, 1999. 7Chinnarao Mokkapati, Terry Tse, Alan Rao “A Practical Risk Assessment Methodology for Safety-Critical Train Control

Systems”, Office of Research and Development Washington, D.C. 20590, DOT/FRA/ORD-09/15 8EN 50126: Railway Applications - The Specification and Demonstration of Dependability, Reliability, Availability,

Maintainability and Safety (RAMS). Issue: March 2000. 9prEN 50129: Railway Applications- Communications, signalling and processing systems - Safety related electronic systems for

signalling. Issue: May 2002 10prEN 50128: Railway Applications- Communications, signalling and processing systems - Software for railway control and

protection systems. Issue: March 2001