1

TECHNIQUES FOR STRONG PARTITIONING OF VME-BASED MULTIPROCESSOR SYSTEMES

Mohamed F. Younis* Mohamed Aboutabl**

* Honeywell International Inc., Advanced Systems Technology Group 9140 Old Annapolis Road/MD 108, Columbia, MD 21045, USA

mohamed.younis@honeywell.com

** Wireless Network Group, Lucent Technologies 550 Clark Dr., Mt. Olive, NJ 07828

aboutabl@lucent.com

Abstract

The VME bus has been widely used in embedded systems due to its rigorous design, stable

specifications, and the fact that it has enjoyed a strong support from multiple vendors. However, the

VME bus still lacks some features that are essential to safety-critical systems, such as avionics. In

this paper, we address the issue of lacking the capability of fault containment in the current

specification of the VME bus and propose a strong partitioning protocol for inter-processor

communication for a VME-based system. This protocol enforces fault containment without any

modifications to the VME hardware. The design and implementation of a proof-of-concept prototype

is presented. Numerous software-based as well as hardware-based fault injection experiments have

been conducted to demonstrate the resilience of the system to any fault propagation attempts. The

performance of the strong partitioning protocol is evaluated and compared to alternative techniques

for multiprocessing on the VME bus. Finally, the benefits of this technology for a space launch

vehicle and commercial avionics are discussed.

Keywords: Fault Containment, Safety-Critical Systems, Distributed Processing, Inter-Processor

Communication, The VME Bus System, Integrated Modular Avionics.

1. Introduction

Recently, a new trend towards the use of commercial-of-the-shelf (COTS) solutions has emerged and

found a strong acceptance in the real-time embedded systems arena. The driving force behind this

2

trend was, and still is, lowering the production cost and time-to-market cycle associated with the

traditional proprietary designs. The COTS-based solutions decrease the development and

maintenance cost as a result of the mass production of components, the reduced levels of spares, and

the competition among different vendors.

Meanwhile, the avionics industry has endorsed a new approach for systems design, namely the

Integrated Modular Avionics (IMA) [1]. Multiple functions are integrated in a modular system thus

sharing common resources and promoting the reusability of existing subsystems. However, function

integration increases the risk of the failure of multiple functions due to a fault in one of the system

components. In an IMA environment, an application might share a resource with others and thus, its

correct operation becomes dependent on their use of the resource. When multiple avionics

application modules share a resource such as memory or a communication bus, partitioning is

particularly challenged in the way applications observe access protection and usage quota for that

resource. Therefore, access to shared resources needs to be carefully controlled in an IMA both

spatially and temporally.

Thus, the IMA approach inherently requires strong partitioning among the different modules

coexisting within the same system. Strong partitioning calls for well-defined boundaries among

modules to ensure the continuity of operation in the presence of partial failure [2]. Containing the

effects of faults is very crucial for the integrated environment to guarantee that a faulty component

may not cause other components to fail and risk generating a total system failure. Strong partitioning

also facilitates the modification, debugging, integration, and certification of the modules involved in

building a system with the level of complexity found in the avionics industry.

Using COTS components in the implementation of integrated modular avionics brings the

challenge of guaranteeing strong partitioning, a feature which COTS products usually are not

designed to have. Since COTS components will lose their cost advantage if they have to be modified

to fit in an IMA design, special techniques need to be used to maintain strong partitioning. This

paper presents techniques for spatially partitioning of COTS-based VME (VERSA Module

Eurocard) systems for integrated modular avionics. To the best of our knowledge obstacles for strong

partitioning of the VME bus have not been previously identified or addressed in the literature. While

strong partitioning generally covers the temporal and spatial aspects, in this paper we focus only on

3

spatial partitioning of the VME bus system, and thus our reference to strong partitioning in the

balance of the paper means only the spatial aspect of it.

In the following subsections, we discuss the background for our research work and describe the

fault model addressed by this paper. We also include a brief overview of the VME bus and shed the

light on its inability to stop faults in one module from affecting other healthy modules in a

multiprocessor setup. Section 2 presents techniques for strong partitioning of a VME system. The

implementation of these techniques in a test environment is described in section 3. Fault injection

experiments are discussed in section 4, followed by an analysis of performance measurements in

section 5. Section 6 discusses related work. Section 7 concludes the paper and discusses the benefits

of the work in current aerospace programs within Honeywell.

1.1 Motivation

The backplane bus is one of the most important components in integrated modular avionics. While

many backplane buses have been proposed, only a few are actually used. Selecting a backplane bus is

influenced by many design and engineering factors, such as performance, reliability, and fault-

tolerance. Although such issues are very important to ensure certain levels of safety of commercial

jet aircraft and high availability of military aircraft, the cost of the bus and associated modules is a

major concern.

Most of the currently available dependable backplane bus systems, such as ARINC 659 [3], are

expensive and supplied by very few vendors. It is clear that there is a need for an affordable bus

system that provides the required level of dependability and complies with the IMA design

methodology. The VME bus system [4] is a prime candidate because it is both rigorously defined and

widely supported. In addition, there is an expanding selection of VME bus modules (boards) and

vendors offering competitive prices and continuous support. Moreover, the VME bus provides an

open architecture that facilitates the integration of multiple vendors’ modules. Such a feature makes

the VME bus an attractive choice for integrated avionics on highly visible projects such as the

Venture Star space launch vehicle, supported by NASA for its 21st century space shuttle [5].

However, as discussed in Section 1.3, the VME bus architecture does not impose strong

partitioning. This paper focuses only on spatial partitioning of the VME system. The VME

4

architecture allows faults to propagate from one module to the other. For example, a faulty module

can perform a wild-write, which corrupts the shared memory area and disturbs the computation on

other modules [6]. Containing the effects of faults is very crucial for the integrated environment to

guarantee that a faulty component may not cause other components to fail and risk generating a total

system failure. For instance, in an ideal IMA-based avionics, a failure in the cabin temperature

control system must not negatively influence critical flight control systems such as the flight

manager. Lacking the capability to isolate faults limits the use of the VME bus in mission-critical

applications.

Fault containment is not the only capability that the VME bus lacks to be fault tolerant. There are

weaknesses in both fault detection and recovery as well. The bus structure, as explained in section

1.2, allows isolation of modules and even a total bus break down because of a failure in the

centralized bus controller [7]. While the VME bus lacks such fault detection and recovery

capabilities to be fault tolerant, the potential of fault propagation over the bus must be prevented to

minimize the possibility of complete system failure.

1.2 An Overview of The VME bus

The VME bus is multi-processor computer

architecture. It handles data transfer rates in

excess of 40 Mbytes/sec using parallel data

transfer. Although the origin of the term 'VME'

has never been formally defined, it is

commonly used to refer to VERSA Module

Eurocard standard which is a combination of

the Motorola VERSAbus electrical standard,

and the Eurocard mechanical form factor [4].

The VME bus is asynchronous and non-multi

used to coordinate data transfer. Every module (bo

interfacing with the bus. Data is passed between

where the slowest module participating in a cycle

in the VME bus provides reasonable capabilities to

System controller VME Backplane

Daisy-chained interrupt ack. and bus grant

Figure 1: The VME bus architecture

plexed. Because it is asynchronous no clocks are

ard) includes a VME chip that hosts the logic for

modules using interlocked handshaking signals

sets the cycle speed. Using asynchronous protocol

integrate products from various vendors.

5

The VME bus architecture is depicted in fig. 1. The left-most module plays the role of a system

controller that is responsible for arbitrating the bus usage among the various modules.

Communication between two modules follows a master-slave (sender-receiver) protocol. Before a

master (sender) can transfer data it must first acquire the bus via the system controller. The system

controller acts as a central bus arbiter that grants the bus according to prioritized requests handling or

simply using round robin. Bus grant signals are propagated back to the requester through a daisy

chain over all modules between the requester and the system controller, as shown in fig. 1.

There are seven levels of system interrupts supported by the VME bus. Although modules can

issue an interrupt on any of these levels, only one module should acknowledge and handle the

interrupt. The daisy chain again is used to propagate the acknowledgement signal from module to

module until it reaches the interrupter. The interrupter then identifies itself to the handler by placing

an identification word on the bus. Due to the daisy chaining used for bus mastership requests and

interrupts, modules are position-sensitive. The first slot is reserved for the system controller. The

closer the module to the system controller, the higher its privilege of getting the data bus and the

higher the priority of this module’s interrupts becomes.

In addition to the VME system interrupt, the VME standard supports a mechanism, called

location monitor interrupt, for module-to-module interrupt. The VME chip can be programmed to

generate an interrupt to the local processor whenever an address of a pre-defined set of addresses

appears on the VME bus. This feature is often used for event-notification among modules or for

supporting inter-processor communication through mail boxes (message queues). Location monitor

interrupts are completely local to the target module while the VME system interrupts are sensible by

all modules with only one module designated as a handler.

The VME bus provides support for multiprocessing using shared memory. To avoid

inconsistency while updating shared memory, read-modify-write bus cycles are used. The read-

modify-write cycle allows updating shared memory as an atomic transaction and prevents race

conditions.

6

Typically, in a VME system, a module has its

local memory map, which is accessible though the

module’s local bus. A system-wide VME memory

map is composed of all shared memory allocated on

all the modules. Every module can map part or all its

local memory to the VME address space. In

addition, a module can restrict access to its shared

memory area to read-only to make it functions as an

out-going buffer. The starting address in the VME

space is user-defined. Address translation is

performed by the VME interface chip on each

module with conflicts detected at initialization time.

Fig. 2 demonstrates the mapping of local memory to

the VME address space.

Although the VME bus does provide reasonable co

vendors, fast parallel data transfer and a wide produ

support for fault-tolerance in VME-based systems is ver

1.3 Fault Propagation Over the VME bus

The VME bus relies on all the connected modules fo

using a specific failure control line. VME bus modu

diagnostics to detect faults. In addition, the VME bus

transfer. If the slave (receiver) does not acknowled

retransmits. The bus provides neither error detection no

no redundancy in either the transmission lines or the tra

Generally, the built-in-test and transmission timin

absence of verification of transmitted data, faults on

healthy modules through inter-processor communicat

faulty module and can jeopardize the behavior of th

CPU 1 Localmemory

VME AddressSpace

0

Max. localRAM size

4 G

Max. localRAM size

0

CPU 2 Localmemory

Figure 2: Mapping local memory to

VME space

mpatibility to integrate products from various

cts-availability through many manufactures,

y limited.

r detecting faults and reporting these faults

les are expected to have on-board firmware

master (sender) monitors the time for data

ge the message, the master times out and

r correction for the transferred data. There is

nsferred data.

g-out provide limited fault coverage. In the

one module can manifest errors that affect

ion. Faults cannot be contained within the

e complete system. In addition, the shared

7

memory model cannot prevent a wild-write by a faulty module, which can corrupt the memory of

other healthy module [6]. Such a problem is usually referred to as fault propagation from one faulty

module to other healthy modules.

In addition to fault propagation problems, the VME bus system allows a single point of failure by

using a centralized mastership control of the bus. The system controller organizes all

communications among modules. Faults in the system controller can bring down the whole system.

Moreover, the reliability of daisy chain mechanism is questionable. Each module either accepts the

bus grant or passes it over to the next module. Failure of one module might break the chain and

affect other modules. Breaking the chain can prevent other modules from getting mastership of the

bus and from reacting to interrupts. Communication among modules can be highly affected and the

whole system can break down.

From the above discussion, we conclude that the VME bus needs enhancements to strengthen its

fault-tolerance capabilities, specifically in containing faults and recovery from a total bus failure. In

this paper we focus on fault containment. Generally, transient stuck failures in the bus lines and

modules are very common in aerospace applications. Electrostatic discharge caused by lightening or

air friction, electromagnetic inductance (EMI) when the bus is close to an EMI source like the engine

or a cooling fans and loose or dusty connectors are among the common causes for transient faults

over the bus. Statistics from Honeywell’s aerospace maintenance facilities indicate that 90% of the

reported failures in the bus and its modules are transient. Thus containing the effect of these

transient failures avoids the disturbance of other control functions while performing local recovery

within a faulty module or between two communicating modules to tolerate a transient stuck fault in

the VME bus. The following issues need to be addressed in order to improve the fault containment in

a VME bus system:

Validating the inter-module data transfer to detect transmission errors over the VME bus.

Preventing fault propagation from one module to others through the use of shared memory.

Section 2 provides a discussion of our approach to address these issues. We present techniques

for strong partitioning of multiprocessor applications that maintain fault containment on the VME

bus. The suggested techniques do not require any modification to the standard and utilize the existing

8

features of the VME modules, and consequently maintain the plug-and-play advantage of VME-

based hardware products.

While fault-tolerance is important for an IMA design, the recovery mechanism depends heavily

on the criticality of the function to the aircraft operation. For functions like weather radar, it might be

sufficient to manually switch to a backup unit. Meanwhile, a flight control unit has to be empowered

with an automatic recovery mechanism. Regardless of the criticality of the function, fault

containment has to be guaranteed. Failure in multiple non-critical control modules might be regarded

as a serious breakdown that endangers the safety of the flight.

It is worth noting that we use redundancy to address recovery from failure in the VME bus,

including those caused by the system controller or the daisy chain. A redundancy management

system [7,8,9,10,11] is used to tolerate bus failure caused by the bus controller or the daisy chain.

Fault containment is the focus of this paper and tolerance of bus failure is not further discussed.

1.4 Fault Model

Strong partitioning implies the fail silent model [12], that any erroneous behavior of a faulty

component should not affect other healthy components. This erroneous behavior can be the result of

a hardware or software fault regardless of the nature (generic, accidental or intentional) and the

duration (permanent, transient or intermittent) of that fault. However, we assume that there are

application-specific semantic checks that verify the validity of the communicated data to detect

errors due semantic-related generic faults in the application software. We also assume that the

system is not liable to a Byzantine-type fault [12]. In this paper, bus failure due to either bus damage,

stuck control lines or the failure of bus arbiter are not addressed. Only faults in modules and the data

transmission are considered. In addition, we assume that faults occur one at a time with no

simultaneity.

An attempt by a faulty component to corrupt other healthy system components should lead to an

error. Only system components, that communicate with that faulty component, need to be aware of

the error and perform recovery according to the nature of the application. On the other hand,

operations on healthy components that do not communicate with the faulty component will not be

affected. This paper covers only mechanisms for fault containment, assuming system-level recovery

9

of the faulty component is performed by other means such as a redundancy management system

[9,13].

2. Fault Containment Techniques

Since low cost is an important feature of the VME bus, enhancing the fault containment capabilities

should avoid changing the design and the layout of the currently available modules. Changing the

design of a VME module will not only require reengineering and revalidation which increases the

manufacturing cost, but will also again limit the number of vendors who agree to adopt such

modifications. Thus, the suggested approach should preserve the current hardware design of the

modules as much as possible. In the following subsections, we describe a message-based approach

for inter-processor data transfer that guarantees fault containment.

2.1 Inter-module Data Transfer

The VME bus features parallel data transfer between modules. There are no error detection or

correction bits associated with the transmitted data. Adding such bits will significantly affect the

VME module’s design and, therefore, is not an option. As an alternate approach, an error detection

code, e.g. cyclic redundancy check, can be appended to the end of the data similar to the integrity

check of multi-part messaging protocol [12]. A data transmission service within the operating system

kernel can generate this error detection code. Although the software-generated error detection code is

more time-consuming than the hardware-based implementation, the software approach requires no

module redesign. The receiver module should validate the error detection code before accepting the

received data. Using such information redundancy within the transferred data fits the multiprocessing

scheme proposed in the next subsection.

2.2 Strongly Partitioned Multiprocessing

Strong partitioning is one of the most important IMA requirements, which the VME bus lacks. The

VME bus supports multiprocessing by using a shared memory mechanism, which permits faults in

one module to affect other non-faulty modules by writing erroneous data to their memory [6]. To

avoid the potential of such wild-write problem, we suggest the use of a message passing mechanism

instead. While the use of message-based inter-module communications over the VME bus sounds

like a simple solution to potential fault propagation when using shared memory, supporting robust

messa

restric

immu

In

for ou

buffer

messa

If a se

its ow

sender

comm

Th

ID, re

detect

Read-Only MessageBuffer (3) Read

msg

Processor

(1) Write message

Sender Board Receiver Board

Processor

Ack

(4) Write msg ID

(5) Pollfor ack

(2) Notify

ge passing without imposing hardware changes to the bus is not easy. The main challenge is to

t the use of commonly available features of the VME system so that every module can be

ne to failures caused by erroneous behavior of other non-healthy modules.

our proposed approach, each module dedicates a segment of its local memory as a buffer area

tgoing data messages and acknowledgments of incoming messages, as illustrated in fig. 3. This

is the only memory segment globally accessible to other modules. In addition, access to a

ge/ack buffer is restricted to read-only for any module that does not physically host that buffer.

nder module wants to transfer data to a receiver module, the sender writes a message into the

n message buffer and notifies the receiver. The receiver then reads that message from the

’s buffer and acknowledges it. Defining a read-only VME global address window is a very

on feature on VME modules.

Figure 3: A fault containment communication protocol

DataSender ID Receiver ID Message ID Error Detection Code

Message Format

Figure 4: Message format for robust inter-module data transfer

10

e format of the inter-module message is shown in fig. 4. The message consists of the sender

ceiver ID, a unique message ID, data and error detection code. The sender performs error

ion encoding for the entire message. The receiver checks the contents of the message before

11

accepting it. The receiver can check the sender ID and receiver ID fields to verify its retrieval of the

right message. In addition, transmission errors can be detected or recovered using the information

redundancy in the form of the error detection code in the message

Synchronization of the execution of application tasks can be achieved either; by polling the

message buffer of the sender for the required message, or by notifying the receiver by generating an

interrupt1 as soon as a message is being written by the sender in the designated address. If polling is

used, the message ID can be useful to overcome race conditions if the receiver tried to read the

message before it is ready. This may occur if the VME bus has a higher priority than the local bus.

Atomic bus cycles would not avoid such race conditions since they do not prevent local write to the

shared memory and do support only atomic write to VME memory, an operation prohibited by our

approach. The sender’s message buffer can be partitioned so that, separate buffers are dedicated to

each recipient. Receivers can expect a unique location for their messages. The adopted application

execution-synchronization mechanism is a designer’s decision.

Using such communication technique, errors in the sender can be isolated and prevented from

propagating to the receiver because no module will be granted write access to the memory of others.

Errors in the message can be either in data, sender ID, receiver ID, message ID, or message format.

The receiver should be able to detect errors in the message body by validating the message format,

the error detection code, the sender ID and the receiver ID. The message ID can be checked to

guarantee the right message sequence and to prevent race conditions. Any error in the message

detected by the receiver will invalidate the entire message and a recovery action will be taken. An

addressing fault in the receiver that may get it to read from the wrong module or the wrong address

within the right module will invalidate the message format and/or the sender ID. Furthermore, the

mapping of each module’s message buffer to the global address space of the VME system can be set

up during system configuration such that an addressing error cannot change a valid global address

1 Can be implemented by using the address-monitoring feature provided by the VME bus.

12

into another valid address. Maintaining a suitable hamming distance among buffer addresses can

guard the system against permanent or transient stuck failure of one or more address bits. Thus, the

system will be functionally partitioned. Faults remain within the faulty module and will not affect

other modules. Table 1 summarizes potential errors and the detection mechanisms.

Table 1: A summary of errors and the detection mechanisms

Error Detection Mechanism

Reading from the wrong module invalid sender ID

Reading from non-existent module VME time out

Reading the wrong message invalid message ID

Transmission error invalid error detection code

Reading wrong address, correct module invalid message format

2.3 Applicability of the Approach to VME Hardware

As we discussed earlier, enhancing the fault containment capabilities and the support for strong

functional partitioning on the VME bus should not impose significant modifications to both the

hardware manufacture and the application developer. This section presents the feasibility of the

techniques illustrated in the previous section. After conducting a market survey, we found that the

VMEchip2 [14], from Motorola, and the SVC64 [15], from Tundra, are the most commonly used

VME bus interface chips on VME modules. We studied the applicability of these techniques in VME

modules that use these interface chips.

Considering both interface chips, the strong partitioning protocol can be totally implemented in

software. In fact, it is possible to use commercial-off-the-shelf (COTS) operating systems by

extending the kernel service to include message handling. The generation and validation of the error

detection code within the message can be included in the message handler. Thus, to demonstrate the

applicability of the fault containment techniques, it is sufficient to show the feasibility of partitioning

local memory and mapping the message buffer within the VME global address space.

Both the VMEchip2 and SVC64 provide capabilities for software configurable global addressing

of the on-board memory of a module. There are software configurable map decoders that can be used

to control the global addressing of the VME modules’ local memories. A module can map a portion

13

of its local memory to the global VME bus address space rendering this portion accessible to other

modules. The hosting module can further restrict access to such portion to read-only activities. Any

attempt to violate these access restrictions will generate a bus error at the intruder module.

To prevent the possibility of other modules reading by mistake from the wrong module, these

chips allow the use of different combinations for the most significant bits for the VME address of

each module so that the hamming distance will be more than 1 bit. Thus, we can guarantee that

modules cannot read from the wrong module unless there is more than one transmission error. Given

that the number of modules is limited (maximum 22 according to the IEEE standard [4]), it is

feasible to achieve a reasonable hamming distance among the modules. Assuming, for example, that

each module allocates up to 64K byte of message buffer, which requires 16 bit addressing (out of the

32 bit address bus size), it possible to maintain a hamming distance of 8 bits among module

addresses and to isolate up to 7 simultaneous errors. Reading from a non-existing module will be

timed out by the VME bus and can be detected. Errors in the least significant 16 bits of the address

can be detected by validating the messages. Reading from a different location within the sender’s

message buffer will contain neither the right format nor the correct message semantic.

The strong partitioning protocol has been implemented and tested on modules that use the

SVC64 chip. The next section discusses the implementation and test environment in more details.

3. Implementation and Test Environment

A proof-of-concept prototype has been built using COTS components, as shown in fig. 5. The

prototype includes a VME backplane that hosts six VME-171 PowerPC 80 MHz processor

modules from DY4 Systems Inc. VxWorks, a real-time operating system from WindRiver Systems

Inc., is installed and integrated with the hardware. The strong partitioning inter-processor

communication protocol has been implemented and integrated within VxWorks. Testing and fault

injection experiments have been performed. In addition, the performance of multiprocessing

communication using the protocol is measured and compared to other approaches. This section

illustrates the implementation of the strong partitioning protocol within this test environment.

Task A Task B

ITC Service

Message queue

Figure 6: Inter-task communication service

3.1

In

thr

an

ex

ser

qu

ca

co

mo

co

ba

int

mo

ac

qu

glo

bu

ou

Fi

gure 5: A proof-of-concept demonstration prototype

Software Architecture

a single processor system, one way for tasks to establish communication among each other is

ough the use of message queues. A message queue is an abstraction of a unidirectional channel,

d is typically identified by a “queue ID” or QID. Two communicating tasks use the same QID to

change messages. The delivery of the messages is handled by the Inter-Task Communication (ITC)

vice (typically part of the operating system library functions) which may maintain several message

eues depending on how many communication channels are open; see fig. 6. The ITC service takes

re of synchronizing the access to the message queue and enforcing mutual exclusion.

In order to achieve fault

ntainment within a faulty

dule, the inter-processor

mmunication (IPC) over a VME

ckplane is to follow the protocol

roduced in section 2. Across

dule communication is to be

complished through message

eues that are allocated in a

bally addressable memory

ffer where the sender will leave

tgoing messages for other

IPC daemon

Task A Task C

Processorqueue(1,2)

sendto B

Module 1

IPC daemon

Msg queue

Processorqueue(2,1)

sendto D

VMEbus

Task D Task B

ITC Service ITC Service

Module 2Daemon queue

Processorqueue(1,3)

To module 3

Location Monitor

Hit

LocMonQ

ACK

Figure 7: Implementation of the strong partitioning protocol

14

15

modules to read. Although a separate queue for every pair of communicating tasks can be allocated,

managing the message buffer becomes complicated if tasks are created dynamically. The message

buffer cannot be allowed to grow dynamically since there is a potential that the system runs out of

free memory space within the message buffer. Instead, queues are allocated to communicating

modules. For each (sender, receiver) permutation of modules, there is a processor queue which is

created within the global portion of the sender’s main memory to hold any messages sent to that

specific receiver by any task running on the sender module, as shown in fig. 7. Receivers have read-

only access to the message queues owned by other modules. The sizes of these queues depend on the

amount of available memory and the expected communication traffic density.

An independent task, called the IPC daemon, is created by the system at initialization to handle

inter-processor communication. The IPC daemon maintains all processor queues and performs

message sending and receiving across the VME bus. The IPC daemon can be invoked on demand

when there are requests in either the sending (daemon) or the receiving (LocMonQ) activity queue.

However, in our implementation, the IPC daemon is scheduled to run periodically to fit the real-time

nature of tasks in avionics applications [16].

Fig. 7 shows an example where task A on module

1 sends a message to task B on module 2. In fig. 8, the

sequence of events taking place throughout this

transaction is depicted. Task A first contacts the local

ITC service on module 1, which recognizes that the

receiving task is on an external module. Therefore,

the message is inserted in the IPC daemon queue. The

IPC daemon on module 1 moves the message to the

end of the outgoing processor queue associated with

module 2. Module 2’s IPC daemon is then notified

that it has a message waiting inside module 1. This

notification takes the form of generating a location

monitor interrupt at the recipient module, which is a

feature of the VME system. The corresponding

Write msg

Sender’s IPCdaemon

Notifyreceiver

Readmsg

Validatemsg

Write ACK

Wait forACK

Remove msg

Receiver’s IPCdaemon

Time

Figure 8: Sending a message

16

interrupt service routine at the recipient module records the sender’s module ID at the end of the

LocMonQ queue for later processing by the IPC daemon. When the IPC daemon on module 2 runs, it

fetches the message from module 1 and acknowledges it by writing the message ID in an

acknowledgment cell located inside the global buffer of module 2. After the IPC daemon of module

1 reads the acknowledgement, it then deletes the message from its processor queue. Meanwhile, the

IPC daemon on module 2 delivers the message to the local ITC service, which finally stores the

message in the queue, connected to Task B2. It is worth noting that the ITC service on each processor

still handles local inter-task communication. It contacts the IPC daemon only if the target receiver is

on a different module.

The IPC daemon is scheduled to run

periodically at a frequency determined by the

system designer who takes into consideration the

traffic volume and the real-time constraints of the

communicating tasks. On each invocation, the IPC

daemon checks for all pending acknowledgments,

fetches one or more incoming messages, and sends

one or more outgoing messages, as shown in fig. 9.

Checking for all pending acknowledgements clears

the way for sending new messages as early as pos

message per invocation improves the performance by

CPU cache. The number (N) of messages to be sent

determined by the system designer as part of a con

parameters of the system. Since the IPC daemon cann

that receiver acknowledges the previous message, a ty

queue on the module, i.e. sending a message to every re

2 This is a regular message queue maintained by the operating sys

Send N messages

Check ALL pendingacknowledgements

Receive incomingmessages

Figure 9: The IPC daemon cycle

sible. Sending and receiving more than one

making use of the code that is already in the

in each execution cycle of the IPC daemon is

figuration file that contains all the operating

ot send a message to a receiver module before

pical choice of N is the number of processor

ceiver module.

tem (VxWorks in our implementation).

17

It should be noted that the handling of inter-processor communication traffic could also be

implemented as a library of service calls, which may be invoked when needed. In our application, we

found the use of an IPC daemon task convenient in scheduling the CPU and bus traffic and efficient

in performance, as explained.

3.2 Handling of Communication Failures

There are several faults that may occur during inter-processor communication. A faulty module may

send bogus notifications to other modules or stop acknowledging received messages. A faulty

module may also send a message to the wrong receiver, due to an error in determining the module ID

of that receiver. Moreover, the VME bus itself may suffer from transient and/or permanent faults on

its data and/or address lines. To handle such errors, the protocol calls for using an error detection

code as well as the validation of the sender/receiver identity fields in the header of each incoming

message. In addition, the message’s sender times out when waiting too long for an

acknowledgement.

Although the strong partitioning protocol allows the detection of erroneous behavior of a faulty

module, there is an error scenario that might have negative impact on healthy modules and need to be

carefully considered in the implementation. Such scenario happens when the location monitor

interrupts are used for message notification. A faulty sender module might continuously interrupt

other healthy modules for non-existing messages, thus overloading the IPC daemon on the recipient

modules and degrading the response time for non-faulty messages. It is worth noting that this is a

typical problem in interrupt-based notification and is not a side effect of our protocol. However in

the VME system a subtle problem might be caused when a faulty module, say module m, notifies

another module i with faulty messages disguised under the identity (address) of a module k (different

from m).

To handle the above issue, the strong partitioning protocol keeps track of two classes of

communication failures by requiring each module to maintain two error counters for each of the

other modules it communicates with, representing its own communication experience with these

modules. The first counter, snd_err_cntk, records the history of failures sending messages to

module k, whereas the second counter, rcv_err_cntk, records the history of failures receiving

18

messages from module k. Each one of these counters is to be incremented whenever the

corresponding failure occurs. On the other hand, if a one-way communication activity with a module

is completely successful3, both of its error counters are reset to zero, thus completely forgiving that

module for any failures in the past. However, if an error counter keeps increasing till it reaches a

predetermined threshold, the destiny of the corresponding communication activity is determined as

follows:

• If snd_err_cntk reaches its threshold, all send activities to module k are immediately

suspended, until further notice as explained below.

• If rcv_err_cntk reaches its threshold, an IPC-to-IPC “health-checking” message is sent to the

IPC daemon on module k indicating the imminent suspension of future receive activities unless

module k successfully acknowledges this message. If no acknowledgement is received, future

receive activities from module k are suspended. The health-checking message will only be sent

if the send activity with module k has not been suspended.

In both cases, the one-way communication is resumed only if a communication on the opposite

direction succeeds, since this will reset the error counters back to zero. The forgiveness strategy

allows the distinction between transient and permanent faults. If successful communication could be

re-established with a module, it is considered as an indication of its recovery. It is out of the scope of

this protocol to address the readmission of a faulty module once the 2-way communication with that

module has been suspended.

3.3 Integration with the Operating System

Although inter-task communication within the same VME module can use shared memory or

message passing, in the current implementation we allow only message-passing for communication

among application tasks that do not coexist on the same module. The inter-processor communication

is provided as a transparent message-passing service to application programs. Thus, a task can write

3 A send activity is completely successful only when the sent message has been acknowledged.

19

a message to the queue of another task regardless the location of the recipient. To achieve such

transparency, the library for message handling within the operating system has been replaced with

slightly different library. The new library checks for the location of the queue and invokes the

original operating system library if the target queue is local (on the same module). If the queue is

external (located on a different module), the message will be routed to the message queue of the IPC

daemon for delivery to the destination queue.

The new library refers to a system configuration table that contains, for each message queue in

the multiprocessor system, the ID of the module that hosts this queue. The application developer

prepares the system configuration table after the allocation of tasks to various modules. Each module

reads this configuration table as a part of its initialization. In avionics tasks allocation and scheduling

are always performed off-line in order to pass the FAA certification or comply with the military

validation guidelines. Thus, preparing the configuration table off-line is not imposing restrictions on

the application developer. At run time the IPC daemon uses the system configuration table to resolve

the ID of the module, on which each message queue resides, so that it inserts the message in the

proper processor queue.

In our implementation, the IPC daemon was defined as a high priority user-task. However, the

protocol can be completely integrated within the operating system by defining the IPC daemon as a

system-task and augmenting the library of system calls with the IPC library functions.

3.4 Real-Time Issues

In avionics as well as other hard real-time systems, task and resource scheduling is in the core of the

development of such systems due to the stringent timing constraints that must be satisfied. Generally,

shared resources such as the data bus have a significant role in achieving timeliness. A delayed bus

access for example might cause a task to block for an extended period and thus might miss its

deadline. Generally, issues like interrupt latency, data transfer rate and bus latency need to be

considered while scheduling the VME bus. Using the strong partitioning protocol the system

integrator needs to schedule the IPC daemon as an integral part of the task and VME bus scheduling.

Increased bus traffic might mandate invoking the IPC daemon multiple times per frame (period) to

ensure timely message delivery of inter-processor messages. In addition the overhead of the IPC

daemon has be considered in the schedulability analysis [16] to ensure timeliness under worst-case

20

scenarios. While the use of the protocol can complicate the scheduling of messages, it simplifies the

scheduling of tasks. The IPC daemon acts as a virtual task for all inter-processor communication

decreasing the coupling between tasks and bus scheduling and thus simplifying schedulability

analysis. Timeliness was not considered in our experiments and the real-time issues are not further

discussed in the paper.

4. Fault Injection Experiments

As we have stated in section 1, the strong partitioning protocol is designed to contain certain faults

within the boundary of the faulty module(s). These faults are grouped into two sets:

1. Module faults that result in an attempt to corrupt and/or disrupt the communication among other

healthy modules.

2. Data transmission and addressing faults, resulting from either a faulty module or hardware faults

in the VME bus itself.

The protocol is not intended to handle Byzantine faults, faults in the bus control lines, or failure in

the bus arbitration circuitry.

In order to validate the protocol, we designed tw

software-based, and hardware-based experiments to

The software-based experiments covers the first set

protocol to detect and contain erroneous behavio

hardware faults in a module. Stuck faults in the data

FaultyBo ard

Attempt to WRITE

P ro tectio n

Server

M emo ry

Client

M emo ry

Bus ErrorBus Error

Figure 12: Memory protection violations

IPC Daemon

Client

IPC Daemon

Server

Faulty Board

Corrupted Messages

Figure 11: Trashing of bogus messages.

o types of fault injection experiments; namely

cover the two sets of faults mentioned above.

of faults and mainly challenges the ability of the

r that might have resulted from software or

and address lines are emulated in the hardware-

21

based experiments to verify the protocol’s ability to detect and contain errors due to permanent and

transient faults in the module bus interface or the VME bus lines themselves.

In the experiments we were keen to measure fault coverage but not latency. Since the time for

detecting most of covered faults depends on many factors such as the bus speed and usage pattern,

the module processing power, the frequency of invoking the IPC daemon, the number of open IPC

channels, message traffic, etc., measuring fault latency was not considered informative. In the

following subsections, we describe both types of experiments and discuss the results.

4.1 Software-Based Fault Injection

For the software fault injection experiments, we set up a client-server test application in which the

server is running on one module, while the client is running on another module. The server

continuously accepts messages with requests to perform arithmetic and logic operations from the

Figure 13: Detection of corrupted messages

22

client, and then sends back messages containing the results. All communication activities between

the server and the client follow the strong partitioning protocol. A third module (fault injector) has

been designated to run an erroneous application whose behavior interferes with the server and the

client.

The fault injector module randomly shows one of three types of erroneous behavior. First, the

faulty (fault injector) module would attempt to disrupt the client-server communication by sending

bogus messages (fig. 11). Second, the faulty module would attempt to corrupt the client and/or the

server by sending real messages with invalid data in them. This simulates either a faulty behavior in

the IPC daemon of some modules, or even a transmission error on the VME bus due to permanent

and/or transient electrical faults on the data lines. Finally, the faulty module would attempt to write

to the memory of the two healthy modules (fig. 12).

A total of 2000 fault injection experiments were performed. In each experiment, the fault injector

triggers one of the three mentioned erroneous scenarios. The selection of the injected fault type per

experiment as well as the time intervals between successive experiments was random using uniform

and exponential distributions, respectively. The healthy modules are set to log errors in inter-

processor messages and their error counter thresholds were set to a very large number to allow

experiments to run before shutting down communication with the fault injector module. Other than

memory protection violations made by the fault module, in all cases the client and server modules

were able to detect and prevent the intrusion attempt by the faulty module. Memory protection

violations impact only the fault injector module and therefore were not reported by the healthy

modules. Considering fault containment as a goal, we could achieve 100% coverage, although only

about 2/3 of injected faults could be detected by one of the communicating modules.

Fig. 13 shows a screen capture of the second experiment. While the client and server are

communicating, the faulty module sends a formatted (but corrupted) message to the server. The

server detects the receipt of a corrupted message by validating the error detection code (CRC), logs

such an error and displays the error message “Invalid CRC in message”.

23

4.2 Hardware-Based Fault Injection

Next, we configured the system to test the effect o

lines of the VME bus. We set up three modules s

requests to one of its neighbors, and, in the mean

coming from the other neighbor, as illustrated in fi

The most significant 8 bits of the 32-bit addr

ID, while the least-significant 24 bits represent a lo

hamming distance of 2 is maintained among the

modules’ message buffers.

One of the modules was attached to the VME

allowing us to forcibly change any of the VME bu

signals, as shown in fig. 15. For each of the tested

card for injecting hardware faults:

1. On the module side: Injecting faults on this

emulates what happens when the module is dri

the backplane are non-faulty.

2. On the bus side: This emulates faults in the b

system.

Client 0

Server 0

Clien

Serve

Module 0 Modu

Figure 14: Communication set-

Client 2

Server 2

t 1

r 1

Module 2le 1

up for hardware fault injection

f hardware faults in any of the data or the address

o that each one hosts a client application sending

time, also hosts a server application for requests

g. 14.

ess have been designated to represent the module

cal address inside the selected module. A minimal

modules while configuring the VME addresses

bus via an extender card that contained jumpers

s signal lines such as the data, address, or control

bus lines, there were two locations on the extender

side represents failure only in the module. This

ving the bus with a faulty signal while the lines on

us lines themselves and affects all modules in the

24

We started with module 1 and injected stuck-at-one (+5v) and stuck-at-zero (GND) errors on

each data and address line. The experiments were later repeated for each of the other two modules,

one module at a time. For every pin we repeated the fault injection three times per stuck fault type.

Two out of these three repetitions emulated transient faults, for which the stuck fault were removed

quickly, while in the third one the stuck fault was left for a longer period to emulate permanent

faults. The error counter thresholds were set to a small number to allow capturing the effect of

permanent fault without unreasonably extending the experiment time.

In the following subsections, we describe experiments’ outcome when faults were injected on the

address lines and then on the data lines of module 1. The section is concluded with a summary of the

results of the hardware-based fault injection experiments. As we mentioned at the beginning of this

section, no control lines were corrupted. Failure of bus control circuitry is considered a total system-

wide communication failure that cannot be contained by applying the strong partitioning protocol.

Other research work [8] has addressed tolerating these kinds of failures through redundancy.

4.2.1 Hardware Fault Injection on Address Lines

When errors were injected on the address lines of the module side of the extender board,

communication between module 0 and 2 was not affected. In fact, there was a situation where

communication with module 1 itself was not disrupted. When injecting faults in the module ID

portion of the address lines, the observed results depended on the relationship between the stuck

value of bit and the bit’s value in the module ID.

Bus Extender Board

Figure 15: Hardware-based fault injection setup

25

• Value of stuck bit = the bit value in the ID of module 1

When the value of the corresponding bit in the ID of module 0 and/or module 2 has the same

value of the stuck bit, communication with module 1 continues without a problem. On the

other hand, if the value is different from the stuck bit, module 1 could not access module 0

and/or module 2 although it continued to be accessible to them. Module 1 reported errors while

notifying or reading messages from such module(s). The communicating module(s) reported

errors too while looking for acknowledgement from module 1. Since module 1 became unable

to access the message buffer of module 0 and/or module 2, it could not read messages sent by

either or both of them and thus did not acknowledge.

• Value of stuck bit ≠ the bit value in the ID of module 1

Module 1 became inaccessible to both modules 0 and 2. Modules 0 and 2 reported bus errors

during notifying module 1 of a message, reading a message and looking for acknowledgement

from module 1 since the VME bus circuitry considered this as an attempt to access a non-

existing module.

When faults were injected on the least significant 24 bits of the module side, one of the following

errors occurred:

a) Message notifications going out of module 1 missed their target. Module 1 timed out waiting for

an acknowledgement to arrive from another module that never got the notification in the first

place.

b) An attempt to access a non-existing memory region was made. Module 1 received a memory

protection violation error.

c) A message with the wrong format was fetched. Module 1 identified the received message as

invalid and completely ignored it.

d) A properly formatted message intended for a different receiver was fetched. Module 1 recognized

the received message as invalid and ignored it.

When the address lines were corrupted on the bus side, the three modules had a bus error most of

the time. If the corrupted address line was a part of the module ID portion of the address, there were

some cases in which the system continued without a failure or with a partial failure. When an address

26

line was stuck on a value that was consistent with the ID of two or more modules, communication

between these modules continued. For example, when we assigned the modules ID to be F0, 80 and

60 for modules 0,1 and 2 respectively, and injected a stuck at 1 error on the most significant address

line, communication between the first and second modules was not affected. Corruption of any the

least significant 24 address bits consistently led to a bus error on all modules.

4.2.2 Hardware Fault Injection on Data Lines

Next, we injected errors in the data lines. We enforced stuck-at-1 and stuck-at-0 faults on each data

line, one at a time. When injecting the fault at the bus side of the extender card, all the three modules

encountered a CRC validation error. In the case of injecting the fault at the module side, all messages

involving module 1 were rejected as they failed to pass the CRC validation at the receiving module.

4.2.3 Summary of Experiments’ Results

We ran 3 experiments per line per type of stuck faults. Therefore for each module a total of 384 fault

injection experiments were conducted to cover the 64 data and address interface lines to the VME

bus (64 lines × 3 experiments × 2 types). Similarly a total of 384 experiments involved the physical

bus address and data lines. Considering the bus and all three modules, a total of 1456 experiments

were conducted using the described setup, of which 1/3 were permanent and 2/3 were transient

faults. In all fault-injection experiments that we conducted, communication between the two healthy

modules was never interrupted unless we injected a fault on the bus side. The system recovered from

all transient faults. Fault coverage from fault containment point-of-view was 100%. However not all

faults were detected. Only 35% of transient faults injected at the module’s bus interface were

detected, while 68% of transient bus errors were detected. Given the extensive message traffic

included in the experiment setup, 91% of transient faults injected on the data lines of the bus and

53% of stuck faults on the data lines of the module interface to the bus were detected. Less detection

rate were observed for transient faults on address lines since only a subset of the VME address space

is used by the three modules in our experiment. Most of permanent faults were detected. Some of the

permanent faults injected on address lines went undetected because they affected lines that are

always zero or one. Validation of the CRC appended to the IPC messages constantly detected

permanent stuck faults on the data bus. Detailed results are shown in table 2.

5. P

In this s

the stro

mechan

empiric

perform

5.1 M

A prof

perform

source

of these

a RAM

instrum

conside

For

4.1. Th

header

total siz

Sinc

we tim

MBu

Table 2: Summary of the results of the hardware-based fault-injection experiments

Address lines Data lines

27

erformance Analysis

ection we describe a set of experiments, which we performed to measure the performance of

ng partitioning protocol. The results were compared to a commercially available alternative

ism for multiprocessing. An upper bound on the data transfer rate across the VME bus was

ally extracted and used as a reference point. We first describe the measurement tools and the

ance metric that we used in the experiments.

easurement Tools and Performance Metric

iling tool called WindView, supplied by WindRiver Inc., was used to monitor the

ance of the strong partitioning protocol. This tool enabled us to mark certain points in the

code as software events. As the test application was running on the target modules, each one

events was time stamped with a resolution of 1 microsecond, and the readings were saved in

file. The readings were later downloaded off-line to the host computer for analysis. The

entation of each event was associated with a 5-microsecond overhead that was taken into

ration while interpreting the results.

performance measurements, we used the same client-server application discussed in section

e client and the server exchanged fixed-size messages of 256 bytes each. A 32-byte message

(sender ID, receiver ID, etc.) was attached by the IPC daemon at each module, bringing the

e of the message travelling across the VME bus to 288 bytes.

e the application programs were running on separate modules without clock synchronization,

e-stamped events on each module independently and measured the elapsed time between

Transient Permanent Transient PermanentTotal Detected Total Detected Total Detected Total Detected

odules 384 67 192 160 384 204 192 192

s 128 56 64 60 128 117 64 64

28

events. Since both the server and client programs were running on exactly similar modules, we did

not expect a need to perform any calibration for the speed of the processors. However, a test was

performed using a logic analyzer to ensure that. The test involves executing a small program that

generates a bus request periodically in a loop and measuring the time between successive requests

through connecting the bus request line to the logic analyzer. The test experiment was repeated for

the modules involved and similar timing results were observed, confirming that no calibration is

necessary.

5.2 Maximum Measurable Transfer Rate over the VME bus

In order to obtain an upper bound on the most efficient data transfer rate across the VME bus, we

measured the time required for an assembly language routine to perform a memory write of 256

bytes from one module to the other. The time for writing 256 bytes was measured using a logic

analyzer and found to be 144 microseconds. This experiment provided us with an upper bound for

the bus transfer rate, and was later used to validate the accuracy of the measurements obtained using

WindView. It should be noted that this upper bound could be only achieved using optimized

assembly code.

5.3 Measurement for the Strong Partitioning Protocol

We next measured the end-to-end delivery time of the 256 bytes data, including the message header,

when the strong partitioning protocol was applied. We recorded 50 different readings for the send

and receive phases of the message delivery. Table 3 shows the average of these 50 readings. It should

be noted that the reported time for message validation includes the consumed time for checking the

message header and verifying the CRC code. The time for putting the message in the processor

queue includes the time for forming the message header and appending the CRC code. The average

end-to-end delivery time was found to be 302.3 microseconds ( send 92.4 + receive 209.9 ). It should

be noted that the actual data transfer across the VME bus is performed during the receive phase.

During the experiment we observed significant variance in the timing of first execution of the

IPC daemon and in message-handling library of VxWorks. Analysis of the timing behavior of the

IPC daemon pointed out the dominant contribution of a cache miss to the increased execution time of

first invocation of the IPC daemon. Excluding the reading related to the first time execution of the

29

IPC daemon led to a very negligible variance. This conclusion was further validated by repeating the

experiment while disabling the instruction cache of the PowerPC processor. The variability of the

message-handling library of VxWorks was persistent and could only be minimized by disabling the

cache. With that analysis of the variance the 50 readings were considered sufficient.

Table 3: Performance measurements for the partitioning protocol

Action Description Duration in µµµµsec Receive One Message

Handle location monitor interrupt 37.6 Fetch message across the VME bus 170.0 Validate and acknowledge message 2.3

Total for receive 209.9 Send One Message

Get message from daemon queue 49.1 Put message in processor queue 42.1 Notify receiver 1.2

Total for Send 92.4

5.4 Measurement for VxMP Shared Memory Protocol

In this experiment, we measured the performance of the shared memory approach applied by

VxMP, which is a multiprocessor extension of the VxWorks operating system. VxMP provides

an abstraction of shared memory so that applications can refer to the shared object by name. VxMP

only resolves the addresses for the programmer without maintaining mutual exclusion for accessing

the shared memory relying on hardware support for that. Obviously VxMP provides only shared

memory management without fault containment. It should be noted that VxMP requires the shared

memory to be physically located on a separate module by itself. Thus transferring data between

modules usually requires two bus transmissions; first the sender writes to the shared memory, next

the receiver reads that data from the shared memory.

The client-server application has been rewritten using VxMP to exchange 256-byte. Table 4

shows the performance results for sharing 256 bytes data. The end-to-end time for sending 256 byte

using VxMP is 433 µ sec (217.6 + 215.4).

30

Table 4: Performance measurements for VxMP

Action Description Duration in µµµµsec Receive One Message

Handle location monitor interrupt 30.4 Read message across the VME bus 187.2

Total for receive 217.6 Send One Message

Put message into shared memory 213.4 Notify receiver 2.0

Total for Send 215.4

5.5 Analysis of the Results

Comparing the end-to-end delivery time in both tables 2 and 3, it is clear that the strong partitioning

protocol does not only guarantee fault containment, but it also out-performs VxMP. The protocol

incurred a reasonable performance penalty relative to the upper bound exhibited by the experiment in

section 5.2.

It should be noted that the current implementation of the protocol is a proof-of-concept. There

are many opportunities for optimization, which significantly enhance the performance. In addition,

we noticed that significant performance improvement could be achieved if multiple messages are

sent or received in one invocation of the IPC daemon. Since most of the advanced processors, such

as the PowerPC, have instructions cache, the daemon code is usually fetched to this cache upon

activation of the daemon. Thus, the first send or receive operation always suffers a cache miss. We

observed a 35% improvement in the message handling, not including the time for data transfer over

the VME bus, after the first send or receive activity.

6. Related Work

This paper proposes a mechanism, which can be fully implemented in software and integrated with

the operating system, to support message passing on shared memory multiprocessor architecture. On

the contrary, there is plenty of work on supporting shared memory on a loosely coupled system,

which is referred to as distributed shared virtual memory [17]. Examples of this include the

TreadMarks system [18] and Mirage [19]. A survey for different approaches can be found in [20].

To support recovery from a failure on distributed shared virtual memory system; a checkpointing

mechanism is often introduced [21,22]. During normal fault-free operation, snapshots of the system

31

state (registers and memory contents) are stored in a repository to assist the applications recover from

a failure. The reader is referred [23] to for a survey of various recovery techniques and strategies.

Achieving fault containment in shared memory multiprocessor systems has been the subject of

intense study at the Computer System laboratory at Stanford for the FLASH multiprocessor

architecture [6,13,24]. The approach taken is to partition the multiprocessors into cells. Every cell

controls a portion of the hardware and shared resources. A cell may include one or multiple

processors. Each cell runs an independent multiprocessor operating system. A hardware solution has

been suggested to protect a cell from a wild-write by any faulty cell. The operating system

continuously runs across-cell checks for failure detection and a distributed recovery routine needs to

be executed upon the detection of a fault. Our approach is different in that we rely on the memory

management capabilities of the VME bus interface chip to protect modules from wild-write, since we

cannot introduce any hardware change. We prevent across-module writing and consequently restrict

inter-processor communication only to massage passing and do not support memory sharing.

However, we do not see that as a drawback since virtual memory is usually not recommended in

mission critical real-time applications due to its negative impact on system predictability [16]. In

addition, our fault detection is quite simple. We rely on timing out messages, a feature, which is built

in the VME bus interface logic. Furthermore, the strong partitioning protocol, presented above, is

easy to implement and integrate with COTS single processor operating systems.

Fault containment has been addressed by design in [10,25,26,27]. In the MAFT architecture [10]

and MARS [27], fault containment is channel-based. A channel includes a processor, memory and

redundancy management module. Communication among channels is message based. Across

channel voting and synchronization are used to detect, isolate and tolerate faults in any of the

channels. On the other hand, the software implemented fault tolerant (SIFT) architecture allows the

use of shared memory for inter-processor communication [26]. The SIFT approach calls for the use

of a tailored data bus that allows a processor to read and write to its own memory while giving it

read-only access to other memories. In our case the VME data bus could not be altered and

techniques to enforce memory partitioning had to be developed. In [25] a dedicated dual port

memory is included for the exclusive use of each pair of processors to communicate with each other.

The memory is empowered with error detection/correction code to detect stuck failures. Recovery of

permanent faults in memory is performed by making the affected pair of processors to communicate

32

through a third processor. Again, in our case supporting strict spatial partitioning while allowing

inter-procssor communication has to be implemented without changing the VME standard.

A different definition of fault containment from ours has been used in [28,29]. Containing faults

meant minimizing rather than eliminating the effect of a component failure on other healthy

components. The work mainly accelerates failure recovery using self-stabilization protocols, which

are algorithm-based approach to achieving fault-tolerance. The technique is to minimize the number

of components affected by a faulty component so that the self-stabilization protocol not only

guarantees convergence to a recovery but also expedites that recovery and thus limiting the impact of

the failure on the system.

7. Conclusion

While achieving fault containment in systems designed from scratch has been shown in the literature,

COTS systems bring a unique challenge since they are usually developed without considering fault

containment as a requirement. Therefore, any approach to preventing fault propagation from a faulty

module to healthy module in a COTS system, such as the VME, is constrained by its present design

that could not be changed. In this paper, it has been shown that the VME bus can be strongly

partitioned without imposing changes to the VME hardware. A multiprocessor message passing

communication protocol has been presented. The message passing protocol provides fault

containment over the bus. Messages can be verified with respect to sender and receiver IDs as well

as the message ID. In addition, information redundancy in the form of error detection code can be

provided within the message to verify data transmission over the bus.

The strong partitioning protocol has been implemented in software without imposing any

hardware modifications. A proof-of-concept prototype has been built, in which the strong

partitioning protocol has been integrated with COTS real-time operating system. The protocol has

been added as an operating system service, which is transparent to programmers. The fault injection

experiments demonstrated the efficiency of the approach in fault containment.

Although the increased data transfer due to message overhead and information redundancy will

affect the performance of the VME bus, it has been shown that the burden on the bus bandwidth is

33

limited. Clearly the longer the data size within the message, the more efficient the strong partitioning

protocol.

The technology presented in this paper is crucial for many dependable multiprocessor space and

avionics programs. For instance, at Honeywell, the technology has been applied to the vehicle

management computer (VMC) for an unmanned space shuttle. The VMC integrates multiple mission

critical control modules such as mission manager, flight control and vehicle subsystems manager

with other less critical modules within a VME cabinet. It is extremely important to contain any fault

within the faulty module and protect other modules in the system. The integration of utility control

system is another project that benefits from this technology. The goal is to integrate multiple utility

control units, which are currently federated on the aircraft, into one cabinet. Utility controls such as

electric power system, cabinet pressure and bleed air system are regarded as medium-criticality

control functions on the aircraft. However, integrating multiple utility control modules makes the

integrated unit highly critical since it may affect the safety of the flight. The strong partitioning

protocol is to be used for handling inter-module communication to prevent fault propagation and

protect the system from a complete failure when one module fails.

Acknowledgement

Thanks are due to Billy He, Eric Burton, Tim Steele and Chris Dailey for their help in the

implementation. The authors are also indebted to Dr. Yang-Hang Lee, Dr. Dar-Tzeng Peng and

members of the reliable systems group at Honeywell for their constructive critiques. A shorter

version of this paper appeared in the proceedings of the 1998 IEEE Fault Tolerant Computing

Symposium.

References

[1] Design guide for integrated modular avionics: ARINC report 651 (Annapolis, MD: Aeronautical Radio Inc., 1991).

[2] J. Rushby, Partitioning in avionics architecture: requirements, mechanisms and assurance,

Technical Report CR-1999-209347, NASA, 1999. [3] Backplane data bus: ARINC specification 659 (Annapolis, MD: Aeronautical Radio Inc.,

1993).

34

[4] IEEE standard for a versatile backplane bus: VME bus std 1014-1987 (New York, NY: The Institute of Electrical and Electronics Engineers, 1988).

[5] J. Keller, Avionics innovation marks new space shuttle, Military & Aerospace Electronics,

8(4), 1997, 1-7. [6] J. Chapin, et al., Hive: Fault containment for shared-memory multiprocessors, Proc. 15th ACM

Symposium on Operating Systems Principles (SOSP-15), 1995,15-25. [7] M. Younis and J. Zhou, A VME-based fault tolerant architecture, Technical Report, Honeywell

International, Inc., 1997. [8] O. Sokolsky, M. Younis, H.H. Kwak, I. Lee, & J. Zhou, Verification of the redundancy

management system for space launch vehicle: a case study, Proc. 4th IEEE Real-Time Technology and Applications Symposium (RTAS’98), Denver, Colorado, 1998, 220-229.

[9] R. Kieckhafer, C. Walter, A. Finn, & P. Thambidurai, The MAFT architecture for distributed

fault tolerance, IEEE Transactions on Computers, 1988, 398-405. [10] P. Thambidurai, A.M. Finn, R.M. Kieckhafer, & C.J. Walter, Clock synchronization in MAFT,

Proc. IEEE 19th International Symposium on Fault-Tolerant Computing, 1989,142-149. [11] J. Zhou, "Design capture for system dependability," Proc. of Complex Systems Engineering

Synthesis and Assessment Workshop, Silver Spring, Maryland, 1992, 107-119. [12] D. Pradhan, Fault-tolerant computer system design (Englewood Cliffs, NJ: Prentice-Hall,

1996). [13] M. Rosenblum, et al., Implementing efficient fault containment for multiprocessors: confining

faults in a shared-memory multiprocessor environment, Communications of the ACM, 39(9), 1996, 52-61.

[14] Motorola MVME162LX embedded controller programmer's reference guide, (Tempe, AZ:

Motorola, Inc., 1993). [15] Tundra Universe SCV64 Trooper II VME bus interface component manual (Ontario, Canada:

Tundra, 1996). [16] W. Halang & A. Stoyenko, Constructing predictable real-time systems (Boston-Dordrecht-

London: Kluwer Academic Publishers, 1991). [17] K. Li & P. Hudak, Memory coherence in shared virtual memory systems, ACM Transactions

on computer Systems, 7(11), 1989, 321-357. [18] C. Amza, et al. TreadMarks: Shared memory computing on networks of workstations, IEEE

Computer, 29(2), pp. 18-28, February 1996.

35

[19] B. Fleisch, & G. Popek, Mirage: A coherent distributed shared memory design, Proc. of the 12th ACM Symposium on Operating Systems Principles, published in Operating Systems Review 23(5) Special Issue, 1989, 211-223.

[20] M. Stumm and S. Zhou, “Algorithms implementing distributed shared memory”, IEEE

Computer, 23(5), 1990, 54-64. [21] B. Fleisch, “Reliable distributed shared memory”, Proc. 2nd Workshop on Experimental

Distributed Systems, 1990, 102-105. [22] M. Stumm, & S. Zhou, Fault tolerant distributed shared memory algorithms, Proc. 2nd IEEE

Symposium on Parallel and Distributed Processing, 1990, 719-724. [23] C. Morin & L. Puaut, A Survey of recoverable distribute shared virtual memory systems, IEEE

Transactions on Parallel and Distributed Systems, 8(9), 1997, 959-969. [24] D. Teodosiu, et al., Hardware fault containment in scalable shared-memory multiprocessors,

Proc. 24th ACM International Symposium on Computer Architecture (ISCA-24), 1997, 73-84. [25] G. Khan, “Fault-tolerant architecture for high performance embedded system applications”,

Proc. IEEE Conference on Computer Design: VLSI in Computers & Processors, Austin, Texas, 1998, 384-389.

[26] J. Wensley, et. al., SIFT: Design and analysis of a fault-tolerant computer for aircraft control,

Proceedings of the IEEE, 66(10), 1978,1240-1255. [27] H. Kopetz, et. al. Distributed fault-tolerant real-time systems: the MARS approach, IEEE

Micro, 9(1), 1989, 25-40. [28] S Ghosh, A Gupta, T Herman, & SV Pemmaraju, Fault-containing self-stabilizing algorithms,

Proc. 5th Annual ACM Symposium on Principles of Distributed Computing (PODC96), 1996, 45-54.

[29] S Ghosh, & SV Pemmaraju, Tradeoffs in fault-containing self-stabilization, Proc. 3rd

Workshop on Self-Stabilizing Systems, 1997, 157-169. Biography

Mohamed F. Younis received B.S. degree in computer science and M.S. in engineering mathematics

from Alexandria University in Egypt in 1987 and 1992, respectively. In 1996, he received his Ph.D.

in computer science from New Jersey Institute of Technology.

36

Dr. Younis is currently a research scientist with the Honeywell Advanced Systems Technology

Group, where he is leading multiple projects on architectural support for integrated modular

avionics, addressing issues related to fault-tolerant and real-time computing and software integration

in distributed processing environment. He participated in developing the Redundancy Management

System for the Vehicle and Mission Computer of NASA’s X-33 space launch vehicle. Dr. Younis’

technical achievements at Honeywell are widely recognized within and outside the company. His

technical interests include fault tolerant computing, system integration, distributed real-time and

complex systems and compile-time analysis. Dr. Younis has served different committees and has

numerous publications in his research field.

Mohamed Aboutabl