A UNIFIED TEST MODEL A UNIFIED TEST MODEL FOR HARDWARE AND FOR HARDWARE AND SOFTWARE SYSTEMSSOFTWARE SYSTEMS

József SZIRAYJózsef SZIRAY

Department of InformaticsDepartment of Informatics

Széchenyi UniversitySzéchenyi University

Egyetem tér 1, H-9026 Egyetem tér 1, H-9026 Győr, HungaryGyőr, Hungary

I. INTRODUCTIONI. INTRODUCTION

Reliable operation and application of Reliable operation and application of hardware and software systems impose hardware and software systems impose rigorous requirements for their rigorous requirements for their development. The hardware and software development. The hardware and software have to undergo some specific verification have to undergo some specific verification and validation procedures.and validation procedures.

The task ofThe task of verificationverification:: To check the To check the consistence between the individual consistence between the individual development phases. development phases. ValidationValidation is meant is meant for the checking of the system whether it for the checking of the system whether it conforms to the user requirements.conforms to the user requirements.

Adequate performance of Adequate performance of verification and validation implies verification and validation implies the following advantages:the following advantages:

It helps in deciding whether the next development phase can be started.

It may reveal errors in an early period of the development.

It provides data and information for the quality and reliability in any of the phases.

It may point out at an early state that the hardware and software do not meet the requirements.

All this has to involve previously planned checking, as All this has to involve previously planned checking, as well as intensive testing of the hardware and software well as intensive testing of the hardware and software in each phase. Testing is an integral part of both in each phase. Testing is an integral part of both verification and validation. verification and validation.

Here emphasis is placed on the so-called Here emphasis is placed on the so-called safety-criticalsafety-critical computer systemscomputer systems where the most important criterion where the most important criterion is safe operation, without endangering life and great is safe operation, without endangering life and great values. values.

This type of computing system is taken into account This type of computing system is taken into account here, since it needs the most thoroughly planned and here, since it needs the most thoroughly planned and performed procedures, in comparison with other performed procedures, in comparison with other systems.systems.

II. FAULT MODEL AND II. FAULT MODEL AND BASIC NOTIONSBASIC NOTIONS

A. Hardware faults:A. Hardware faults:1)1) SpecificationSpecification faults (user faults, external faults): Faults which faults (user faults, external faults): Faults which

occur at the beginning of the development cycle, and manifest occur at the beginning of the development cycle, and manifest themselves in the malfunctioning of the hardware, by not themselves in the malfunctioning of the hardware, by not meeting the real user requirements.meeting the real user requirements.

2)2) Design faultsDesign faults: These faults occur in the course of designing the : These faults occur in the course of designing the hardware itself. It means the erroneous or incomplete design hardware itself. It means the erroneous or incomplete design implementation of the specification. E. g., erroneous logic implementation of the specification. E. g., erroneous logic design.design.

3)3) Operational faultsOperational faults: Faults which originate from the : Faults which originate from the manufacturing process, as well as occur in the course of normal manufacturing process, as well as occur in the course of normal functioning, when the hardware itself goes wrong.functioning, when the hardware itself goes wrong.

– Stuck-at-constant logic level faults.Stuck-at-constant logic level faults.– Bridging (coupling) faults.Bridging (coupling) faults.– Behavioral (functional) faults.Behavioral (functional) faults.– Switch-level (transistor) faults.Switch-level (transistor) faults.– Timing (delay) faults.Timing (delay) faults.

B. Software faults:B. Software faults:The fundamental feature of software faults is that they The fundamental feature of software faults is that they are present all the time during operation. The question are present all the time during operation. The question is how their effect manifests itself in various situations. is how their effect manifests itself in various situations. There are two classes of software faults:There are two classes of software faults:

1)1) Specification faultsSpecification faults (user faults, external faults, design (user faults, external faults, design faults): Faults which occur at the beginning of the faults): Faults which occur at the beginning of the development cycle, and manifest themselves in the development cycle, and manifest themselves in the malfunctioning of the software, by not meeting the real malfunctioning of the software, by not meeting the real user requirements.user requirements.

2)2) Programming faultsProgramming faults (development faults, internal (development faults, internal faults, implementation faults): It involves the wide group faults, implementation faults): It involves the wide group of faults that originate from the mistakes made by the of faults that originate from the mistakes made by the programmers in the course of design and coding of the programmers in the course of design and coding of the software.software.

Programming faultsProgramming faultsPossible types of faults:Possible types of faults:

Erroneous performance of a Erroneous performance of a

function.function.

Missing functions.Missing functions.

Data-management errors in Data-management errors in

accessing the data base.accessing the data base.

Starting and termination Starting and termination

faults.faults.

Faults in the user interface.Faults in the user interface.

Underflow or overflow of Underflow or overflow of

boundary values.boundary values.

Coding error.Coding error.

Algorithmic error.Algorithmic error.

Initialization error.Initialization error.

Error in the control flow.Error in the control flow.

Data transmission error.Data transmission error.

Input-Output errorsInput-Output errors

Race among program Race among program

blocks.blocks.

Workload error.Workload error.

III. THE VERIFICATION AND III. THE VERIFICATION AND VALIDATION MODELVALIDATION MODEL

A. The mapping model for hardware:A. The mapping model for hardware:  A hardware system can be considered as an entity which A hardware system can be considered as an entity which

maps an input set to an output setmaps an input set to an output set. For the sake of . For the sake of simplicity, combinations and sequences of inputs will be simplicity, combinations and sequences of inputs will be considered as a single input. The response to these inputs considered as a single input. The response to these inputs is made by producing an output or a set of outputs.is made by producing an output or a set of outputs.

Let the set of all possible inputs to the hardware under Let the set of all possible inputs to the hardware under consideration be denoted by consideration be denoted by HWINPDHWINPD ( (input domaininput domain), ), and the set of all possible output responses by and the set of all possible output responses by HWOUTDHWOUTD ((output domainoutput domain). Now the one-to-one mapping of ). Now the one-to-one mapping of HWINPD into HWOUTD by the hardware system will be HWINPD into HWOUTD by the hardware system will be defined in the following form:defined in the following form:

  

HHWM (HWINPD) = HWOUTD.WM (HWINPD) = HWOUTD.(1)(1)

Relation (1) means that the properties of Relation (1) means that the properties of the system, i. e., the mapping HWM, the system, i. e., the mapping HWM, determine the correspondence between determine the correspondence between the elements of HWINPD and the the elements of HWINPD and the elements of HWOUTD. elements of HWOUTD.

Furthermore, we denote the set of inputs which Furthermore, we denote the set of inputs which cause malfunction by cause malfunction by HWINERHWINER, while the set , while the set of erroneous outputs by of erroneous outputs by HWOUTERHWOUTER, which is , which is produced as a response toproduced as a response to HWINER. HWINER. For these For these sets the following mapping relation holds true:sets the following mapping relation holds true:

HHWM (HWINER) = HWOUTER.WM (HWINER) = HWOUTER.(2)(2)

The scheme for the above two mappings is The scheme for the above two mappings is depicted in Figure 1.depicted in Figure 1.

Figure 1. Hardware mapping Figure 1. Hardware mapping scheme with faults:scheme with faults:

HWINPD

HWINER

HWM

HWOUTER

Inputdomain

Hardwaresystem:Mapping

Outputdomain

HWOUTD

If we involve tests for fault detection in our If we involve tests for fault detection in our mapping model, it will result in the mapping model, it will result in the modification of sets HWINER and HWOUTER. modification of sets HWINER and HWOUTER. Suppose a test set Suppose a test set HWINERTHWINERT is capable of is capable of detecting a subset of yet undetected detecting a subset of yet undetected design design faultsfaults. In this case HWINERT and HWINER . In this case HWINERT and HWINER necessarily have to possess common necessarily have to possess common elements. As a usual consequence, this test elements. As a usual consequence, this test must result in fault removal. After this, the must result in fault removal. After this, the new mapping relation for the corrected new mapping relation for the corrected hardware or hardware design ishardware or hardware design is

HHWM (HWINER WM (HWINER HWINERT) = HWINERT) = HWOUTERHWOUTER (3)(3)

where the minus sign stands for where the minus sign stands for subtraction between sets. The subtraction between sets. The corresponding scheme is seen in Figure corresponding scheme is seen in Figure 2.2.

Figure 2. Mapping scheme after Figure 2. Mapping scheme after testing:testing:

HWINERT

HWINER

HWM

HWOUTER

Inputdomain

HWINPD

Hardwaresystem:Mapping

Outputdomain

HWOUTD

At this point At this point tthe following notations are introduced:he following notations are introduced:

            The set of inputs which manifest themselves in design The set of inputs which manifest themselves in design faults: faults: HWDEFIHWDEFI. The output mapping of this set is . The output mapping of this set is HWDEFOHWDEFO..

            The set of inputs which manifest themselves in The set of inputs which manifest themselves in operational faults: operational faults: HWOPFIHWOPFI. The output mapping of . The output mapping of this set is this set is HWOPFOHWOPFO..

            The set of test inputs which have been devised to The set of test inputs which have been devised to detect design faults, i. e., the set of verification tests: detect design faults, i. e., the set of verification tests: HWDFTHWDFT..

            The set of test inputs which have been devised to The set of test inputs which have been devised to detect operational faults, i. e., the set of fault-detection detect operational faults, i. e., the set of fault-detection tests: tests: HWOPFTHWOPFT..

            The set of inputs which manifest themselves in The set of inputs which manifest themselves in user faults: user faults: HWUSFIHWUSFI. The output mapping of this . The output mapping of this set is set is HWUSFOHWUSFO..

            The set of test inputs which have been devised to The set of test inputs which have been devised to detect user faults, i. e., the set of validation tests: detect user faults, i. e., the set of validation tests: HWUSFTHWUSFT..

Now that we have a way of mapping various input sets Now that we have a way of mapping various input sets into output sets, we can give a general model to into output sets, we can give a general model to describe the mapping scheme of verification, describe the mapping scheme of verification, testing, and validation related to a specific hardware testing, and validation related to a specific hardware system. The mapping relations are as follows:system. The mapping relations are as follows:

HWM (HWINPD) = HWOUTD.HWM (HWINPD) = HWOUTD. (4)(4)HHWM (HWDEFI WM (HWDEFI HWDEFT) = HWDEFO. HWDEFT) = HWDEFO. (5)(5)HHWM (HWOPFI WM (HWOPFI HWOPFT) = HWOPFO. HWOPFT) = HWOPFO. (6)(6)HWM (HWUSFI HWM (HWUSFI HWUSFT) = HWUSFO. HWUSFT) = HWUSFO. (7)(7)

Here relation (5) expresses the design-verification Here relation (5) expresses the design-verification mapping, relation (6) expresses the mapping for mapping, relation (6) expresses the mapping for operational testing, while relation (7) is for the operational testing, while relation (7) is for the validation mapping. From these relations it can be validation mapping. From these relations it can be seen that (5) represents the undetected design seen that (5) represents the undetected design faults, (6) represents the uncovered operational faults, (6) represents the uncovered operational faults, whereas (7) represents the undetected user faults, whereas (7) represents the undetected user faults.faults.

TThese hese test test sets are advisable to be combined with sets are advisable to be combined with each other, in order to increase their fault detection each other, in order to increase their fault detection capability. Some useful combinations:capability. Some useful combinations:

HWM (HWDEFI HWM (HWDEFI (HWDFT (HWDFT HWOPFT)) = HWDEFO. HWOPFT)) = HWDEFO.(8)(8)

HWM (HWOPFI HWM (HWOPFI (HWDFT (HWDFT HWOPFT)) = HWOPFT)) = HWOPFO.HWOPFO. (9)(9)

HWM (HWUSFI HWM (HWUSFI (HWDEFT (HWDEFT HWUSFT)) = HWUSFT)) = HWUSFO.HWUSFO. (10)(10)

HWM (HWUSFI HWM (HWUSFI (HWDEFT (HWDEFT HWOPFT HWOPFT HWUSFT)) = HWUSFO.HWUSFT)) = HWUSFO. (11)(11)

B. The mapping model for software:B. The mapping model for software:

Software faults are derived from programming Software faults are derived from programming errors, as well as erroneous or imperfect errors, as well as erroneous or imperfect specification. specification.

Here we will refer to these categories as Here we will refer to these categories as development faultsdevelopment faults and and user faultsuser faults, respectively., respectively.

The first category of fault is a result of inadequate The first category of fault is a result of inadequate development process, whereas the second one development process, whereas the second one does not conform to the ultimate user does not conform to the ultimate user requirements.requirements.

A software system can be considered as an A software system can be considered as an entity which maps an input set to an output entity which maps an input set to an output set. A system has many possible inputs. The set. A system has many possible inputs. The response to these inputs is made by producing response to these inputs is made by producing an output or a set of outputs.an output or a set of outputs.

The software faults manifest themselves in The software faults manifest themselves in program malfunction when the faulty code program malfunction when the faulty code segment is executed with a set of inputs which segment is executed with a set of inputs which expose the actual fault. At the same time, the expose the actual fault. At the same time, the rest of the code may work properly for the rest of the code may work properly for the other inputs. other inputs.

Let the set of all possible inputs to the software under Let the set of all possible inputs to the software under

consideration be denoted by INPD (input domain), and the set consideration be denoted by INPD (input domain), and the set

of all possible output responses by OUTD (output domain). of all possible output responses by OUTD (output domain).

Now the mapping of INPD into OUTD by the software system Now the mapping of INPD into OUTD by the software system

will be defined in the following form:will be defined in the following form:

SWM (SWINPD) =SWOUTD.SWM (SWINPD) =SWOUTD. (12)(12)

Relation (1) means that the properties of the system, i. e., the Relation (1) means that the properties of the system, i. e., the

mapping SWM, determine the correspondence between the mapping SWM, determine the correspondence between the

elements of INPD and the elements of OUTD.elements of INPD and the elements of OUTD.

We denote the set of inputs which cause We denote the set of inputs which cause malfunction by SWINER, while the set of malfunction by SWINER, while the set of erroneous outputs by SWOUTER, where erroneous outputs by SWOUTER, where these sets are associated with these sets are associated with programming (development) faults. programming (development) faults.

For these sets the following mapping For these sets the following mapping relation holds true:relation holds true:

SWM (SWINER) = SWOUTER.SWM (SWINER) = SWOUTER.(13)(13)

If we involve tests for fault detection in our mapping model, If we involve tests for fault detection in our mapping model,

it will result in the modification of sets INER and OUTER. it will result in the modification of sets INER and OUTER.

Suppose a test set INERT is capable of detecting a subset of Suppose a test set INERT is capable of detecting a subset of

yet undetected faults. In this case INERT and INER yet undetected faults. In this case INERT and INER

necessarily have to possess common elements. As a usual necessarily have to possess common elements. As a usual

consequence, this test must result in fault removal. It means consequence, this test must result in fault removal. It means

that the corrected software will produce a modified output that the corrected software will produce a modified output

domain with a subset OUTER which does not represent the domain with a subset OUTER which does not represent the

detected and so deleted faults any longer. After this, the new detected and so deleted faults any longer. After this, the new

mapping relation for the corrected software ismapping relation for the corrected software is

SWM (SWINER -SWINERT) = SWOUTERSWM (SWINER -SWINERT) = SWOUTER (14)(14)

Notations:Notations:

The following notations are introduced:The following notations are introduced:• The set of inputs which manifest themselves in The set of inputs which manifest themselves in

development faults: SWDEFI. The output development faults: SWDEFI. The output mapping of this set is SWDEFO.mapping of this set is SWDEFO.

• The set of test inputs which have been devised to The set of test inputs which have been devised to detect development faults, i. e., the set of detect development faults, i. e., the set of verification tests: SWDEFT.verification tests: SWDEFT.

• The set of inputs which manifest themselves in The set of inputs which manifest themselves in user faults: SWUSFI. The output mapping of this user faults: SWUSFI. The output mapping of this set is SWUSFO.set is SWUSFO.

• The set of test inputs which have been devised to The set of test inputs which have been devised to detect user faults, i. e., the set of validation tests: detect user faults, i. e., the set of validation tests: SWUSFT. SWUSFT.

The mapping relations for The mapping relations for verification and validation:verification and validation:

SWM (SWINPD) = SWOUTD.SWM (SWINPD) = SWOUTD. (15)(15) SWM (SWDEFI – SWDEFT) = SWDEFOSWM (SWDEFI – SWDEFT) = SWDEFO (16)(16) SWM (SWUSFI – SWUSFT) = SWUSFOSWM (SWUSFI – SWUSFT) = SWUSFO (17)(17)

Here relation (16) expresses the verification mapping, while Here relation (16) expresses the verification mapping, while relation (17) expresses the validation mapping. From these relation (17) expresses the validation mapping. From these relations it can be seen that (16) represents the relations it can be seen that (16) represents the undetected development faults, whereas (17) represents undetected development faults, whereas (17) represents the undetected user faults.the undetected user faults.

For the purpose of decreasing the number of For the purpose of decreasing the number of undetected faults it is worth combining the existent undetected faults it is worth combining the existent test sets as shown below:test sets as shown below:

SWM (SWDEFI - (SWDEFT SWM (SWDEFI - (SWDEFT SWUSFT)) = SWUSFT)) = SWDEFO.SWDEFO.

(18)(18)

SWM (SWUSFI - (SWDEFT SWM (SWUSFI - (SWDEFT SWUSFT)) = SWUSFT)) = SWUSFO.SWUSFO.

(19)(19)These relations are illustrated in Figure 3.These relations are illustrated in Figure 3.

Figure 3. Mapping scheme for Figure 3. Mapping scheme for verification and validation of the verification and validation of the software:software:

SWUSFI

SWUSFT

SWM

SWUSFO

Input domain

SWINPD

Softwaresystem:Mapping

Output domain

SWOUTD

SWDEFO

SWDEFT

SWDEFI

IV. THE USE OF FORMAL IV. THE USE OF FORMAL METHODSMETHODS

The term formal methods describes the use of The term formal methods describes the use of mathematical techniques in the specification, mathematical techniques in the specification, design, and analysis of computer hardware and design, and analysis of computer hardware and software. A specification must be software. A specification must be unambiguous, complete, consistent and unambiguous, complete, consistent and correct. Documents written in natural correct. Documents written in natural languages are always susceptible to languages are always susceptible to misunderstanding. It is also difficult to ensure misunderstanding. It is also difficult to ensure that such documents represent full and correct that such documents represent full and correct description of the required system, or even to description of the required system, or even to demonstrate that they are consistent.demonstrate that they are consistent.

Formal methods are based on the use of formal Formal methods are based on the use of formal languages which have very precise and strict rules. languages which have very precise and strict rules. This feature enables the specifications to be defined in This feature enables the specifications to be defined in a manner that can be interpreted unambiguously. It a manner that can be interpreted unambiguously. It also makes possible the automatic checking of the also makes possible the automatic checking of the specifications in order to find omissions and specifications in order to find omissions and inconsistencies, i. e., to prove the completeness and inconsistencies, i. e., to prove the completeness and consistency. Languages intended for this purpose are consistency. Languages intended for this purpose are called called system specification languagessystem specification languages, or , or formal formal specification languagesspecification languages. Their use offers many . Their use offers many potential advantages in all phases of the development potential advantages in all phases of the development cycle.cycle.

One of the greatest advantages of describing a One of the greatest advantages of describing a system in a formal manner is that system in a formal manner is that automated automated transformationstransformations among the necessary design among the necessary design phases may then be carried out on this phases may then be carried out on this description. As far as hardware is concerned, the description. As far as hardware is concerned, the most known specification language is most known specification language is VHDLVHDL (VLSI (VLSI Hardware-Description Language)Hardware-Description Language). Here the final . Here the final product is the layout design of the integrated product is the layout design of the integrated circuit which realizes the operation of the circuit which realizes the operation of the specified and described hardware. This result specified and described hardware. This result justifies the term for this tool kit as a justifies the term for this tool kit as a “silicon “silicon compiler”compiler”. .

The final phase of a software The final phase of a software development is the source code development is the source code that can be executed on a that can be executed on a computer. A typical standardized computer. A typical standardized formal language for this purpose is formal language for this purpose is UMLUML (Unified Modeling Language) (Unified Modeling Language), , and its associated software tools. and its associated software tools.

The other advantage is that The other advantage is that automated testsautomated tests can can be performed at the various design phases. This be performed at the various design phases. This allows software tools to check for certain classes allows software tools to check for certain classes of error, but also allows different descriptions to of error, but also allows different descriptions to be compared to decide if they are equivalent. This be compared to decide if they are equivalent. This process is nothing else then the process is nothing else then the verification itselfverification itself. . Figure 4 shows that each step of transformation is Figure 4 shows that each step of transformation is followed by the task of confirming that the task followed by the task of confirming that the task has been carried out correctly. This involves has been carried out correctly. This involves demonstrating that the description which forms demonstrating that the description which forms the input to a given phase is functionally the input to a given phase is functionally equivalent to that produced at its output.equivalent to that produced at its output.

Figure 4. Application Figure 4. Application process of formal process of formal methods:methods:

Formal specification

Transformation and checking

1. Design phase

Transformation and checking

2. Design phase

Transformation and checking

Final design phase

Verification

Verification

Verification

In an In an ideal caseideal case, the transformation procedures , the transformation procedures in the above outlined process are completely in the above outlined process are completely automated, having no human interaction, and automated, having no human interaction, and are performed without any faults at any phase. are performed without any faults at any phase. If so, the external test input sequences for If so, the external test input sequences for verification can be completely omitted. In our verification can be completely omitted. In our test model in relation (5) it means thattest model in relation (5) it means that

HWDEFI = HWDEFI = ,,

where the symbol where the symbol represents the empty represents the empty set. set.

By substitution into (5) we obtainBy substitution into (5) we obtain

HHWM (WM ( HWDEFT) = HWDEFT) = HHWM (WM () = HWDEFO = ) = HWDEFO = .. (20)(20)

The same considerations apply to software verification, i.e., The same considerations apply to software verification, i.e., for the relation (16):for the relation (16):

SWDEFI = SWDEFI = ,, and thusand thus  SWM (SWM ( SWDEFT) = SWM ( SWDEFT) = SWM () = SWDEFO = ) = SWDEFO = .. (21)(21)  

On the other hand, the initial formal specification of On the other hand, the initial formal specification of the software is always carried out by a the software is always carried out by a manual waymanual way, , and therefore design errors may never be excluded and therefore design errors may never be excluded here, even if there is an automated consistency here, even if there is an automated consistency checking available. The reason for this is that a checking available. The reason for this is that a consistent design in itself does not guarantee the consistent design in itself does not guarantee the perfect fulfillment of the user requirements. Also, there perfect fulfillment of the user requirements. Also, there is no guarantee for meeting the safety requirements is no guarantee for meeting the safety requirements either. As a consequence, the either. As a consequence, the external validation external validation testingtesting is always necessary to apply. In the above ideal is always necessary to apply. In the above ideal case:case:

HWM (HWUSFI - HWUSFT) = HWUSFO.HWM (HWUSFI - HWUSFT) = HWUSFO. (22)(22)SWM (SWUSFI - SWUSFT) = SWUSFO.SWM (SWUSFI - SWUSFT) = SWUSFO. (23)(23)

Fully automatic performance is yet Fully automatic performance is yet not applicable, that is, close not applicable, that is, close cooperation and interaction of the cooperation and interaction of the skilled designers are further skilled designers are further required. At present, the main required. At present, the main advantage of formal methods is that advantage of formal methods is that they enable to perform the design they enable to perform the design and verification processes with and verification processes with greater reliabilitygreater reliability..

Conclusion:Conclusion:

In this pIn this presentationresentation a general model a general model for hardware/software testing has for hardware/software testing has been proposed. The significance of been proposed. The significance of the model is that it alleviates the the model is that it alleviates the clear clear differentiationdifferentiation between tests for between tests for verification, fault detectionverification, fault detection,, and and validationvalidation. This feature is important . This feature is important and useful in the process of test and useful in the process of test design and evaluation, especially in design and evaluation, especially in the case of safety-critical systemsthe case of safety-critical systems..