Model-Based SystemsModel-Based Systems The Model-Based Systems (MBS) paradigm refers to a...

Model-Based Systems

The Model-Based Systems (MBS) paradigm refers to a methodology that

allows for description of various kinds of systems for various tasks in a uniform

way. For example, MBS has been used to specify monitoring tasks in medical

systems, for planning in cognitive systems, and control and diagnosis in hard-

ware and software systems. Consequently, research in MBS is spread across

various application domains and different tasks.

As lots of scientific workshops are application specific or system and task

oriented, it is difficult to exchange experiences and novel concepts across the

various application domains and tasks. In recent years MBS technology has in-

creasingly contributed to mastering the inherent and ever increasing complexity

of software and software-enabled systems. Thus it is the aim of this workshop to

cross-fertilize the established concepts in model-based software engineering and

MBS technology to further leverage model-oriented techniques in the software

engineering domain. MBS 2008 workshop attracted researchers and practition-

ers dealing with modeling for specific reasoning tasks, knowledge representation,

qualitative reasoning, and related areas such as model-based testing and fault

detection and localization.

The MBS 2008 - Workshop on Model-Based Systems is the fourth workshop

of a series of workshops on the this topic. Previous workshops were collocated

with the ECAI 2004 in Valencia, Spain, the IJCAI 2005 in Edinburgh, United

Kingdom and the ECAI 2006 in Riva del Garda, Italy.

The submissions to the MBS 2008 cover a wide range of topics with in

the area of model-based systems. They range from more application-oriented

solutions to modeling problems including automated generation and debugging

of models to more theoretical contributions in the areas of diagnosis, qualitative

reasoning and testing. The good mixture of theoretical and application-oriented

articles from various domains promises a very interesting and fruitful workshop.

Finally we like to thank all the authors who have submitted to this work-

shop. Moreover we like to thank all members of the program committee for

their careful reviews.

Bernhard Peischl, Neal Snooke, Gerald Steinbauer and Cees Witteveen

July 2008

i

Organizing Committee

Bernhard Peischl Technische Universitat Graz, Austria

Neal Snooke University of Wales, Aberystwyth, UK

Gerald Steinbauer Technische Universitat Graz, Austria

Cees Witteveen Delft University of Technology, The Netherlands

Program Committee

Gautam Biswas Vanderbilt University

Bert Bredeweg Universiteit van Amsterdam, The Netherlands

Marie-Odile Cordier IRISA Campus de Beaulieu, France

Carlos J. Alonso Gonzlez Universidad de Valladolid, Spain

Bernhard Peischl Technische Universitat Graz, Austria

Caudia Picardi Universit di Torino, Itanly

Belarmino Pulido Junquera Universidad de Valladolid, Spain

Martin Sachenbacher Technische Universitat Munchen, Germany

Paulo Salles Universidade de Brasilia, Brazil

Neal Snooke University of Wales, Aberystwyth, UK

Gerald Steinbauer Technische Universitat Graz, Austria

Cees Witteveen Delft University of Technology, The Netherlands

ii

Table of Contents

Comparing GDE and Conflict-based Diagnosis

Ildiko Flesch, Peter J.F. Lucas . . . . . . . . . . . . . . . . . . . 1

On computing minimal conflicts for ontology debugging

Kostyantyn Shchekotykhin, Gerhard Friedrich, Dietmar Jannach 7

Supporting Conceptual Knowledge Capture Through Automatic

Modelling

Jochem Liem, Hylke Buisman, Bert Bredeweg . . . . . . . . . . . 13

Automated Learning of Communication Models for Robot Con-

trol Software

Alexander Kleiner, Gerald Steinbauer, Franz Wotawa . . . . . . . 19

Relaxation of Temporal Observations in Model-Based Diagnosis

of Discrete-Event Systems

Gianfranco Lamperti, Federica Vivenzi, Marina Zanella . . . . . 25

The Concept of Entropy by means of Generalized Orders of Mag-

nitude Qualitative Spaces

Llorens Rosello, Francesc Prats, Monica Sanchez, Nuria Agell . . 31

Model-based Testing using Quantified CSPs: A Map

Martin Sachenbacher, Stefan Schwoon . . . . . . . . . . . . . . . 37

iii

Comparing GDE and Conflict-based DiagnosisIld ik o Flesch1 and Peter J.F. Lucas2

Abstract. Conflict-based diagnosis is a recently proposed methodfor model-based diagnosis, inspired by consistency-based diagnosis,that incorporates a measure of data conflict, called the diagnosticconflict measure, to rank diagnoses. The probabilistic informationthat is required to compute the diagnostic conflict measure is rep-resented by means of a Bayesian network. The general diagnosticengine is a classical implementation of consistency-based diagnosisand incorporates a way to rank diagnoses using probabilistic infor-mation. Although conflict-based and consistency-based diagnosis arerelated, the way the general diagnostic engine handles probabilisticinformation to rank diagnoses is different from the method used inconflict-based diagnosis. In this paper, both methods are comparedto each other.

1 INTRODUCTION

In the last two decades, research into model-based diagnostic soft-ware has become increasingly important, mainly because the com-plexity of devices, for which such software can be used, hasrisen considerably and trouble shooting of faults in such deviceshas therefore become increasingly difficult. Basically, two typesof model-based diagnosis are being distinguished in literature: (i)consistency-based diagnosis [2, 8], and (ii ) abductive diagnosis [7].In consistency-based diagnosis a diagnosis has to beconsistentwiththe modelled system behaviour and observations made on the actualsystem, whereas in abductive diagnosis the observations have to beimplied by the modelled system given the diagnosis [1]. In this pa-per, we focus on consistency-based diagnosis as implemented in thegeneral diagnostic engine, GDE for short, [2]. In addition, particu-lar probabilistic extensions to consistency-based diagnosis as imple-mented in GDE are considered [2].

There is also a third kind of model-based diagnosis that can be bestseen as a translation of consistency-based diagnosis from a mixedlogical-probabilistic setting to a purely probabilistic setting, usinga statistical measure of information conflict. The method has beencalledconflict-based diagnosis; it exploits Bayesian-network repre-sentations for the purpose of model-based diagnosis [4].

Although both GDE and conflict-based diagnosis takeconsistency-based diagnosis as a foundation, the way uncer-tainty is handled, as well as the way in which diagnoses are ranked,are different. The aim of this paper is to shed light on the differencesand similarities between these two approaches to model-baseddiagnosis. It is shown that conflict-based diagnosis yields a rankingthat, under particular circumstances, is more informative than thatobtained by GDE.

1 Department of Computer Science, Maastricht University, email:[email protected]

2 Institute for Computing and Information Sciences, Radboud University Ni-jmegen, email:[email protected]

The paper is organised as follows. In Section 2, the necessary basicconcepts from model-based diagnosis, including GDE, and the use ofBayesian networks for model-based are reviewed. Next, in Section 3,the basic concepts from conflict-based diagnosis are explained. Whatcan be achieved by the method of probabilistic reasoning in GDE issubsequently compared to the method of conflict-based diagnosis inSection 4. Finally, in Section 5, the paper is rounded off with someconclusions.

2 PRELIMINARIES

2.1 Model-based Diagnosis

In the theory of consistency-based diagnosis [8, 2, 3], the structureand behaviour of a system is represented by alogical diagnostic sys-temSL = (SD, COMPS), where

• SD denotes thesystem description, which is a finite set of logicalformulae, specifying structure and behaviour;

• COMPS is a finite set of constants, corresponding to thecompo-nentsof the system that can be faulty.

The system description consists ofbehaviour descriptionsandcon-nections. A behavioural description is a formula specifyingnormalandabnormal(faulty) functionality of the components. Anabnor-mality literal of the formAc is used to indicate that componentc isbehaving abnormally. whereas literals of the form¬Ac are used toindicate that componentc is behaving normally. A connection is aformula of the formic ≡ oc′ , whereic andoc′ denote the input andoutput of componentsc andc′, respectively.

A logical diagnostic problemis defined as a pairPL =(SL, OBS), whereSL is a logical diagnostic system and OBS is afinite set of logical formulae, representingobservations.

Adopting the definition from [3], a diagnosis in the theory ofconsistency-based diagnosis is defined as follows. Let∆C consistof the assignment of abnormal behaviour, i.e.Ac, to the set of com-ponentsC ⊆ COMPS and normal behaviour, i.e.¬Ac, to the re-maining componentsCOMPS−C, then∆C is aconsistency-baseddiagnosisof the logical diagnostic problemPL iff the observationsare consistent with both the system description and the diagnosis;formally:

SD ∪ ∆C ∪ OBS 2 ⊥.

Here,2 stands for the negation of the logical entailment relation�,and⊥ represents a contradiction.

Usually, one is in particular interested insubset-minimaldiag-noses, i.e. diagnoses∆C , where the setC is subset minimal. Thus,a subset-minimal diagnosis assumes that a subset-minimal numberof components are faulty; this often corresponds to the most-likelydiagnosis.

1

X1

A1

A2

X2

R1

10

1

0 predicted[1] observed

1 predicted[0] observed

1

01

0

111

11

10

Figure 1. Full adder with all outputs computed under the assumption ofnormality and observed and predicted outputs;i1 (1), ı2 (0) andi3 (1)

indicate the inputs of the circuit ando1 (1) ando2 (0) its observed outputs.

EXAMPLE 1 Figure 1 presents the full-adder example, which con-sists of two AND gates (A1 andA2), one OR gate (R1) and twoexclusive-OR (XOR) gates (X1 and X2). Note that thepredictedoutputo1 contradicts with theobservationo1, which is also the casefor gateX2. As a consequence, the assumption that all componentsare behaving normally is invalid; thus, this isnota consistency-baseddiagnosis. However, a consistency-based diagnosis would be to as-sume the malfunctioning of componentX1, as this would restoreconsistency. 2

2.2 GDE

Next, GDE is briefly described, where [2] is used as a point of refer-ence; however, the terminology defined above in this paper is adoptedthroughout this section. For example, where [2] speaks of a ‘candi-date’ in this paper the term ‘diagnosis’ is used.

The logical reasoning implemented by GDE can best be seen as anefficient implementation of consistency-based diagnosis. GDE canalso deal with uncertainty by attaching a prior probability of mal-functioning to components. After an observation is made, the priorprobability becomes a posterior probability, conditioned on this ob-servation. Based on new observations, there may be previous diag-noses which become inconsistent with the observations and the sys-tem description. The set of diagnoses that are still possible is denotedby R and called the set ofremainingdiagnoses; it can be partitionedinto two disjoint subsets: (i) the set of diagnoses that imply the ob-servations, called the set ofselecteddiagnoses and denoted byS, and(ii ) the set of diagnoses that neither predict nor contradict the obser-vations, called the set ofuncommitteddiagnoses, denoted byU . Bydefinition,R = S ∪ U andS ∩ U = ∅.

The posterior probability of a set of behaviour assumptions thatis either inconsistent (not inR), a selected diagnosis (inS), or anuncommitted diagnosis (inU ) is computed as follows:

P (∆C | OBS) =

8

>

<

>

:

0 if ∆C 6∈ RP (∆C)P (OBS)

if ∆C ∈ SP (∆C)/mP (OBS)

if ∆C ∈ U

(1)

wherem = 1/P (OBS | ∆C).Finally, the probabilityP (OBS) is computed as follows:

P (OBS) =X

∆C∈R

P (OBS, ∆C)

=X

∆C∈S

P (OBS, ∆C) +X

∆C∈U

P (OBS, ∆C)

=X

∆C∈S

P (∆C) +X

∆C∈U

P (∆C)

m. (2)

Computation ofP (∆C) is made easy in GDE by assuming indepen-dence between components behaving normally or abnormally.

One of the consequences of this assumption is the followingproposition.

Proposition 1 LetPL = (SD, OBS) be a logical diagnostic systemwith associated joint probability distributionP as defined above forGDE, such thatP (Ac) ≪ P (¬Ac) for eachc ∈ COMPS, and let∆C and ∆C′ be two consistency-based diagnoses that are both ineitherS or U , then it holds that:

P (∆C | OBS) ≥ P (∆C′ | OBS) if C ⊆ C′.

Proof. The result follows from the assumption of independence to-gether withP (Ac) ≪ P (¬Ac):

P (∆C) =Y

c∈C

P (Ac)Y

c∈COMPS−C

P (¬Ac)

≥Y

c∈C′

P (Ac)Y

c∈COMPS−C′

P (¬Ac) = P (∆C′)

Filling this result into Equation (1) gives the requested outcome.2

For further detail of GDE the reader is referred to the paper by DeKleer and Williams [2]. The following example illustrates how GDEworks.

Table 1. Comparison of the values of the diagnostic conflict measure andGDE for the full-adder circuit with observationsOBS = ω =

{i1, ı2, i3, o1, o2} and the probability distributionP , assuming thatP (ac) = P (oc | ac) = 0.001.

k X2 R1 X1 A1 A2 conf[P δk ](ω) GDE’sP (∆k | OBS)

1 1 1 1 1 1 – –2 1 1 1 1 0 – –3 1 1 1 0 1 – –4 1 1 1 0 0 – –5 1 1 0 1 1 −0.4255 0.994026 1 1 0 1 0 −0.4255 9.9502 · 10−4

7 1 1 0 0 1 −0.3006 9.9502 · 10−4

8 1 1 0 0 0 −0.3006 9.9601 · 10−7

9 1 0 1 1 1 – –10 1 0 1 1 0 – –11 1 0 1 0 1 – –12 1 0 1 0 0 – –13 1 0 0 1 1 −0.3006 9.9502 · 10−4

14 1 0 0 1 0 −0.3006 9.9601 · 10−7

15 1 0 0 0 1 −0.3006 9.9601 · 10−7

16 1 0 0 0 0 −0.3006 9.9701 · 10−10

17 0 1 1 1 1 – –18 0 1 1 1 0 −0.1249 9.9502 · 10−4

19 0 1 1 0 1 – –20 0 1 1 0 0 0 9.9502 · 10−7

21 0 1 0 1 1 −0.1247 9.9502 · 10−4

22 0 1 0 1 0 −0.1249 9.9601 · 10−7

23 0 1 0 0 1 0.0002 9.9601 · 10−7

24 0 1 0 0 0 0 9.9701 · 10−10

25 0 0 1 1 1 0 9.9502 · 10−4

26 0 0 1 1 0 0 9.9601 · 10−7

27 0 0 1 0 1 0 9.9601 · 10−7

28 0 0 1 0 0 0 9.9701 · 10−10

29 0 0 0 1 1 0 9.9601 · 10−7

30 0 0 0 1 0 0 9.9701 · 10−10

31 0 0 0 0 1 0 9.9701 · 10−10

32 0 0 0 0 0 0 9.9801 · 10−13

EXAMPLE 2 Reconsider the full-adder shown in Figure 1, where

2

each component can only be normal or abnormal. Assume that theprobability of faulty behaviour of a component is equal toP (Ac) =0.001. Without any observations, the diagnosis space consists of25 = 32 members, where the diagnosis∆∅ = {¬Ac | c ∈ COMPS}is the most probable diagnosis with probabilityP (∆∅) = (1 −P (Ac))

5 = (0.999)5 ≈ 0.995. When more components are as-sumed to be faulty, the probabilities decrease quickly to very smallvalues.

Now, suppose thatOBS = {i1, ı2, i3, o1, o2}. The new prob-abilities obtained from GDE are shown in the right-most columnof Table 1, where ‘1’ for a component means normal behaviourand ‘0’ means abnormal behaviour. The diagnoses∆k, for k =1, 3, 4, 9, . . . , 12, 17, 19, respectively, are eliminated by these obser-vations. Furthermore, since there are no diagnoses in the setR thatimply the two output observations, the set ofS is empty and, thus,the set of uncommitted diagnosesU is equal toR. Then, the posteriorprobability of a diagnosis∆k can be computed as follows:

P (∆k | OBS) =P (∆k)/m

(P

∆C∈U P (∆C))/m=

P (∆k)P

∆C∈U P (∆C),

where hereP

∆C∈U P (∆C) ≈ 1.002 · 10−3. 2

In the example, the probability of the∆k ’s that still can be diagnosesbecome about 1000 times more likely when conditioning on the ob-servations than without observations. However, either with or with-out observations, the diagnosis with the fewest number of abnormal-ity assumptions is the most likely one. Thus the resulting diagnosticreasoning behaviour is very similar to that obtained by exploiting theconcept of subset-minimal diagnosis.

2.3 Bayesian Networks and the Conflict Measure

Let P (X) be a joint probability distribution of the set of discretebinary random variablesX. A single random variable taking the val-ues ‘true’ or ‘false’ is written as (upright)y and y, respectively. Ifwe refer to arbitrary values of a set of variablesX, sometimes a sin-gle variable, this will be denoted by (italic)x. Let U, W, Z ⊆ X bedisjoint sets of random variables, thenU is said to beconditionallyindependentof W givenZ, if for each valueu, w andz:

P (u | w, z) = P (u | z), with P (w, z) > 0. (3)

A Bayesian networkB is defined as a pairB = (G, P ), whereG = (V, E) is an acyclic directed graph, with set of verticesVand set of arcsE, P is the associated joint probability distributionof the set of random variablesX which is associated 1–1 withV .We will normally use the same names for variables and their as-sociated vertices. The factorisation ofP respects the independencestructure ofG as follows:P (x) =

Q

y∈x P (y | π(y)), whereπ(y)denotes the values of the parent set of vertexY . Finally, we will fre-quently make use of marginalising out particular variablesW writtenasP (u) =

P

w P (u, w).Bayesian networks specify probabilistic patterns that must be ful-

filled by observations. Observations are random variables that ob-tain a value through an intervention, such as a diagnostic test. Theset ofobservationsis denoted byω. Theconflict measurehas beenproposed as a tool for the detection of potential conflicts betweenobservations and a given Bayesian network and is defined as [5]:

conf(ω) = logP (ω1)P (ω2) · · ·P (ωm)

P (ω), (4)

with ω = ω1 ∪ ω2 ∪ · · · ∪ ωm.

v n

u P (u) = 0.2

P (v | u) = 0.8P (v | u) = 0.01

P (n | u) = 0.9P (n | u) = 0.1

Figure 2. Example of a Bayesian network.

The interpretation of the conflict measure is as follows. A zeroor negative conflict measure means that the denominator is equallylikely or more likely than the numerator. This is interpreted as that thejoint occurrence of the observations is in accordance with the prob-abilistic patterns inP . A positive conflict measure, however, impliesnegative correlation between the observations andP indicating thatthe observations do not matchP very well.

EXAMPLE 3 Consider the Bayesian network shown in Figure 2,which describes that stomach ulcer (u) may give rise to both vomit-ing (v) and nausea (n).

Now, suppose that a patient comes in with the symptoms of vom-iting and nausea. The conflict measure then has the following value:

conf({v, n})=logP (v)P (n)

P (v, n)=log

0.168 · 0.26

0.1448≈−0.5.

As the conflict measure assumes a negative value, there is no con-flict between the two observations. This is consistent with medicalknowledge, as we do expect that a patient with stomach ulcer dis-plays symptoms of both vomiting and nausea.

As a second example, suppose that a patient has only symptomsof vomiting. The conflict measure now obtains the following value:

conf({v, n}) = log0.168 · 0.74

0.0232≈ log 5.36 ≈ 0.7.

As the conflict measure is positive, there is a conflict between thetwo observations, which is in accordance to medical expectations.2

2.4 Bayesian Diagnostic Problems

A Bayesian diagnostic systemis denoted as a pairSB = (G, P ),whereP is a joint probability distribution of the vertices ofG, inter-preted as random variables, andG is obtained by mapping a logicaldiagnostic systemSL = (SD, COMPS) to a Bayesian diagnosticsystemSB as follows [6]:

1. componentc is represented by itsinput Ic andoutputOc vertices,where inputs are connected by an arc to the output;

2. to each componentc there belongs anabnormality vertexAc

which has an arc pointing to the outputOc.

Figure 3 shows the Bayesian diagnostic system corresponding to thelogical diagnostic system shown in Figure 1.

LetO denote the set of all output variables andI the set of all inputvariables, leto andi denote (arbitrary) values of the set of output andinput variables, respectively, and let

δC = {ac | c ∈ C} ∪ {ac | c ∈ COMPS − C}

be the set of values of the abnormality variablesAc, with c ∈COMPS. The latter definition establishes a link between∆C in log-ical diagnostic systems and the abnormality variables in Bayesiandiagnostic systems.

3

I1 I2

I3 OX1

OX2

OA1

OA2

OR1

AX1

AX2

AA1

AA2

AR1

Figure 3. Thegraphical representation of a Bayesian diagnostic systemcorresponding to the full-adder in Figure 1.

Due to the independences that hold for a Bayesian diagnostic sys-tem, it is possible to simplify the computation of the joint probabilitydistributionP by exploiting the following properties:

Property 1:the joint probability distribution of a set of output vari-ablesO can be factorised as follows:

P (o) =X

i,δc

P (i, δc)Y

c∈COMPS

P (oc | π(oc)) ; (5)

Property 2:the input variables and abnormality variables are mutu-ally independent of each other, formally:P (i, δc) = P (i)P (δc).

Recall that logical diagnostic problems are logical diagnostic sys-tems augmented with observations; Bayesian diagnostic problemsare defined similarly. The input and output variables that have beenobserved are now referred to asIω andOω, respectively. The un-observed input and output variables will be referred to asIu andOu respectively. The set of actual observations is then denoted byω = iω ∪ oω. Thus, aBayesian diagnostic problemPB = (SB, ω)consists of (i) a Bayesian diagnostic system representing the compo-nents, their behaviour and interaction, and (ii ) a set of observationsω [4].

In Bayesian diagnostic problems, the normal behaviour of compo-nentc is expressed in a probabilistic setting by the assumption that anormally functioning component yields an output value with proba-bility of either0 or 1. Thus,

P (oc | π(oc)) ∈ {0, 1},

when the abnormality variableAc ∈ π(Oc) takes the value ‘false’,i.e. isac. For the abnormal behaviour of a componentc it is assumedthat the random variableOc is conditionally independent of its parentsetπ(Oc) if componentc is assumed to function abnormally, i.e.Ac

takes the value ‘true’, written as:

P (oc | π(oc)) = P (oc | ac).

Thus, the fault behaviour of an abnormal component cannot be influ-enced by its environment. We use the abbreviationP (oc | ac) = pc.Note that this assumption isnotmade when a component is behavingnormally, i.e. whenac holds.

3 CONFLICT-BASED DIAGNOSIS

There exists a 1–1 correspondence between aconsistency-based di-agnosis∆C of a logical diagnostic problemPL and aδC for whichit holds thatP (ω | δC) 6= 0 if PB is the result of the mapping de-scribed above, applied toPL. The basic idea behind conflict-based

diagnosis is that the conflict measure can be used to rank theseconsistency-based diagnoses (cf. [4]). We start with the definitionof the diagnostic conflict measure.

Definition 1 (diagnostic conflict measure) Let PB = (SB, ω) bea Bayesian diagnostic problem. Thediagnostic conflict measure, de-noted byconf[P δC ](·, ·), is defined forP (ω | δC) 6= 0, as:

conf[P δC ](iω, oω) = logP (iω | δC)P (oω | δC)

P (iω, oω | δC), (6)

with observationsω = iω ∪ oω.

Using the independence properties of Bayesian diagnostic problemswe obtain [4]:

conf[P δC ](iω, oω) = log

P

i P (i)P

ou

Q

c P (oc | π(oc)P

iuP (iu)

P

ou

Q

c P (oc | π(oc)).

whereπ(Oc) may include input variables fromI .The diagnostic conflict measure can take positive, zero and nega-

tive values having different diagnostic meaning. Note that the numer-ator of the diagnostic conflict measure is defined as the probabilityof the individual occurrence of the inputs and outputs, whereas thedenominator is defined as the probability of the joint occurrence ofthe observations. Intuitively, if the probability of the individual oc-currence of the observations is higher than that of the joint occur-rence, then the observations do not support each other. Thus, moreconflict between diagnosis and observations yields higher (more pos-itive) values of the diagnostic conflict measure. This means that thesignof the diagnostic conflict measure, negative, zero or positive, canalready be used to rank diagnoses in a qualitative fashion.

This interpretation gives rise to the following definition.

Definition 2 ((minimal) conflict-based diagnosis) Let PB =(SB, ω) be a Bayesian diagnostic problem and letδC be aconsistency-based diagnosis ofPB (i.e. P (ω | δC) 6= 0). Then,δC

is called aconflict-based diagnosisif conf[P δC ](ω) ≤ 0. A conflict-based diagnosisδC is calledminimal, if for each conflict-based di-agnosisδC′ it holds thatconf[P δC ](ω) ≤ conf[P δ

C′ ](ω).

In general, the diagnostic conflict measure has the important prop-erty that its value can be seen as the overall result of a local analysisof component behaviours under particular logical and probabilisticnormality and abnormality assumptions. A smaller value of the diag-nostic conflict measure is due to a higher likelihood of dependencebetween observations, and this indicates a better fit between observa-tions and component behaviours. Consider the following example.

EXAMPLE 4 Reconsider the full-adder circuit example from Fig-ure 1. Let as beforeω = {i1, ı2, i3, o1, o2}. The diagnostic conflictmeasures for all the possible diagnoses are listed in Table 1.

As an example, the diagnostic conflict measures for the diag-nosesδ5, δ6, δ7 and δ8 are compared to one another for the casethat the probabilitypX1 = P (oX1 | aX1) = 0.001 and it is ex-plained what it means that, according to Table 1,conf[P δ5 ](ω) =conf[P δ6 ](ω) < conf[P δ7 ](ω) = conf[P δ8 ](ω).

First, the diagnosesδk, for k = 6 andk = 7, will be considered inmore detail in order to explain the meaning of the diagnostic conflictmeasure. The difference in value of the diagnostic conflict measurefor these two diagnoses can be explained by noting that forδ6 it isassumed that the adder A1 functions normally and A2 abnormally,whereas forδ7 it is the other way around. The diagnostic conflictmeasure of the diagnosisδ6 is higher than that forδ7, because if A1

4

functions normally, then its output has to be equal to 0, whereasifA2 functions normally, then its output has to be equal to 1. Note thatit has been observed for R1 that the output is equal to 0. Because 0is the output of the OR gate R1, its inputs must be 0; therefore, theassumption that A1 functions normally with output 0 offers a betterexplanation for the output 0 of the R1 gate than the assumption inδ7

that A2 functions normally (which yields output value 1). Further-more, since in both diagnosesδ6 andδ7 component X1 is assumedto be faulty, and the output of the X1 acts as the input of A2, theassumption about the output of A2 is already relaxed. This also ex-plains the preference of diagnosisδ6 aboveδ7 and whyδ6 is rankedhigher thanδ7.

Next, the diagnosesδ7, δ8, δ13, δ14, δ15, andδ16 are compared toone another and we explain why it is reasonable that these diagnoseshave the same diagnostic conflict measure value (−0.3006). Notethat both diagnosesδ7 and δ8 include faulty assumptionaX1 andaA1, andδ13, δ14, δ15 andδ16 include the faulty behavioursaR1 andaX1. Note that for both{aX1, aA1} and{aR1, aX1}, one input ofthe X2 and the two inputs of R1 are relaxed. Therefore, they yieldthe same qualitative information about fault behaviour of the system.Below, these results are compared with those by GDE. 2

The example above illustrates that comparing the value of the di-agnostic conflict measure for different diagnoses gives considerableinsight into the behavioural abnormality of a system.

4 COMPARISON

In this section, the diagnostic conflict measure and GDE’s proba-bilistic method are compared to each other in terms of the differencein ranking they give rise to. To start, the main differences betweenthe diagnostic conflict measure and GDE are summarised, which isfollowed by an example. The example is used to illustrate that thediagnostic conflict measure yields a ranking that, for the probabilitydistribution defined earlier, conveys more useful diagnostic informa-tion than the ranking by GDE.

The following facts summarise the differences and similarities be-tween the diagnostic conflict measure and GDE:

1. an abnormality assumption∆C is a diagnosis according to GDEiff its associated diagnostic conflict measure is defined, i.e. [4]

P (ω | δC) 6= 0 ⇔ SD ∪ ∆C ∪ OBS 2 ⊥.

2. computation of the diagnostic conflict measure requires the con-ditional probabilitypc = P (oc | ac), i.e. the probability that thecomponent’s output isoc when the component is faulty, this prob-ability is assumed to be always 0 or 1 by GDE.

3. in GDE the probabilityP (ac), i.e. the probability that componentc functions abnormally, acts as the basis for ranking diagnoses;this probability is not needed to rank diagnoses using conflict-based diagnosis, because it is summed out in the computation ofthe diagnostic conflict measure.

4. the ranking of a conflict-based diagnosis is based on a local analy-sis of interactions between inputs and outputs of components, tak-ing into account the probability of particular faulty behaviours ofcomponents, and thus can be interpreted as a measure of how wellthe diagnosis, observations and system behaviour match; GDE of-fers nothing that is to some extent similar.

5. in GDE assuming more components to be functioning abnormallyrenders a diagnosis less likely, as proved in Proposition 1; a sim-ilar property does not hold for conflict-based diagnosis using thediagnostic conflict measure.

All properties above have already been discussed extensively. There-fore, only the last issue is illustrated.

EXAMPLE 5 Consider the Bayesian diagnostic problem discussedabove. Table 1 summarises the results of GDE and conflict-baseddiagnosis, which makes it easier to compare the results. Note thatδk ≡ ∆k andω ≡ OBS.

Consider again the Bayesian diagnostic problemPB with setof observationsω = {i1, ı2, i3, o1, o2} and the two diagnosesδ5 = δ{X1} = {aX2, aR1, aX1, aA1, aA2} andδ6 = δ{X1,A2} ={aX2, aR1, aX1, aA1, aA2}.

According to Table 1 the posterior probabilities computed by GDEare equal toP (∆5 | OBS) = 0.99402 and P (∆6 | OBS) =9.9502 · 10−4. Thus,∆5 is much more likely than∆6, which isdue to the inclusion of an extra abnormality assumption in∆6 incomparison to∆5. Consequently, the ranking obtained is compat-ible with subset-minimality. However, using the diagnostic conflictmeasure gives, according to Table 1, for both diagnoses the value of−0.4255. This means that relaxing one extra logical and probabilisticconstraint, i.e.A2 in addition toX1, has no effect on the likelihoodof the diagnosis in this case.

Next consider the diagnoses∆7 and∆6, which both have the samenumber of components assumed to be abnormal, and thus obtain thesame ranking according to GDE. However,δ6 andδ7 have a differentdiagnostic conflict measure, as explained in Example 4. 2

This example again illustrates that GDE and conflict-based diagnosisrank diagnoses differently. Conflict-based diagnosis really looks intothe system behaviour and, based on a local analysis of strength of thevarious constraints, comes up with a ranking.

5 CONCLUSION AND FUTURE WORK

Conflict-based diagnosis is a new concept in the area of model-baseddiagnosis that has been introduced recently [4]. In this paper, we havecompared this new method with the well-known probabilistic methodemployed in GDE. It was shown that the probabilistic method un-derlying conflict-based diagnosis yields detailed insight into the be-haviour of a system. As the obtained information differs from in-formation obtained from GDE, it may be useful as an alternative orcomplementary method.

In the near future, we intend to implement the method as part ofa diagnostic reasoning engine in order to build up experience withregard to the practical usefulness of the method.

REFERENCES[1] L. Console and P. Torasso. A Spectrum of Logical Definitions of

Model-based Diagnosis, Computational Intelligence, 7:133–141, 1991.[2] J. de Kleer and B.C. Williams. Diagnosing multiple faults, Artificial

Intelligence, 32:97–130, 1987.[3] J. de Kleer, A.K. Mackworth, and R. Reiter. Characterizing diagnoses

and systems. Artificial Intelligence, 56:197–222, 1992.[4] I. Flesch, P.J.F. Lucas and Th.P. van der Weide. Conflict-based diagno-

sis: Adding uncertainty to model-based diagnosis, Proc. IJCAI-2007,pp. 380–388, 2007.

[5] F.V. Jensen. Bayesian Networks and Decision Graphs. Springer-Verlag,New York, 2001.

[6] J. Pearl. Probabilistic Reasoning in Intelligent Systems:Networks ofPlausible Inference. Morgan Kauffman, San Francisco, CA, 1988.

[7] D. Poole, R. Goebel and R. Aleliunas. A logical reasoning system fordefaults and diagnosis, In: The knowledge Frontier, Ed. N. Cerone andG. Calla, Springer-Verlag, pp. 331–352, 1987.

[8] R. Reiter. A theory of diagnosis from first principles. Artificial Intelli-gence, 32:57–95, 1987.

5

On computing minimal conflicts for ontology debuggingKostyantyn Shchekotykhin and Gerhard Friedrich1 and Dietmar Jannach 2

Abstract. Ontology debugging is an important stage of the ontol-ogy life-cycle and supports a knowledge engineer during the ontol-ogy development and maintenance processes. Model based diagnosisis the basis of many recently suggested ontology debugging methods.The main difference between the proposed approaches is the methodof computing required conflict sets, i.e. a sets of axioms such that atleast one axiom of each set should be changed (removed) to make on-tology coherent. Conflict set computation is, however, the most timeconsuming part of the debugging process. Consequently, the choiceof an efficient conflict set computation method is crucial for ensuringthe practical applicability of an ontology debugging approach.

In this paper we evaluate and compare two popular minimal con-flict computation methods: QUICKXPLAIN and SINGLE JUST. First,we analyze best and worst cases of the required number of coherencychecks of both methods on a theoretical basis assuming a black-boxreasoner. Then, we empirically evaluate the run-time efficiency of thealgorithms both in black-box and in glass-box settings.

Although both algorithms were designed to view the reasoner asa black box, the exploitation of specific knowledge about the rea-soning process (glass-box) can significantly speed up the run-timeperformance in practical applications. Therefore, we present modifi-cations of the original algorithms that can also exploit specific datafrom the reasoning process.

Both a theoretical analysis of best- and worst-case complexity aswell as an empirical evaluation of run-time performance show thatQUICKXPLAIN is preferable over SINGLE JUST.

1 MOTIVATIONWith an increasing number of applications that rely on ontologies,these knowledge bases are getting larger and more complex. Thus,corresponding knowledge bases can include definitions of thousandsof concepts and roles from different domains. RDF search engineslike Watson [2] for instance facilitate the creation of composite on-tologies that reuse the definition of concepts and roles published onthe Web. Moreover, the community of ontology users is getting moreheterogeneous and nowadays includes many members from variousindustrial and scientific fields. Hence, different faults can be easilyintroduced during creation and maintenance of ontologies.

Recent debugging methods as described in [4, 8, 9, 11] help theuser to localize, understand, and correct faults in ontologies andare already implemented in popular ontology development tools likeProtege3 or Swoop4.

All currently suggested approaches for ontology debugging aimat the automated computation of a set of changes to the ontology1 University Klagenfurt, Austria, email: [email protected] Dortmund University of Technology, Germany, email: dietmar.jannach@u

do.edu3 http://www.co-ode.org4 http://code.google.com/p/swoop/

that restore the coherence of its terminology (diagnosis). In orderto accomplish this task efficiently, current diagnosis approaches arebased on the computation of axiom subsets that define an incoherentterminology (conflict sets).

Diagnosis techniques: Currently, two approaches are used for thecomputation of diagnoses in ontology debugging: Pinpointing [12]and Reiter’s model-based diagnosis (MBD) [10]. Pinpoints are usedto avoid the computation of minimal hitting sets of conflict setsby approximating minimal diagnoses by their supersets. However,the pinpoints themselves are computed on the basis of all mini-mal conflicts. In contrast, in MBD approaches (minimal) conflictsare computed on demand and diagnoses are computed with increas-ing cardinality by constructing a hitting-set tree (HSTREE). Conse-quently, this method will find those diagnoses first that suggest min-imal changes and avoids both the computation of very implausiblemulti-fault diagnoses and the costly computation of the set of all min-imal conflicts. Note that Reiter’s original proposal does not work cor-rectly for non-minimal conflicts [5] and shows limited performancefor non-minimal conflicts. A modified diagnosis method was how-ever introduced in [4] which avoids these deficits. The general ques-tion whether pinpoints or leading diagnoses are more appropriate asan output of the debugging process is still open.

Conflict computation: Current approaches like SINGLE JUST[8]and QUICKXPLAIN[4] either treat the underlying reasoner as ablack-box or a glass-box.

In (invasive) glass-box approaches the developer of the debuggingmethod can exploit specifics of the theorem prover. In [9], for in-stance, a conflict computation approach was proposed which requiresmodifications of existing reasoning algorithms as its aim is to com-pute sets of conflicts during the reasoning process as efficiently aspossible. The main drawback of such glass-box approaches howeveris that they can be used only for a particular description logic [1],like SHOIN (D) [9]. Hence, only a particular reasoner (or evena version of a reasoner) and a particular type of logic can be used.Moreover, glass-box modifications to reasoning systems often re-move existing optimizations and thus are typically slower then theirnon-modified analogues. In addition, glass-box approaches to con-flict set minimization do not guarantee the minimality of returnedconflict sets and further (black-box) minimization is required [8].

On the other hand, black-box algorithms are completely indepen-dent from the reasoning process and just use the boolean outputs ofthe theorem prover. These algorithms are therefore logic-independentand can exploit the full power of highly optimized reasoning meth-ods. Still, in case of an unsatisfiable set of axioms, all the axiomsare considered as a conflict since no further information is available.Conflicts are typically minimized by additional calls to a theoremprover. In order to make black-box approaches applicable in caseswhere theorem proving is expensive, the number of such calls must

7

be minimized.In current systems, two main approaches for conflict set compu-

tation are used, SINGLE JUST [8] and QUICKXPLAIN [4]. In gen-eral, both of them can be used in glass-box and black-box settings.In this paper we show that QUICKXPLAIN is preferable over SIN-GLE JUST in both settings based on a theoretical analysis of best andworst cases and an empirical performance evaluation for a simulatedaverage case. In addition, we propose modifications to the originalalgorithms to further improve the run-time performance in glass-boxsettings.

The reminder of the paper is organized as follows, Section 2 pro-vides theoretical study of conflict set computation methods and in-cludes a brief description of the main algorithms as well as the anal-ysis of extreme cases. In Section 3 we present the results of empiricalevaluation of QUICKXPLAIN and SINGLE JUST in both black- andglass-box settings. The paper closes with a discussion of the futurework.

2 COMPUTING MINIMAL CONFLICTS

We will focus on the comparison of two popular algorithms QUICK-XPLAIN [6] and SINGLE JUST [8]. The presented comparison ispossible because application scenarios and strategies of these algo-rithms are similar. Both methods are designed to compute only oneminimal conflict set per execution. The combinations of QUICKX-PLAIN + HSTREE [4] and SINGLE JUST + HSTREE (also referredas ALL JUST) [8] are used to obtain a set of minimal diagnoses di-agnoses or to enumerate minimal conflict sets (justifications). There-fore, QUICKXPLAIN and SINGLE JUST can be compared both theo-retically and empirically.

Algorithm: QUICKXPLAIN(B, C)Input: trusted knowledge B, set of axioms COutput: minimal conflict set CS

(1) if isCoherent(B ∪ C) or C = ∅ return ∅;(2) AX ← getFaultyAxioms(C);(3) if AX 6= ∅ then C ← C ∩AX;(4) return computeConflict(B, B, C)

function computeConflict(B, ∆, C)(5) if ∆ 6= ∅ and not isCoherent(B) then return ∅;(6) if |C| = 1 then return C;(7) int n← |C|; int k := split(n)(8) C1 ← {ax1, . . . , axk} and C2 := {axk+1, . . . , axn};(9) CS1 ← computeConflict(B ∪ C1, C1, C2);(10) if CS1 = ∅ then C1 ← getFaultyAxioms(C1);(11) CS2 ← computeConflict(B ∪ CS1, CS1, C1);(12) return CS ← CS1 ∪ CS2;

function getFaultyAxioms(C)(13) AX ← getConflictSet glassBox();(14) if AX = ∅ then return C;(15) else return AX;

Figure 1. Generalized QUICKXPLAIN algorithm

QUICKXPLAIN. This algorithm (listed in Figure 1) takes two pa-rameters as an input, B a set of axioms that are considered as cor-rect by a knowledge engineer and C a set of axioms, which shouldbe analyzed by the algorithm. QUICKXPLAIN follows a divide-and-conquer strategy and splits the input set of axioms C into two subsetsC1 and C2 on each recursive call. If the conflict set is a subset of ei-ther C1 or C2, the algorithm significantly reduces the search space.If for instance the splitting function is defined as split(n) = n/2then the search space will be reduced in half just with one call to areasoner. Otherwise, the algorithm re-adds some axioms ax ∈ C2 toC1. With the splitting function defined above, the algorithm will adda half of all axioms of the set C2.

The choice of the splitting function is crucial since it affects thenumber of required coherency checks. The knowledge engineer candefine a very effective splitting function for a concrete problem, e.g.,if there exists some a priori knowledge about faulty axioms of anontology. However, in the general case it is recommended to use thefunction that splits the set C of all axioms into two subsets of thesame size since the path length from the root of the recursion tree toa leaf will contain at most log2 n nodes. Thus, if the cardinality of thesearched minimal conflict set |CS| = k in the best case, i.e. when allk elements belong to a single subset C1, the number of required co-herency checks is log2

nk

+2k. The worst case for QUICKXPLAIN isobserved when the axioms of a minimal conflict set always belong todifferent sets C1 and C2, i.e., if for instance a minimal conflict sethas two axioms and one is positioned at the first place of set C andthe other one at the last. In this case the number of coherency checksis 2k(log2

nk

+ 1) [6].Note that we modified the original QUICKXPLAIN algorithm

(Figure 1) such that it can be used with both black- and glass-box approaches. The algorithm issues two types of calls to areasoner, isCoherent(T ) and getConflictSet glassBox(). Thefirst function returns true if the given terminology T is coherent.getConflictSet glassBox() returns a set of axioms AX that areresponsible for incoherence (CS ⊆ AX). This function can only beused if the reasoner supports glass-box debugging. If this is the casethe reasoner will able to return the set AX which was generated dur-ing the reasoning process. If only black-box usage is possible thena practical implementation of an ontology debugger should overridethis function with one that returns an empty set. In this case the mod-ified algorithm is equal to the original one given in [6].

Moreover, the first part of the algorithm (lines 1-4) is required tocheck if an ontology is actually incoherent. This check is requiredfor two reasons. First, the result of conflict set computation for an al-ready coherent ontology using a reasoner as a black-box will includeall axioms of this ontology; second, a feedback of a glass-box rea-soner executed at this stage can significantly reduce the search spaceof QUICKXPLAIN. The same can also be noted for SINGLE JUST.

SINGLE JUST This algorithm (see Figure 2) follows an expand-and-shrink strategy and has two main loops. The first one creates a setof CS that includes all axioms of the minimal conflict set and thesecond one minimizes CS by removing axioms that do not belong tothe minimal conflict set.

The algorithm includes two functions select(T ) andfastPruning(T ) that can be tuned to improve its performance.The first function starts by selecting a predefined number of axiomsnum from the given set. The number of axioms that are selected cangrow with a certain factor f (see [7]). The fastPruning functionimplements a pruning strategy for CS with a sliding windowtechnique. The pruning algorithm takes the size of the window

8

Algorithm: SINGLE JUST(B, C)Input: trusted knowledge B, set of axioms COutput: minimal conflict set CS

(1) if isCoherent(B ∪ C) or C = ∅ return ∅;(2) AX ← getFaultyAxioms(C);(3) if AX 6= ∅ then C ← C ∩AX;(4) return computeConflict(B, C)

function computeConflict(B, C)(5) CS ← B;(6) do(7) CS ← CS ∪ select(C \ CS);(8) while (isCoherent(CS));(9) CS ← fastPruning(getFaultyAxioms(CS));(10) for each ax ∈ CS do(11) CS ← CS \ {ax}(12) if isCoherent(CS) then CS ← CS ∪ {ax};(13) else CS ← getFaultyAxioms(CS);(14) return CS;

Figure 2. Generalized SINGLE JUST algorithm

window and the set of axioms CS as an input and outputs a set ofaxioms CS′ ⊆ CS. In the form it was implemented in OWL-API5,the pruning algorithm partitions the input set CS with n axioms intop = n/window parts Pi, i = 1, . . . , p and then sequentially testscoherency of each set CSi = CS \ Pi, i = 1, . . . , p. Note also thatOWL-API includes two variants of the pruning method, one withconstant and one with shrinking window size. In further analysisand evaluation we will consider only the variant with the constantwindow size.

Let us consider the best and worst cases for SINGLE JUST. Inthe best case, all axioms of a minimal conflict set CS belong tosome partition set Pi. Thus, given an axioms set C of cardinalityn that contains a minimal conflict set CS of cardinality k, the algo-rithm will make at most 1 + p + min(window, num) coherencychecks. In the worst case, the first iteration will require at leastlogf

`1− 1−f

numn

´coherency checks and both the sliding window

and final minimization p + min(k/p, 1)n checks, if all k axiomsof the minimal conflict set belong to different partitions Pi.

The theoretical analysis of the two algorithms thus shows thatQUICKXPLAIN has a smaller interval of possible number of co-herency checks in comparison to SINGLE JUST (see Figure 3)6.

Note also that the interaction with the reasoner used in SIN-GLE JUST in Figure 2 is organized in the same way as inQUICKXPLAIN, i.e., by means of the functions isConsistent andgetFaultyAxioms. However, if a black-box approach to ontologydebugging is used, the modified algorithm presented on the 2 is equalin terms of number of consistency checks to the original one sug-gested in [8]. Moreover, both generalized algorithms can also beused to detect conflict sets that cause unsatisfiability of a certainconcept. This is possible if we introduce one more input parame-

5 Unfortunately, the authors of the SINGLE JUST algorithm did not provide aspecification of the fast pruning method neither in [7] nor in [8]. Thereforewe analyzed the OWL-API (http://owlapi.sourceforge.net/ checked on June7, 2008) implementation that is referred by the authors in [8].

6 The values of SINGLE JUST parameters were taken from the OWL-APIimplementation.

10 100 103 104 105 106

QuickXPlain Single_Just

106 axioms

Number of coherency checks

104 axioms

102 axioms

1

Figure 3. Intervals for numbers of possible coherency checks required toidentify a minimal conflict set of cardinality k = 8 in an ontology of naxioms. QUICKXPLAIN parameters: split(n) = n/2. SINGLE JUST:number of axioms on the first iteration num = 50, increment factor

f = 1.25, window size window = 10.

ter Concept and rewrite the coherency checking function such thatisCoherent(C, Concept) returns false if Concept is unsatisfiablewith respect to the terminology defined by a set of axioms C. Other-wise this function should return true.

The algorithms presented on Figures 1 and 2 can also exploit thestructural relations between axioms by means of specifically imple-mented functions split, select and fastPruning. One can thus forinstance select and/or partition axioms so that axioms with intersect-ing sets of concepts will be considered first.

3 EMPIRICAL EVALUATIONThe theoretical analysis of the algorithms showed that QUICKX-PLAIN is preferable over SINGLE JUST since it has much lower vari-ation of the number of required reasoner calls. Nevertheless, the ex-tremum conditions of the discussed best and worth cases are ratherspecific. Therefore, an analysis of the average case has to be donein order to make the comparison complete. However, evaluating thiscase is problematic, since there are no publicly available collectionsof incoherent ontologies that are published on the Web and are suit-able for such tests. Moreover, there is no a priori knowledge on thedistribution of conflicts. In other words, we do not know how thefaulty axioms are most often positioned in an ontology. Therefore, wesimulated the occurence of faults in the ontology in order to obtaina measure of the numbers of coherency checks required by QUICK-XPLAIN and SINGLE JUST. These statistics can be then used to cal-culate the average number of required coherency checks. Moreover,for our purposes it is enough to generate and then compute only oneconflict set, since none of the analyzed algorithms can improve itsperformance on subsequent executions by using data from the previ-ous runs.

The test case generation method was designed under the follow-ing assumptions: (1) All axioms have the same probability to be partof a conflict set (uniform distribution). Thus, for an ontology with naxioms, the probability for each axiom to be a source of a conflictis 1/n. (2) The cardinalities of minimal conflict sets follow the bi-nomial distribution with the number of trials t equal to the maximallength of the dependency path from an axiom selected according tothe first assumption to all axioms that directly or indirectly depend onconcepts declared in the selected axiom. The value t corresponds tothe maximum possible cardinality of a conflict that can be generatedfor a selected axiom. The success probability, which is the second

9

MyGrid

Sweet-JPL

BCS3

Galen

Bike9

MGED Ontology

Gene Ontology

Sequence Ontology

0 50 100 150 200

QuickXplain Find_Just

MyGrid

Sweet-JPL

BCS3

Galen

Bike9

MGED Ontology

Gene Ontology

Sequence Ontology

0 10 20 30 40 50


MyGrid

Sweet-JPL

BCS3

Galen

Bike9

MGED Ontology

Gene Ontology

Sequence Ontology

0 50 100 150 200

QuickXplain Single_Just

MyGrid

Sweet-JPL

BCS3

Galen

Bike9

MGED Ontology

Gene Ontology

Sequence Ontology

0 10 20 30 40 50


coherency checks

Figure 4. Average number of consistency checks for QUICKXPLAIN andSINGLE JUST using reasoner as a black-box

parameter of the distribution, is set to 1/t. Hence minimal conflictsets of smaller cardinality are more likely to appear.

The process starts with the generation of a uniformly distributednumber i for the interval [1, n]. This number corresponds to an ax-iom axi that is the initial axiom of a conflict set. Then the algo-rithm queries a reasoner for a taxonomy of a given ontology tofind out all axioms that contain concept definitions that are eitherdirectly or indirectly dependent on a concept defined in axi. Thelength of a longest dependency path t is then used to generate thevalue c, which corresponds to the minimal cardinality of a conflictset according to the second assumption. Next, we retrieve all axiomswhich define concepts that subsume one of the concepts declared inaxi such that subsumption is made over the c − 1 other concepts(C1 v ... v Cc−1 v Cc). If more then one axiom is retrieved thenwe select randomly one of them (denoted as axj). Both axioms aremodified to create a conflict (e.g. by inserting a new concept defi-nition in axi and its negation in axj). Thus, the generation methodgenerates faults that correspond to local or propagated causes of un-satisfiability that were observed by Wang et al [14] in many real-world ontologies.

Note that the real cardinality of a minimal conflict is unknownprior to the execution of a conflict compution algorithm, since we donot investigate all possible dependencies of the modified axioms.

In the tests we used Pellet 1.5.17 as a reasoner and the SSJ sta-tistical library8 to generate the required random numbers. As can beseen in Figure 4 and Figure 5, in the average case (after 100 simu-lations) QUICKXPLAIN outperformed SINGLE JUST in all eight testontologies MyGrid (8179 axioms), Sweet-JPL (3833 axioms), BCS3(432 axioms), Galen (3963 axioms), MGED Ontology (236), Bike9(215), Gene Ontology (1759) and Sequence Ontology (1745). In thistest we measured both the number of checks and the elapsed time.

Note also that the results that we obtained when using Pellet can ingenerally also be transferred to other reasoners that – in these settingshave shown to have comparable performance [13]. All experimentshave been performed on a MacBookPro (Intel Core Duo) 2 GHz with2 GB RAM and 1.5 GB maximum Java memory heap size.

Beside using the reasoner as a black-box, both variants of QUICK-XPLAIN and SINGLE JUST can also be used in glass-box settings.However, the theoretical analysis of these cases is not trivial, since

7 http://pellet.owldl.com/8 http://www.iro.umontreal.ca/∼simardr/ssj/indexe.html

MyGrid

Sweet-JPL

BCS3

Galen

Bike9

MGED Ontology

Gene Ontology

Sequence Ontology

0 50 100 150 200


MyGrid

Sweet-JPL

BCS3

Galen

Bike9

MGED Ontology

Gene Ontology

Sequence Ontology

0 10 20 30 40 50


MyGrid

Sweet-JPL

BCS3

Galen

Bike9

MGED Ontology

Gene Ontology

Sequence Ontology

0 50 100 150 200


MyGrid

Sweet-JPL

BCS3

Galen

Bike9

MGED Ontology

Gene Ontology

Sequence Ontology

0 10 20 30 40 50


milliseconds

Figure 5. Average running times for black-box QUICKXPLAIN andSINGLE JUST

we cannot predict the number of axioms that will be returned by aglass-box method on each iteration of the conflict computation al-gorithm. The computed conflict set can include extra axioms thatdo not belong to the searched minimal conflict set because of non-deterministisms of a reasoning algorithm such as max-cardinality re-strictions or special features of the tracing algorithm itself. Therefore,our empirical evaluation of different combinations of QUICKXPLAIN

and SINGLE JUST is based on two different glass-box implementa-tions: invasive [9] and naıve non-invasive.

In general, all glass-box methods implement tracing of axioms thatwere used by the reasoner to prove the unsatisfiability of a concept.The invasive method first stores all correspondences between sourceaxioms and internal data structures of the reasoner and tracks thechanges in internal data structures during the normalization and ab-sorption phases (see [9] for details). In [9], it is also suggested to addtracing to the SHOIN (D) tableaux expansion rules to enable a veryprecise tracking of axioms so that the resulting axioms set will be assmall as possible. The main drawback of this approach however isthat such a modification disables many key optimizations, which arecritical for the excellent performance of modern OWL reasoners [8].

In the non-invasive approach that we developed for our evaluation,we only track which concepts were unfolded by the reasoner and thensearch for all axioms in which these concepts are defined using theOWL-API. This method does not analyze the details of the reasoningprocess and thus, the resulting set of axioms is only in the best caseequals to the set returned by the invasive method. However, such anapproach can have a shorter execution time, since it does not requirechanges in the optimized reasoning process except for the insertionof a logging method for unfolded concepts.

Pellet 1.5.1 already includes an implementation of the invasivemethod (explanations of clashes) and can also be configured to turnon logging which is required for the non-invasive method. The onlymodification to the reasoner was to add a fast fail behavior in thesatisfiability check. By default, Pellet searches for all unsatisfiableconcepts. However, for the minimal conflict set computation algo-rithm it is enough to find just one such concept, since in this case theterminology is already incoherent.

We performed the tests of the glass-box methods using the sametest bundle that was used for the black-box tests. The evaluationshows that QUICKXPLAIN is faster in both approaches (see Figures 6and 7). When using the feedback from the glass-box satisfiabilitycheck, QUICKXPLAIN performed better then SINGLE JUST in all

10

MyGrid

Sweet-JPL

BCS3

Galen

Bike9

MGED Ontology

Gene Ontology

Sequence Ontology

0 3 6 9 12 15


MyGrid

Sweet-JPL

BCS3

Galen

Bike9

MGED Ontology

Gene Ontology

Sequence Ontology

0 3 6 9 12 15


MyGrid

Sweet-JPL

BCS3

Galen

Bike9

MGED Ontology

Gene Ontology

Sequence Ontology

0 50 100 150 200


MyGrid

Sweet-JPL

BCS3

Galen

Bike9

MGED Ontology

Gene Ontology

Sequence Ontology

0 10 20 30 40 50


milliseconds

Figure 6. Average running times for non-invasive glass-box

MyGrid

Sweet-JPL

BCS3

Galen

Bike9

MGED Ontology

Gene Ontology

Sequence Ontology

0 3 6 9 12 15


MyGrid

Sweet-JPL

BCS3

Galen

Bike9

MGED Ontology

Gene Ontology

Sequence Ontology

0 3 6 9 12 15


MyGrid

Sweet-JPL

BCS3

Galen

Bike9

MGED Ontology

Gene Ontology

Sequence Ontology

0 50 100 150 200


MyGrid

Sweet-JPL

BCS3

Galen

Bike9

MGED Ontology

Gene Ontology

Sequence Ontology

0 10 20 30 40 50


milliseconds

Figure 7. Average running times for invasive glass-box

the cases. Note also that the difference in the average running timesfor QUICKXPLAIN and SINGLE JUST in invasive and non-invasiveglass-box settings is not significant as both glass-box methods canin general reduce the search space of minimal conflict computationalgorithms very rapidly.

4 CONCLUSIONS & FUTURE WORKAdequate debugging support is an important prerequisite for thebroad application of ontologies in real-world scenarios and in recentyears, different techniques for the automated detection of problem-atic chunks in the knowledge bases have been developed.

One of the most critical and time-intensive tasks in most debug-ging approaches is the detection of small sets of axioms that containthe faults (conflict sets).

In general, efficient conflict computation and minimization is cen-tral not only in debugging scenarios, as conflict sets are also helpfulto compute justifications for axioms and assertions which in turn canserve as a basis of an explanation facility [8].

In this paper we have analyzed two recent proposals for the iden-tification of conflicts, both in black-box and glass-box applicationscenarios. Both the theoretical analysis as well as an empirical eval-uation showed that QUICKXPLAIN is currently the more efficientmethod for that purposes.

Due to the lack of publicly available mass data about typical ontol-ogy faults, artificial tests had to be used in the experiments. For future

work, it would be useful, if ontology editors like Protege or Swoopwould support anonymous user feedback for debugging purposes, asstatistics on the number of conflict sets and their average cardinality.This data would help to make even more precise evaluations of theaverage case.

Finally, note that in the context of this work we generally un-derstand axioms as valid description logics statements of any kind.Concept definitions where the left-hand sides are atomic, are themost frequent form of axioms, since available ontology editors sup-port mainly this presentation. However, if the terminology includesaxioms with a different structure, approaches like those presentedin [3, 7] can be used to transform the axioms. These approaches sup-port fine-grained debugging of ontologies, will allow us to locatefaults within parts of the original axioms. Although the evaluationin this paper was limited to the more coarse-grained case, the con-flict computation techniques can be applied also for the fine-graineddebugging approaches.

REFERENCES[1] The Description Logic Handbook: Theory, Implementation, and Ap-

plications, eds., Franz Baader, Diego Calvanese, Deborah L. McGuin-ness, Daniele Nardi, and Peter F. Patel-Schneider, Cambridge Univer-sity Press, 2003.

[2] Mathieu d’Aquin, Claudio Baldassarre, Laurian Gridinoc, Sofia An-geletou, Marta Sabou, and Enrico Motta, ‘Watson: A gateway for nextgeneration semantic web applications’, in Poster session of the Interna-tional Semantic Web Conference, ISWC, (2007).

[3] Gerhard Friedrich, Stefan Rass, and Kostyantyn Shchekotykhin, ‘Ageneral method for diagnosing axioms’, in DX’06 - 17th InternationalWorkshop on Principles of Diagnosis, eds., C.A. Gonzalez, T. Escobet,and B. Pulido, pp. pp. 101–108, Penaranda de Duero, Burgos, Spain,(2006).

[4] Gerhard Friedrich and Kostyantyn Shchekotykhin, ‘A General Diagno-sis Method for Ontologies’, Proceedings of the 4 thInternational Se-mantic Web Conference (ISWC-05), 232–246, (2005).

[5] Russell Greiner, Barbara A. Smith, and Ralph W. Wilkerson, ‘A correc-tion to the algorithm in Reiter’s theory of diagnosis’, Artificial Intelli-gence, 41(1), 79–88, (1989).

[6] Ulrich Junker, ‘QUICKXPLAIN: Preferred explanations and relax-ations for over-constrained problems.’, in Association for the Advance-ment of Artificial Intelligence, pp. 167–172, San Jose, CA, USA,(2004).

[7] Aditya Kalyanpur, Debugging and repair of OWL ontologies, Ph.D.dissertation, University of Maryland, College Park, MD, USA, 2006.Adviser-James Hendler.

[8] Aditya Kalyanpur, Bijan Parsia, Matthew Horridge, and Evren Sirin,‘Finding all justifications of OWL DL entailments’, in Proc. ofISWC/ASWC2007, Busan, South Korea, volume 4825 of LNCS, pp.267–280, Berlin, Heidelberg, (November 2007). Springer Verlag.

[9] Aditya Kalyanpur, Bijan Parsia, Evren Sirin, and James Hendler, ‘De-bugging unsatisfiable classes in OWL ontologies’, Web Semantics: Sci-ence, Services and Agents on the World Wide Web, 3(4), 268–293,(2005).

[10] Raymond Reiter, ‘A theory of diagnosis from first principles’, ArtificialIntelligence, 23(1), 57–95, (1987).

[11] Stefan Schlobach, ‘Diagnosing terminologies.’, in Proc of AAAI, eds.,Manuela M. Veloso and Subbarao Kambhampati, pp. 670–675. AAAIPress / The MIT Press, (2005).

[12] Stefan Schlobach, Zhisheng Huang, Ronald Cornet, and Frank Harme-len, ‘Debugging incoherent terminologies’, J. Autom. Reason., 39(3),317–349, (2007).

[13] Evren Sirin, Bijan Parsia, Bernardo Cuenca Grau, Aditya Kalyanpur,and Yarden Katz, ‘Pellet: A practical OWL-DL reasoner’, Technicalreport, UMIACS Technical Report, (2005).

[14] H. Wang, M. Horridge, A. Rector, N. Drummond, and J. Seidenberg,‘Debugging OWL-DL Ontologies: A Heuristic Approach’, Proceed-ings of the 4 thInternational Semantic Web Conference (ISWC-05),745–757, (2005).

11

Supporting Conceptual Knowledge Capture ThroughAutomatic Modelling

Jochem Liem and Hylke Buisman and Bert BredewegHuman Computer Studies Laboratory, Informatics Institute, Faculty of Science,

University of Amsterdam, The Netherlands.Email: {jliem,bredeweg}@science.uva.nl, [email protected]

Abstract. Building qualitative models is still a difficult and lengthyendeavour for domain experts. This paper discusses progress towardsan automated modelling algorithm that learns Garp3 models basedon a full qualitative description of the system’s behaviour. In contrastwith other approaches, our algorithm attempts to learn the causalitythat explains the system’s behaviour. The algorithm achieves goodresults when recreating four well-established models.

1 Introduction

In this paper we focus on the ground work required to advance to-wards an automated modelling program. The input is considered tohave a qualitative representation, i.e. a state graph that represents thepossible situations that can emerge from a system, and the values ofthe quantities in each situation. Furthermore, the input is assumed tohave no noise nor any inconsistencies. The completed algorithm isenvisioned to support researchers in articulating their conceptual un-derstanding. As such it will help to establish theories that explain thephenomena provided as input data.

2 Related Work

Recently, researchers in the machine learning community proposedinductive process modellingas a new research agenda [4]. They ar-gue that models should not just be accurate, but should also pro-vide explanations (often requiring variables, objects and mechanismsthat are unobserved). In their work, quantitative process models arelearned from numerical data. Based on changing continuous vari-ables (observations), a set of variables, a set of processes (describedby generalized functional forms), and constraints such as variabletype information, a specific process model is generated that explainsthe observed data and predicts unseen data accurately. As in our ap-proach, there is an explicit notion of individual processes, variables(quantities) and subtype hierarchies to represent different types.

Our approach differs from this work in two ways. Firstly, we learnqualitative models based on qualitative data, making our approach aviable alternative when no numerical data is available. Secondly, ourapproach represents causality more explicitly through causal depen-dencies. We argue that this representation provides a better explana-tion than equations. However, our generated models cannot performnumerical predictions.

An earlier approach to learning qualitative models isQualitativeInduction(QUIN) [1]. QUIN searches for qualitative patterns in nu-meric data and outputs the result as ”qualitative trees” (similar to de-

cision trees). The choices in the qualitative tree can be seen as condi-tional inequalities for specific model fragments in our approach. Aswith the inductive process modelling approach equations are usedto represent the causality, in this case Qualitative Differential Equa-tions (QDEs). Similar work to QUIN learns models for the JMorvenlanguage, which uses fuzzy quantity spaces to specify variables [7].However, this work also uses QDEs, which leaves the representationof causality implicit.

3 QR Model and Simulation Workbench: Garp3

The automatic model building algorithm is implemented in Garp31

[2]. Garp3 allows modellers to represent their knowledge about thestructure and the important processes in their system asmodel frag-ments, which can be considered formalisations of the knowledge thatapplies in certain general situations.

Next to model fragments, differentscenarioscan be modelled.These represent specific start states of a system. Garp3 can run sim-ulations of models based on a particular scenario. The result of sucha simulation is a state graph, in which each state represents a particu-lar possible situation of the system, and the transitions represent thepossible ways a situation can change into another.

The simulation engine takes a scenario as input, and finds all themodel fragments that apply to that scenario. The consequences ofthe matching model fragments are added to the scenario to create astate description from which new knowledge can be inferred such asthe derivatives of quantities. Given the completed state description,the possible successor states are inferred. The complete state graphis generated by applying the reasoning to the new states.

In Garp3 the structure of a system is represented usingentities(objects) andconfigurations(relations). For example, a lion huntingon a zebra would be represented as two entities (lion and zebra) anda configuration (hunts).

Quantitiesrepresent the features of entities and agents that changeduring simulation. A quantity has a magnitude and a derivative, re-presenting its current value and trend. The magnitude and derivativeare each defined by a quantity space that represents the possible val-ues the magnitude and the derivative can have. Such a quantity spaceis defined by a set of alternatingpoint andinterval values.

We useMv(Q1) to refer to the current value of the magnitudeof a quantity.Ms(Q1), the sign of the magnitude, indicates whetherthe magnitude is positive, zero or negative (Ms(Q1) ∈ {+, 0,−}).

1 http://www.garp3.org

13

Dv(Q1) refers to the current value of the derivative of a quantity,which has a value from the predefined derivative quantity space(Dv(Q1) ∈ {−, 0, +}). Ds(Q1) refers to the current sign of aderivative. Note that the predefined values of derivatives completelycorrespond to the possible signs of the derivative.

3.1 Causality

Garp3 explicitly represents causality using indirect and direct influ-

ences. Direct influences are represented asQ1I+→ Q2. Influences can

be either positive or negative. The positive influence will increaseDv(Q2) if Ms(Q1) = +, decrease it ifMs(Q1) = −, and have noeffect whenMs(Q1) = 0. For a negative influence, it is vice versa.

The indirect influences, calledproportionalities, are represented as

Q1P+

→ Q2. Similar to influences, proportionalities can be either pos-itive or negative. The positive proportionality will increaseDv(Q2)if Ds(Q1) = +, have no effect if it is stable, and decrease if it isbelow zero. For a negative proportionality, it is vice versa.

3.2 Other Behavioural Ingredients

Other behavioural ingredients in Garp3 are operators, inequalities,value assignments and correspondences. Operators (+ and -) are usedto calculate the magnitude value of quantities (e.g.Q1 − Q2 = Q3,to indicateMv(Q1) − Mv(Q2) = Mv(Q3)). Inequalities can beplaced between different model ingredient types: (1) magnitudes(Mv(Q1) = Mv(Q2)), (2) derivatives (Dv(Q1) < Dv(Q2), (3)valuesQ1(point(Max)) = Q2(point(Max)), (4) operator rela-tions (Mv(Q1) − Mv(Q2) < Mv(Q3) − Mv(Q4), (5) combina-tions of the 1, 2, 3 and 4 (although only between either magnitudeor derivative items). Value assignments simply indicate that a quan-tity has a certain qualitative value (Mv(Q1) = Q1(P lus)). Finally,correspondences indicate that from certain values of one quantity,values of another quantity can be inferred. There are quantity corre-

spondences (Q1Qqs

↔ Q2) and value correspondences (Q1(P lus)Qv→

Q2(P lus)), which can both be either directed or undirected. Thevalue correspondence indicates that ifMv(Q1) = Q1(P lus),Mv(Q2) = Q2(P lus). If the value correspondence is bidirectional,the reverse inference is also possible. Quantity correspondences canbe considered a set of value correspondences between each consecu-tive pair of the values of both quantities. There are also inverse quan-

tity space correspondences (Q1

Q−1

qs

↔ Q2) that indicate that the firstvalue inQ1 corresponds to the last value inQ2, the second to the onebefore last, etc.

4 Algorithm Requirements and Approach

4.1 Assumptions and Scoping

The goal of the automatic model building algorithm is to take a stategraph and a scenario as input, and generate the model that providesan explanation for the behaviour. Our approach focusses on the gen-eration of causal explanation. Several assumptions are made to scopethe work. In further research these assumptions can be alleviated.Firstly, input is assumed to have no noise or inconsistencies. Sec-ondly, the state graph is assumes to be a full envisionment of thesystem’s behaviour.

The second assumption is that a model can be build using a singlemodel fragment. From a causal explanation point of view, it is rea-sonable to assume that influences and proportionalities never disap-

pear, but that their effects are only nullified when quantities becomezero or stable.

Thirdly, the algorithm is focussed on causal explanation and lesson structure. Therefore, the entity hierarchy is assumed known.

4.2 Input and Output

The algorithm takes a complete state graph as input, which includes(1) the quantity names, (2) the quantity spaces, (3) the magnitudesand derivatives of the quantities in different states, (4) the observableinequalities, and (5) the state transitions. Furthermore, the algorithmis provided with the scenario that should produce the state graph,which consists of: (1) the entities, agents and assumptions involved,(2) structural information about the configurations between them, (3)the quantities and their initial values, and (4) the inequalities that holdin the initial state.

The output of the algorithm is one or more Garp3 qualitative mod-els that explain (are consistent with) the input that can be immedi-ately simulated.

4.3 Algorithm Design Approach

Since the semantics of model ingredients are formally defined, onewould assume that it is clear how each ingredient manifests itselfin the simulation results of a model. Otherwise, how would the im-plementation of a simulation engine have been possible? However, inpractice, it is hard even for expert modellers to pinpoint the model in-gredients that are responsible for certain (lack of) behaviour. This hasseveral reasons. Firstly, a large set of inequalities are derived duringqualitative simulation, of which the implications (other inequalities)are difficult to foresee. Secondly, the engine has a lot of intricacies(such as second order derivatives) which makes simulation resultshard to predict. Thirdly, the branching in the state graph that resultsfrom ambiguity is difficult for people to completely envision.

For these reasons, an iterative algorithm design approach is cho-sen. Well-established models are ordered by complexity, and at-tempts are made to generate them using their own output. Each ofthe models requires a different (and increasingly large) set of consid-erations that must be dealt with.

The models chosen are Tree and Shade, Communicating Vessels,Deforestation, Population Dynamics and a set of other even morecomplex models2 Tree and Shade is the least complex model, con-taining only a few quantities, and causal dependencies, and no condi-tions, causal interactions, inequalities or operator relations. Commu-nicating vessels is more complex, as it contains causal interactions,an operator, and inequalities. The deforestation model is differentfrom the previous models as it contains many clusters linked to eachother by proportionalities. Population dynamics is again more com-plex, due to the large amount of quantities, interactions and condi-tions.

4.4 Causality and Clusters

4.4.1 Causal Paths

Important for the algorithm is the concept ofcausal paths. These areseries of quantities connected by influences and proportionalities. Acausal path is defined as a set of quantities that starts with an influ-ence, and is followed by a arbitrary number of proportionalities. For

example:Q1I+→ Q2

P+

→ . . .P−

→ Qn−1P+

→ Qn. A quantity that has no

2 Themodels and references to articles are available at http://www.garp3.org

14

proportionalities leading out of it ends the causal path. If a quantityhas more than one proportionality leading out of it, multiple causalpaths can be defined.

Since each influence represents the causal effect of a process, acausal path can be seen as the cascade of effects of a process. Giventhis perspective, certain successions of causal relations become un-

likely. For example the causal pathQ1I+→ Q2

I+→ Q3

P−

→ Q4I+→ Q5

would imply there are many active processes with short or no cas-cading effects.

4.4.2 Direction of Causality

An important issue in scientific enquiry is the problem of correla-tion and causality. This issue appears when trying to derive causalrelations from the state graph. For example,Ds(Q1) = Ds(Q2)

can be an caused byQ1P+

→ Q2, Q2P+

→ Q1, or evenQ3P+

→ Q1

and Q3P+

→ Q2. Another example of this is in the communicat-ing vessels model. Ideally, a model capturing the idea of a con-tained liquid would distinguish between Volume, Height and Bot-

tom pressure, and have a particular causal account (V olumeP+

→

HeightP+

→ Bottom pressure). However, from the model’s be-haviour this causality may not be derivable, e.g. when the width ofthe containers doesn’t change. As a result, the unique role of thequantities involved can only be inferred when the required varia-tion for that is apparent in the input state-graph. Therefore, it is con-sidered the modeller’s responsibility to provide simulation exampleswhich will allow the algorithm the make these critical distinctions.However, it can be considered the responsibility of the tool to indi-cate to the modeller that the causality between certain sets of quanti-ties cannot be derived, and that examples showing these differencesshould be provided.

4.4.3 Clusters

The algorithm makes use of a specific subset of causal paths calledclusters. We define clusters as groups of quantities that exhibit“equivalent” behaviour. More specifically, a set of quantities con-

stitute a cluster if their values either correspond (Q1Qqs

↔ Q2) or

inversely correspond (Q1Q−1

qs

↔ Q2) to each other. Additionally, thecorresponding derivatives should be equal (Dv(Q1) = Dv(Q2)),while inversely corresponding derivatives should be each other’s in-verse (Dv(Q1) = −Dv(Q2)).

A further constraint is that the corresponding quantities (notinverse) in a cluster must be completely equivalent. Therefore,Mv(Q1) = Mv(Q2) must always hold. If an inequality holds be-tween two quantities, they are considered not to belong to the samecluster.

During implementation it became obvious that clusters are notmeaningful when quantities within a cluster belong to different en-tities. The reason for this originates from the idea of ‘no functionin structure’ [5]. Clusters involving multiple entities would inte-grate causality across individual structural units, which is undesired.Therefore, clusters can only contain quantities that belong to thesame entity.

Quantities cannot be a member of more than one cluster. IfQ1

andQ2 are in a cluster, andQ1 andQ3 are in a cluster, thenQ1,Q2 andQ3 must be in the same cluster. After all, ifQ1 andQ2 haveequivalent behaviour, andQ1 andQ3 have equivalent behaviour, bytransitivityQ2 andQ3 have to exhibit equivalent behaviour.

4.5 Minimal Covering

The key requirement of the model building algorithm is that it ex-plains the input behaviour. However, a second requirement is that thealgorithm does not contain redundant dependencies. That is, the al-gorithm should return the minimal set of dependencies that explainsthe behaviour.

Two dependencies are consideredsubstitutionaryif they have thesame effect on the simulation result (i.e. removing one of them wouldhave no effect, however removing both would).Complementaryde-pendencies are responsible for different aspects of the behaviour, andboth have to be present to explain the data. The aim is to create analgorithm that is minimally covering, i.e. it should only contain com-plementary dependencies.

5 Algorithm

5.1 Finding Naive Dependencies

The goal of this step is to find (non-interacting) dependencies thatare valid throughout the entire model (i.e. are not conditional). Thesecausal relations are callednaive dependencies, and provide the basisfor the rest of the algorithm.

5.1.1 Consistency Rules

Naive dependencies are identified using consistency rules. Each pairof quantities is checked using these rules to determine which of thempotentially holds throughout the state graph. These rules make use ofMv(Qx), Ms(Qx), Dv(Qx), Ds(Qx) of each quantity in a pair, andinequalities that hold between them. These statements are referred toas thestate informationof a quantity.

The consistency rules are derived from the semantics of the causaldependencies (see Section on Garp3). Examples of rules (that shouldhold throughout the state graph) are:

Q1I+→ Q2 if Ms(Q1) = Ds(Q2) (1)

Q1I−→ Q2 if Ms(Q1) = −Ds(Q2) (2)

Q1P+

→ Q2 if Ds(Q1) = Ds(Q2) (3)

Q1P−

→ Q2 if Ds(Q1) = −Ds(Q2) (4)

Q1(Vx)Qv↔ Q2(Vy) if

Mv(Q1) = Q1(Vx) =⇒ Mv(Q2) = Q2(Vy) (5)

Q1Qqs

↔ Q2 if ∀Vn(Q1(Vn)Qv↔ Q2(Vn)) (6)

5.1.2 Redundancy

The set of dependencies that are found contain a lot of redundancy,i.e. many dependencies are substitutionary. For example, in the com-

municating vessels modelheightP+

→ pressure, can be substituted

by pressureP+

→ height. The remainder of the algorithm selects thecorrect substitutionary groups, and uses the selected naive dependen-cies to derive more complex dependencies.

5.2 Determining Clusters

This step tries to determine clusters within the set of naive dependen-cies. The algorithm searches for quantities belonging to the same en-tity that exhibit equivalent behaviour, and tries to expand these candi-date clusters by adding other quantities. Quantities are only added if

15

they exhibit behaviour equivalent to the quantities already containedin the candidate cluster. If no more quantities can be added to a can-didate cluster, the algorithm searches for other candidate clusters. Byonly considering models composed of clusters, the space of possiblemodels is significantly reduced.

The validity of the candidate clusters is checked by determiningif there is overlap between the clusters. All clusters that overlap areremoved. An alternative would be to only remove clusters until nomore overlap is present. However, in practice no situations were en-countered where this was desirable. An example of a found clusteris volume, height and pressure in the communicating vessels model.Note that these clusters are still missing influences (their actuators),these are determined later in the algorithm.

5.3 Generating Causal Paths

This step returns the possible causal orderings within clusters basedon the cluster and naive dependencies sets. For each cluster a validcausal ordering is returned. Through backtracking other possible or-derings are generated.

The quantities in a cluster can be either connected in a linear fash-

ion (Q1P+

→ Q2P+

→ Q3) or using branching (Q1P+

→ Q2 andQ1P+

→Q3). The algorithm prefers linear branching, as branching does notoften occur in practice. Additionally, the reduction of possible mod-els is a significant advantage.

Another constraint that reduces the number of possible models isrequiring clusters that belong to entities of the same type to have the

same causal ordering. For example, if for one containerV olumeP+

→

HeightP+

→ Pressure, than for other containers the same causalordering must hold.

5.4 Actuating Clusters

The goal of the actuating clusters step is to connect clusters by iden-tifying cluster actuations. This step takes the set of clusters with es-tablished causal orderings and the naive dependencies as input.

Clusters can either be actuated by another cluster, or act as an actu-ator itself. Furthermore, clusters can be connected by propagating anactuation. In a model, each cluster should take part in at least one ofthese kind of relations such that all clusters are related in a way. Oth-erwise, the model would include two separate non-interacting sub-systems.

When one cluster actuates another, there is an influence relationbetween the two. Actuations are the most important form of con-necting clusters, since these connections are the cause of change inthe system. They are also the easiest to detect, due to the specificway influences manifest themselves in the state information. For thisreason, actuations by influences are identified first. Two types of ac-tuations though influences are distinguished: (1)equilibrium seekingmechanisms(ESM) and (2)external actuators.

5.4.1 Equilibrium Seeking Mechanisms

ESMs are better known asflows, and are common in qualitative mod-els. Flows cause two unequal quantities to equalize. The flow in thecommunicating vessels model has a non-zero value when the pres-sures in the two containers are unequal. The flow changes the volumeof the containers, and thus the pressures to equalize. An ESM holdsunder the following two conditions: (1)Q1 = Q2−Q3, whereQ1 ∈C1, Q2 ∈ C2, Q3 ∈ C3, where theC ’s are clusters, and (2)

Q4I−→ Q5 andQ4

I+→ Q6, whereQ4 ∈ C1, Q5 ∈ C2, Q6 ∈ C3.

Note that in many casesQ1 = Q4, such as in the communicatingvessels model.

5.4.2 Finding Calculus Relations

The algorithm reduces the search space of finding ESMs using fourconstraints. Firstly, all quantities involved in the operator should bein different clusters (C1, C2 andC3 are unequal). Secondly, the setof naive dependencies should at least contain one influence fromQ1

(to serve as an actuation). Thirdly, bothQ2 andQ3 would be at theend of the causal paths within their cluster, as in most cases this is themost meaningful interpretation. Finally,Q2 andQ3 are required tobe of the same type, as only things of the same type can be subtracted.

5.4.3 External Actuators

External actuators are causes of change more at the edges of the sys-tem compared to ESMs. To identify external actuators, the algorithmconsiders the influences in the naive dependencies that are not partof an ESM. Again, the minimal covering principle is applied to keepthe number of dependencies to a minimum. As a result a cluster willnever have more than one incoming actuation.

An actuation is only considered betweenC1 to C2 if the set ofnaive dependencies contains influences between each possible pair

of quantities, such that∀Qx ∈ C1,∀Qy ∈ C2(QxI+→ Qy). This

removes the influences in the set of naive dependencies that are con-sistent with the behaviour by chance.

Alternative actuations are returned through backtracking. In thefuture, actuations may be chosen based on the structure of the system,as causal relations are more likely to occur parallel to structurallyrelated entities.

5.4.4 Feedback

A common pattern in qualitative models is feedback, which is a pro-portionality originating from the end of a causal path to the quantityactuating the causal path. Feedbacks are simply added if the naivedependencies contain one. The algorithm always adds feedback atthe end of causal paths, since this is what happens in the investigatedmodels. However, it could be the case that feedbacks from halfway acausal chain are also possible.

5.5 Linking Clusters by Propagation

This step connects the clusters that have not yet been connectedthrough proportionalities, based on the naive dependencies. As withclusters, the causal ordering of the clusters cannot be distinguished.Therefore all possibilities are generated. Furthermore, the same de-sign choices as with finding causal paths within clusters have beenmade. Only linear orderings of clusters are allowed (i.e. no branch-ing).

5.6 Setting Initial Magnitudes

An influence has no effect if the magnitude of the quantity fromwhich it originates is unknown. Therefore this step assigns initialvalues to quantities. Note that this step first generates a set of candi-date assignments. When a value can be derived in another way thanthrough assignment, it is removed from the set of value assignmentcandidates.

16

There are six ways to assign initial magnitudes. Firstly, if a valueassignment for the quantity is present in the scenario, it requires noinitialisation. Secondly, if the magnitude can be derived through acorrespondence, the value is known. Thirdly, the result of a minusoperator can be derived if an inequality between its arguments isknown. Based on the possible magnitudes of the result this inequal-ity can be derived. Either this inequality is present in the scenario,or multiple inequalities should be made assumable by adding themas conditions in multiple model fragments. Garp3 automatically as-sumes unprovable values and inequalities if they are conditions inmodel fragments. Note that generating the conditional inequalities iscurrently beyond the scope of the algorithm, as it involves addingmodel ingredients to multiple model fragments. Fourthly, it is pos-sible that a certain magnitude holds everywhere throughout the stategraph. In this case, a value assignment is added as a (conditionless)consequence. Fifthly, a value could hold under certain conditions.However, this would require a value assignments with a conditionalinequalities in separate model fragments. Therefore, it is currentlybeyond the scope of the algorithm. Finally, multiple model fragmentscould be created in which the magnitudes are present as conditions.Garp3 will generate the different states that would result by assum-ing each of the values. As with the conditional value assignments,having value assignments as conditions in multiple model fragmentsis currently beyond the scope of the algorithm.

5.7 Dependency Interactions

This step identifies dependency interactions (influences or propor-tionalities) based on the input behaviour. Dependency interactionsare detected in the same way as naive dependencies, i.e. using a setof consistency rules. Interactions are not found as naive dependen-cies, as the individual dependencies are not consistent with theentirestate graph (as an interaction results in more behaviour than a sin-gle dependency). The algorithm assumes that the interaction consistsopposing dependencies, such as birth vs. death and immigration vs.emigration.

6 Results

The tree and shade model[3] is successfully modelled by the algo-rithm. It returns two models, representing both possible directions ofcausality between Size and Shade. The initial magnitude assignmentcorrectly finds the conditionless value assignment on Growth rate.The models’s simulation results are equivalent to the original model.

The dependencies of thecommunicating vessels modelare cor-rectly found. The algorithm returns 6 models; one for each possi-ble causal ordering of amount, height and pressure. The algorithmalso correctly identifies the ESM-based actuations of the clusters, byproperly finding the min operator. Furthermore, all necessary causaldependencies and correspondences are identified. Model fragmentsthat allow the assumption of initial values are missing (due to thefact that the algorithm generates a single model fragments). Addingan inequality between the pressures of the containers in the scenarioallows the model to simulate without problems.

Thedeforestation model(containing entities ’Woodcutters’, ’Veg-etation’, ’Water’, ’Land’ and ’Humans’) is successfully modelled,including setting initial magnitudes using conditions. The simula-tion is equivalent to that of the original model. The causal orderingdoes differ, as it does not capture the branching of the causal paths inthe original model. The resulting model however, is not consideredwrong by experts, and is arguably better than the original. Over 2000

models are returned when generating all possible results, due to themany possible causal orderings.

Thepopulation dynamics model[3] generates the correct modelsfor the open and closed population scenarios. However, the initialvalues are not set.

The algorithm does not yet give correct results for theheat-ing/boiling, R-Star[6] andAnts’ Garden[8] models. For the heatingmodel this is due to inequalities that hold under specific conditions,which are not taken care of in the algorithm. The R-Star and Ants’Garden are large models that resulted from specific research projects.As such, these models are an order of magnitude more complex thanthe other models. It is therefore not surprising that the algorithm inits current form cannot cope with them.

7 Conclusions & Future Work

This paper presents preliminary work towards an algorithm that auto-matically determines a Garp3 qualitative model, using an enumera-tion of all possible system behaviour as input. The algorithm usesconsistency rules to determine the causal dependencies that holdwithin the system. Using the concept of clusters the search spaceis significantly reduced. Accurate results are generated for a set ofwell-established models. The results seem to suggests that it is pos-sible to derive causal explanations from the behaviour of a system,and that model building support through an automatic model buildingalgorithm is viable.

There are several algorithm improvements planned. The first im-provement is to have a generalised representation for the ambiguitywithin and between clusters. That is, have a single representation forthe complete model space. For simulation purposes an arbitrary in-stantiation can be chosen, as each one has an equivalent result. Sec-ondly, the algorithm has to be improved to be able to create multiplemodel fragments in order to deal with conditional model ingredients.Thirdly, means have to be developed to be able to compare generatedstate graphs with the desired state graph.

ACKNOWLEDGEMENTS

We would like to thank the referees for their insightful comments.

REFERENCES[1] Ivan Bratko and DorianSuc, ‘Learning qualitative models’,AI Mag.,

24(4), 107–119, (2004).[2] B. Bredeweg, A. Bouwer, J. Jellema, D. Bertels, F. Linnebank, and

J. Liem, ‘Garp3 - a new workbench for qualitative reasoning and mod-elling’, in 20th International Workshop on Qualitative Reasoning (QR-06), eds., C. Bailey-Kellogg and B. Kuipers, pp. 21–28, Hanover, NewHampshire, USA, (July 2006).

[3] Bert Bredeweg and Paulo Salles,Handbook of Ecological Modelling andInformatics, chapter Mediating conceptual knowledge using qualitativereasoning, WIT Press, 2008. (in press).

[4] W. Bridewell, P. Langley, L. Todorovski, and S. Dzeroski, ‘Inductive pro-cess modeling’,Machine Learning, 71, 132, (2008).

[5] J. de Kleer and J. S. Brown, ‘A qualitative physics based on confluences’,Artificial Intelligence, 24(1-3), 7–83, (December 1984).

[6] T. Nuttle, B. Bredeweg, and P. Salles, ‘R-star - a qualitative model ofplant growth based on exploitation of resources’, in19th InternationalWorkshop on Qualitative Reasoning (QR’05), eds., M. Hofbaur, B. Rin-ner, and F. Wotawa, pp. 47–53, Graz, Austria, (May 2005).

[7] Wei Pang and George M. and Coghill, ‘Advanced experiments for learn-ing qualitative compartment models’, in21th International Workshop onQualitative Reasoning (QR-07), ed., C. Price, (2007).

[8] P. Salles, B. Bredeweg, and N. Bensusan, ‘The ants garden: Qualitativemodels of complex interactions between populations.’,Ecological Mod-elling, 194(1-3), 90–101, (2006).

17

Automated Learning of Communication Modelsfor Robot Control Software

Alexander Kleiner 1and Gerald Steinbauer 2

and Franz Wotawa 2

Abstract.Control software of autonomous mobile robots comprises a

number of software modules which show very rich behaviorsand interact in a very complex manner. These facts amongothers have a strong influence on the robustness of robot con-trol software in the field. In this paper we present an approachwhich is able to automatically derive a model of the structureand the behavior of the communication within a component-orientated control software. Such a model can be used foron-line model-based diagnosis in order to increase the robust-ness of the software by allowing the robot to autonomouslycope with faults occurred during runtime. Due to the factthat the model is learned form recorded data and the use ofthe popular publisher-subscriber paradigm the approach canbe applied to a wide range of complex and even partially un-known systems.

1 Introduction

Control software of autonomous mobile robots comprises anumber of software modules which show very rich behaviorsand interact in a very complex manner. Because of this com-plexity and other reasons like bad design and implementationthere is always the possibility that a fault occurs at runtimein the field. Such faults can have different characteristics likecrashes of modules, deadlocks or wrong data leading to a haz-ardous decision of the robot. This situation can occur even ifthe software is carefully designed, implemented and tested. Inorder to have truly autonomous robots operating for a longtime without or with limited possibility for human interven-tion, e.g., planetary rovers exploring Mars, such robots haveto have the capability to detect, localize and to cope withsuch faults.

In [8, 7] the authors presented a model-based diagnosisframework for control software for autonomous mobile robots.The control software is based on the robot control frameworkMiro [10, 9] and has a client-server architecture where the soft-ware modules communicate by exchanging events. The ideais to use the different communication behaviors between themodules of the control software in order to monitor the statusof the system and to detect and localize faults. The modelcomprises a graph specifying which modules communicatewith each other. Moreover, the model has information about

1 Institut fur Informatik, Albert-Ludwigs-UniversitatFreiburg, Georges-Kohler-Allee, D-79110 Freiburg, Germany,[email protected]

2 Institute for Software Technology, Graz Universityof Technology, Inffeldgasse 16b/II, A-8010, Austria,{steinbauer,wotawa}@ist.tugraz.at

the type of a particular communication path, e.g, whether thecommunication occurs on a regular basis or sporadically. Fi-nally, the model includes information about which inputs andoutputs of the software modules have a functional relation,e.g, which output is triggered by which input. The model isspecified by a set of logic clauses and uses a component-basedmodeling schema [1]. Please refer to [8, 7] for more details.

The diagnosis process itself uses the well knownconsistency-based diagnosis techniques of Reiter [5]. The mod-els of the control software and the communication were cre-ated by hand by analyzing the structure of the software andits communication behavior during runtime. Because of thecomplexity of such control software or the possible lack of in-formation about the system it is not feasible to do this byhand for large or partially unknown systems.

Therefore, it is desirable that such models can be createdautomatically either from a formal specification of the systemor from observation of the system. In this paper we present anapproach which allows the automatic extraction of all neces-sary information from the recorded communication betweenthe software modules. The algorithm provides all informationneeded for model-based diagnosis. It provides a communica-tion graph showing which modules communicate, the desiredbehavior of the particular communication paths and the rela-tion between the inputs and outputs of the software modules.

These model learning approach was originally developed forand tested with the control software of the Lurker robots [2]used in the RoboCup rescue league. This control software usesthe IPC communication framework [6], which is a very pop-ular event-based communication library used by a number ofrobotic research labs worldwide. However, the algorithm sim-ply can be adapt to other event-based communication frame-works, such as for instance Miro. The next section describesin more detail how the model is extracted from the observedcommunication.

2 Model Learning

Control systems based on IPC use an event-based communi-cation paradigm. Software modules which wants to providedata are publishing an event containing the data. Other soft-ware modules which like to use this data, subscribe for theappropriate event and get automatically informed when suchan event is available. A central software module of IPC is incharge for all aspects of this communication. Moreover, thissoftware module is able to record all the communication de-tails. It is able to record the type of the event, the time theevent was published or consumed, the content of the event,and the names of the publishing and the receiving module.

19

The collected data is the basis for our model learning algo-rithm. Figure 1 depicts such collected data for a small exam-ple control software comprising only 5 modules with a simplecommunication structure. This example will be used in thefollowing description of the model learning algorithm. Thecontrol software comprises two data path. One is the pathfor the self-localization of the robot. The two modules in thepath Odometry and SelfLoc provide data on a regular basis.The other is the path for object tracking. The module Visionprovides new data on a regular basis. The module Trackerprovides data only if new data is available from the moduleVision. The figure shows when the different events were pub-lished. Based on this recorded communication we extract thecommunication model step by step.

2.1 The communication graph

At a first step the algorithm extract a communication graphfrom the data. The nodes of the graph are the different soft-ware modules. The edges represent the different events whichare exchanged between the modules. Each event is representedby at least one edge. If the same event is received by multiplemodules, there is an edge to every receiving module originat-ing from the publishing module. Figure 2 depicts the commu-nication graph for the above example. This graph shows thecommunication structure of the control software. Moreover,it shows the relation of inputs and outputs of the differentsoftware modules because each node knows its connections.Such a communication graph is not only useful for diagnosispurposes, but it is also able to expressively visualize the re-lation of modules from a larger or partially unknown controlsoftware.

Formally the communication graph can be defined as fol-lowing:

Definition 1 (CG) A communication graph (CG) is a di-rected graph with the set of nodes M and the set of labelededges C where:

• M is a set of software modules sending or receiving at leastone event.

• C is a set of connections between modules, the direction ofthe edge points from the sending to the receiving module,the edge is labeled with the name of the related event.

Please note that the communication graph may contain cy-cles. Usually such cycles emerge from acknowledgement mech-anisms between two modules.

The algorithm for the creation of the communication graphis straightforward. The algorithm starts with an empty setof nodes M and edges C. The algorithm iterates trough allrecorded communication events. If either the sender or thereceiver are not in the set of the nodes the sender or thereceiver is added to the set. If there is no edge pointing fromthe sending to the receiving node with the proper label, anew edge with the appropriate label is added between thetwo modules.

Moreover, we define the two functions in : CO 7→ 2C

which returns the edges pointing to a node and the functionout : CO 7→ 2C which returns the edges pointing from anode.

2.2 The communication behavior

In a next step the behavior or type of each event connection isdetermined. For this determination we use the information ofthe node the event connection comes from, and the recordedinformation of the event related to the connection, and allevents related to the sending node.

We can distinguish the following types: triggeredevent connection (1), periodic event connection (2),bursted event connection (3) and random event connec-tion (4). In order to describe the behavior of a con-nection formally we define a set of connection typesCT = {periodic, triggered, bursted, random} and a functionctype : C 7→ CT which returns the type of a particular con-nection c ∈ C.

The type of a event connection is determined by tests likemeasurements of the mean and the standard deviation of thetime between the occurrence of the events on the connection,and comparison or correlation of the occurrence of two events.The criteria used to assign an event connection to one of thefour categories are summarized below:

triggered A triggered event only occurs if its publishingmodule recently received a trigger event. In oder to deter-mine if an event connection is a triggered event connection,the events on connection c ∈ out(m) are correlated to theevents on the set of input connection to the software moduleI = in(m). If the number of events on connection c, which arecorrelated with an event on a particular connection t ∈ in(m),exceed a certain threshold, connection t is named as trigger ofconnection c. The correlation test looks for the occurrence ofthe trigger event prior the observed event. Note each triggerevent can only trigger one event. If connection c is correlatedwith at least one connection t ∈ in(m) connection c is cate-gorized as a triggered connection. Usually, such connectionsare found in modules doing calculations only if new data areavailable.

periodic On a periodic event connection the same eventregularly occurs with a fixed frequency. We calculate from thetime stamps of the occurrence of all events a discrete distri-bution of the time difference between two successive events. Ifthere is a high evidence in the distribution for one particulartime difference, the connection is periodic with a periodic timeof the estimated time difference. For a pure periodic eventconnection one gets a distribution close to a Dirac impulse.Usually, such connections are found with modules providingdata at a fixed frame rate, such as a module sending datafrom a video camera.

bursted A bursted event is similar to the periodic eventbut its regularly occurrence can be switched on and off fora period of time. A event connection is classified as burstedif there exist time periods where the criteria of the periodicevent connection hold. Usually, such connections are foundwith modules which do specific measurements only if the cen-tral controller explicitly enable them, e.g., a complete 3d laserscan.

random For random event connections none of the abovecategories match and therefore no useful information aboutthe behavior of that connection can be derived. Usually, such

20

0

50

100

150

200

250

300

350

0 1 2 3 4 5 6 7 8

Time [s]

msg-objects msg-odometry msg-velocities msg-pose

Figure 1. Recorded communication of the example robot control software. The peaksindicate the occurrence of the particular event.

Vision

T r a c k e r

msg-objects @ 2 Hz

U s e r

msg-velocities @ 2 Hz

O d o m e t r y

Selfloc

msg-odometry @ 12 Hz

msg-pose @ 6 Hz

Figure 2. Communication graph learnedfrom the recorded data of the example control

software.

connections are found in modules which provide data only ifsome specific circumstance occur in the system or its environ-ment.

In the case of the above example, the algorithm correctlyclassified the event connections odometry, objects and poseas periodic and the connection velocity as triggered with thetrigger objects.

2.3 The observers

In order to be able to monitor the actual behavior of the con-trol software, the algorithm instantiates an observer for eachevent connection. The type of the observer is determined bythe type of the connection and its parameters, estimated bythe methods described before. An observer rises an alarmif there is a significant discrepancy between the currentlyobserved behavior of an event connection and the behaviorlearned beforehand during normal operation. The observerprovides as an observation O the atom ok(l) if the behavioris within the tolerance and the atom ¬ok(l) otherwise. Wherel is the label of the corresponding edge in the communica-tion graph. The observations of the complete control OBSsoftware is the union of all individual observations

OBS =

n[

i=1

Oi

where n is the number of observers.

The following observers are used:

triggered This observer raises an alarm if within a certaintimeout after the occurrence of a trigger event no correspond-ing event occurs or if the trigger event is missing prior theoccurrence of the corresponding event. In order to be robustagainst noise, the observer uses a majority vote for a numberof succeeding events, e.g, 3 votes.

periodic This observer raises an alarm if there is a signif-icant change in the frequency of the events on the observedconnection. The observer checks if the frequency of succes-sive events does vary too much from the specified frequency.For this purpose, the observer estimates the frequency of theevents within a sliding time window.

bursted This observer is similar to the observer above. Itdiffers in the fact that this observer starts the frequency checkonly if events occur and does not raise an alarm if no eventsoccur.

random This is a dummy observer which alway providesthe observation ok(l). This observer is implemented for com-pleteness.

2.4 The system description

The communication graph together with the type of the con-nections is a sufficient specification of the communication be-havior of the robot control software. This specification can beused in order to derive a system description for the diagnosisprocess. It is a description of the desired or nominal behaviorof the system. In order to be able to be used in the diagno-sis process, the system description is automatically writtendown as a set of logical clauses. This set can easily be de-rived from the communication graph and the behavior of theconnections.

The algorithm to derive the system description starts withan empty set SD. For every event connection in two steps,clauses are added to the system description. In the first step,a clause for forward reasoning is added. The clause specifiesif a module works correct and all related inputs and outputsbehave as expected. Depending on the type of the connection,we add the following clause to the SD. If connection c is

21

triggered, we add the clause

¬AB(m)^

t∈trigger(c)∧t∈in(m)

ok(t) → ok(c)

and the clause¬AB(m) → ok(c)

otherwise. ¬AB(m) means that the module m is not abnormaland the module works as expected. The atom ok(c) specifiesthat the connection c behaves as expected.

In a second step, a clause for backward reasoning is added.The clause specifies if all output connections c′ of module mbehave as expected the module itself has to behave as ex-pected. We add the clause

^

c′∈out(m)

ok(c′) → ¬AB(m)

Figure 3 depicts the system description obtained for theabove example control software.

3 Model-based diagnosis

For the detection and localization of faults we use theconsistency-based diagnosis technique of [5]. A fault de-tectable by the derived model causes a change in the behaviorof the system. If such an inconsistency between the modeledand observed behavior emerges, a failure has been detected.Formally, we define this by:

SD ∪ OBS ∪ {¬AB(m)|m ∈ M} |=⊥

where the latter set says that we assume that all moduleswork as expected.

In order to localize the module responsible for the detectedfault, we have to calculate a diagnosis ∆. Where ∆ is a set ofmodules m ∈ M we have to declare as faulty (change ¬AB(m)to AB(m)) in order to resolve the above contradiction. Weuse our implementation 3 of this diagnosis process for theexperimental evaluation of the models. Please refer to [8, 7]for the detail of the diagnosis process.

4 Experimental Results

In order to show the potential of our model learning approach,the approach has been tested on three different types of robotcontrol software. We evaluated whether the approach is ableto derive an appropriate model reflecting all aspects of thebehavior of the system. The derived model was evaluated bythe system engineer who has developed the system. Moreover,we injected artificial faults like module crashes in the system,and evaluated if the fault can be detected and localized bythe derived model.

4.1 A small example control software

The example software from the introduction comprises fivemodules. The module Odometry provides odometry data ata regular basis. This data is consumed by the module Self-Loc, which does pose tracking by integrating odometry data,and providing continuously a pose estimate to a visualization

3 The implementation can freely be downloaded athttp://www.ist.tugraz.at/mordams/.

module User. The module Vision provides position measure-ments of objects. The module Tracker uses this measurementsto estimate the velocity of the objects. New velocity estima-tions are only generated if new data is available. The veloc-ity estimates are also visualized by the GUI. Figure 1 showsthe recorded communication of this example. Figure 2 depictsthe communication graph extracted from the recorded data.It correctly represents the actual communication structure ofthe example, and shows the correct relation of event producersand event consumers.

Moreover, the algorithm correctly identified the type of theevent connections. This can be seen by the system descrip-tion the algorithm has derived which is depicted in Figure 3.It also instantiated the correct observer for the four eventconnections. A periodic event observer was instantiated forodometry, objects and pose, and a triggered event observerwas instantiated for velocities.

1. ¬AB(Vision) → ok(objects)2. ¬AB(Odometry) → ok(odometry)3. ¬AB(Tracker) ∧ ok(objects) → ok(velocities)4. ¬AB(Selfloc) → ok(pose)5. ok(objects) → ¬AB(Vision)6. ok(odometry) → ¬AB(Odometry)7. ok(velocities) → ¬AB(Tracker)8. ok(pose) → ¬AB(Selfloc)

Figure 3. The system description automatically derived for theexample control software.

Figure 3 depicts the extracted system description. Clauses1 to 4 describe the forward reasoning. Clauses 5 to 8 de-scribe the backward reasoning. Clause 3 states that the mod-ule Tracker works correctly only if a velocity event occursonly after trigger event. For instance, Clause 6 states that ifall output connections of module Odometry work as expected,consequently the module itself works correct. This automat-ically generated system description was used in some diag-nosis tests. We randomly shutdown modules and evaluate ifthe fault was correctly detected and localized. For this simpleexample the faults were always proper identified.

4.2 Autonomous exploration robot Lurker

In a second experiment we recorded the communication of thecontrol software of the rescue robot Lurker [2] while the robotwas autonomously exploring an unknown area. The robot isshown in Figure 4.

The control software of this robot is far more complex asin the example above, since it comprises all software modulesenabling a rescue robot to autonomously explore an area aftera disaster. Figure 5 shows the communication graph derivedfrom the recorded data, clearly showing the complex structureof the control software.

From the communication graph and the categorized eventconnections a system description with 70 clauses with 51atoms and 35 observers was derived. After a double check withthe system engineer of the control software it was confirmedthat the automatically derived model maps the behavior ofthe system.

22

X s e n s e

mcCl i en t

i n e r t i a@ 58Hz

Lurke rCon t ro l l e r

i n e r t i a@ 25 Hz

local izat ion

i n e r t i a@ 38Hz

e leva t ion

i n e r t i a@ 20 Hz

l u r k e r _ a r m _ p o s@ 11 Hz

l u r k e r _ t o u c h _ p o s@ 11 Hz

b u m p e r@ 11 Hz

t i l t_ack@ 3 Hz

u r g L M S

p o s i t i o n e r _ a c t u a t o r@ 3Hz

m o t o r@ 0 Hz

p o s i t i o n e r _ a c t u a t o r@ 3Hz

r a n g e s c a n _ r a n g e s@ 7 Hz

r a n g e s c a n _ r a n g e s@ 9 Hz

r a n g e s c a n@ 0 Hz

t i l t e d _ r a n g e s@ 5 Hz

Hie ra rchyCon t ro l l e r

3 d s c a n _ r e c e i v e d@ 0 Hz

r e d o n e@ 5 Hz

r o b o t _ c o n t e x t@ 0 Hz

task_f in i sh@ 0 Hz

k a l m a n _ p o s e@ 6 Hz

k a l m a n _ p o s e@ 6 Hz

p o s i t i o n e r _ a c t u a t o r@ 0 Hz

p o s i t i o n e r _ a c t u a t o r@ 0 Hz

p o s e 3 d@ 13 Hz

h e i g h t m a p@ 2 Hz

3 d s c a n _ r e c e i v e d@ 0 Hz

mrfHe igh tmapClas s i f i e r

p a r t i a l _ h e i g h t m a p@ 0 Hz

r e d o n e@ 3 Hz

3 d s c a n _ t r i g g e r@ 0 Hz

t a s k _ a s s i g n _ c l i m b i n g@ 0 Hz

r o b o t _ c o n t e x t@ 3 Hz

m r f _ a r e a _ r e q u e s t@ 1 Hz

RemoteAu tonomy

h i e r a r c h y _ d e b u g@ 3 Hz

a c t i o n _ e x e c u t i o n _ d e b u g@ 18 Hz

a u t o n o m y _ c o n t r o l@ 0 Hz

p a r t i a l _ h e i g h t m a p@ 0 Hz

Figure 5. Communication graph Lurker robot.

Figure 4. The autonomous rescue robot Lurker of theUniversity of Freiburg.

4.3 Teleoperation Telemax robot.

In a final experiment we record data during a teleoperated runwith the bomb-disposal robot Telemax. The robot Telemax isshown in Figurer 6.

Figure 6. The teleoperated robot Telemax.

Figure 7 depicts the communication graph derived from therecorded data. It clearly shows that the control software forteleoperation shows a far less complex communication struc-ture than in the autonomous service. From the communica-tion graph and the categorized event connections a systemdescription with 44 clauses with 31 atoms and 22 observerwas derived.

5 Related Research

There are many proposed and implemented systems for faultdetection and repair in autonomous systems. Due to lack ofspace we refer only a few. The Livingstone architecture byWilliams and colleagues [4] was used on the space probe DeepSpace One to detect failures in the probe’s hardware and torecover from them. Model-based diagnosis also has been suc-cessfully applied for fault detection and localization in dig-ital circuits and car electronics and for software debuggingof VHDL programs [1]. In [3] the authors show how model-based reasoning can be used for diagnosis for a group of robotsin the health care domain. The system model comprises in-terconnected finite state automata. All these methods havein common that the used models of the system behavior aregenerated by hand.

6 Conclusion and Future Work

In this paper we presented an approach which allows theautomated learning of communication models for robot con-trol software. The approach uses recorded event communica-tion. The approach is able to automatically extract a modelof the behavior of the communication within a component-orientated control software. Moreover, the approach is able toderive a system description which can be used for model-baseddiagnosis. The approach was successfully tested on IPC-based

23

X s e n s e

RoboGUI_RobotConnect

i n e r t i a @ 28 Hz

SensorVisual iza t ion


gpsF i l t e r


Telemax

f l ipper_axes @ 9 Hz

b a t t e r y _ s t a t u s @ 1 Hz

o d o m e t r y @ 1 Hz

f l ipper_axes @ 10 Hz

f l i p p e r _ c u r r e n t @ 10 Hz

r o b o t _ c o n f i g u r a t i o n @ 10 Hz

sa fe ty @ 1 Hz

a t t i t u d e @ 2 Hz

dr ive_axes @ 10 Hz

o d o m e t r y @ 10 Hz

usb4VideoSende r

i m a g e _ m e s s a g e @ 3 Hz

Joyst ick

m a n i p u l a t o r _ m o v e m e n t s @ 8 Hz

velocity @ 7 Hz

f l i p p e r _ m o v e m e n t s @ 7 Hz

f l i p p e r _ m o v e m e n t s @ 7 Hz

velocity @ 7 Hz

GPS_module

g p s _ s t a t u s @ 0 Hz

g p s _ s a t e l l i t e s @ 1 Hz

Figure 7. Communication graph Telemax robot.

robot control software like the rescue robot Lurker. IPC is awidely used basis for robot control software. Therefore, ourapproach is instantly usable on many different robot systems.

Currently, we are working on a port for Miro-based systems.This even will increase the number of potential target systemsof our approach. Moreover, we work on the recognition ofadditional event types in order to enrich the generated models.

We believe that the consideration of the content of theevents will lead to significantly better models and diagno-sis. For the modeling the techniques of Qualitative Reason-ing seem to be promising. But it is an open question howsuch qualitative models can be automatically learned fromrecorded data.

REFERENCES

[1] Gerhard Friedrich, Markus Stumptner, and Franz Wotawa,‘Model-based diagnosis of hardware designs’, Artificial Intel-ligence, 111(2), 3–39, (1999).

[2] Alexander Kleiner and Christian Dornhege, ‘Real-time Lo-calization and Elevation Mapping within Urban Search andRescue Scenarios’, Journal of Field Robotics, (2007).

[3] Roberto Micalizio, Pietro Torasso, and Gianluca Torta, ‘On-line monitoring and diagnosis of a team of service robots:A model-based approach’, AI Communications, 19(4), 313 –340, (2006).

[4] Nicola Muscettola, P. Pandurang Nayak, Barney Pell, andBrian C. Williams, ‘Remote agent: To boldly go where noAI system has gone before’, Artificial Intelligence, 103(1-2),5–48, (August 1998).

[5] Raymond Reiter, ‘A theory of diagnosis from first principles’,Artificial Intelligence, 32(1), 57–95, (1987).

[6] Reid Simmons, ‘Structured Control for Autonomous Robots’,IEEE Transactions on Robotics and Automation, 10(1),(1994).

[7] Gerald Steinbauer, Martin Morth, and Franz Wotawa, ‘Real-Time Diagnosis and Repair of Faults of Robot Control Soft-ware.’, in RoboCup 2005: Robot Soccer World Cup IX, vol-ume 4020 of Lecture Notes in Computer Science, pp. 13–23.Springer, (2006).

[8] Gerald Steinbauer and Franz Wotawa, ‘Detecting and lo-cating faults in the control software of autonomous mobilerobots.’, in 16th International Workshop on Principles of Di-agnosis (DX-05), pp. 13–18, Monetrey, USA, (2005).

[9] Hans Utz, Advanced Software Concepts and Technologies forAutonomous Mobile Robotics, Ph.D. dissertation, Universityof Ulm, Neuroinformatics, 2005.

[10] Hans Utz, Stefan Sablatng, Stefan Enderle, and Gerhard K.Kraetzschmar, ‘Miro – middleware for mobile robot appli-cations’, IEEE Transactions on Robotics and Automation,Special Issue on Object-Oriented Distributed Control Archi-tectures, 18(4), 493–497, (August 2002).

24

Relaxation of Temporal Observations inModel-Based Diagnosis of Discrete-Event Systems

Gianfranco Lamperti and Federica Vivenzi and Marina Zanella1

Abstract. Temporal observations play a major role in model-based

diagnosis of discrete-event systems. Although the reaction of a sys-

tem generates a sequence of visible labels, in real contexts, where

the system is large and distributed, what is perceived by the observer

is uncertain in nature, namely an uncertain temporal observation. On

the one hand, the degradation of the sequence of labels to the tem-

poral observation has never been subject of formal investigation. On

the other, considerable effort has been spent on similarity-based di-

agnosis, where temporal observations are checkedfor subsumption, a

property that enables reuse of model-based reasoning. The notion of

coverage between temporal observations was proposed to check sub-

sumption efficiently. This paper unifies the concepts of degradation

and coverage by means of the new notion of observation relaxation.

This consists of three algebraic operators applied to the domain of

temporal observations, namely � (logical relaxation), � (temporal re-

laxation), and ˛ (augmentation). A formal result is that any temporal

observation relevant to the reaction of a system can always be repre-

sented by a relaxation expression.

1 INTRODUCTION

Model-based diagnosis of discrete-event systems (DESs) [2] has

been an active research area in this first decade of the 2000s

[4, 13, 3, 14, 6, 15]. A diagnosis task takes as input an observation

of the system to be diagnosed. In case such a system is discrete, its

observable events range over a finite domain of discrete values. An

observation is temporally uncertain if the generation order of ob-

served events is not precisely known, what is known is instead a par-

tial order that conforms to the actual generation order: an event can

be observed before another that was generated by the DES before it,

and, given the reception order of events, it is impossible to devise the

relative emission order of all the pairs of events belonging to the ob-

servation. Therefore, several sequences of observable events comply

with a temporally uncertain observation.

Features and models of DES observations have been investigated

for diagnosis purposes in several directions:

� Defining the different kinds of uncertainty affecting a given obser-

vation [9];

� Splitting an uncertain observation into sub-observations to be in-

crementally considered by diagnostic tasks [10, 14];

� Studying the effects of some properties (such as correct slicing [7]

or stratification [12]) of a fragmented uncertain observation on the

diagnostic results, and recognizing whether a given fragmented

uncertain observation exhibits such properties;

1 Universita di Brescia, Italy, e-mail: [email protected], feder-ica.vivenzi.gmail.com, [email protected]

� Proposing algorithms for comparing uncertain observations in or-

der to reuse model-based reasoning [11, 12].

All these research lines assume an existing uncertain observation.

This paper instead assumes an existing completely certain sequence

of observable events (the sequence of events generated by the consid-

ered DES) and addresses how such a sequence is transformed into an

uncertain observation, by introducing the notion of relaxation. The

aim of the paper is providing a formal framework for mimicking what

happens in the real world, so as to endow the notion of an uncertain

observation with a physical motivation. However, after relaxation had

been defined, the authors found out that this notion was strictly re-

lated to the notions of subsumption and coverage already introduced

for quite different purposes [11, 12].

Owing to space reasons, the proofs of all propositions and theo-

rems in the paper are omitted.

2 TEMPORAL OBSERVATION

Discrete-event systems are dynamic systems, typically modeled as

networks of components. Each component is a communicating au-

tomaton [1] that reacts to input events by state-transitions which pos-

sibly generate new events towards other components. When a DES

reacts, it performs a sequence of transitions, some of which are visi-

ble. For each visible transition, an observable label is generated. The

whole sequence of these observable labels (ordered according to the

generation order) is the signature of the reaction. However, what is

actually perceived by the external observer about the reaction is a

degradation of the signature, namely the temporal observation.

Formally, let L be a finite domain of labels, possibly including the

null label �. A temporal observation is a DAG

O D .N ; L; A/ (1)

where N is the set of nodes, with each N 2 N being marked with

a non-empty subset of L, and A W N 7! 2N is the set of arcs. A

‘�’ temporal precedence relationship among nodes of the graph is

defined as follows:

� If N 7! N 0 2 A then N � N 0;

� If N � N 0 and N 0 � N 00 then N � N 00;

� If N 7! N 0 2 A then ÀN 00 2 N .N � N 00 � N 0/.

Based on the last property, we say that O is in canonical form (that is,

without any redundant temporal precedence). When no precedence

relationship is defined between N and N 0, such nodes are temporally

unrelated, written N ªN 0.

The set of candidate labels marking a node N is the logical con-

tent of the node, written kN k. We assume kN k ¤ f�g.

25

Figure 1. Certain observation O` (left) and uncertain observations O2

(center) and O1 (right).

An observation which includes a node whose temporal content

is not a singleton is affected by logical uncertainty. An observa-

tion which includes a pair of temporally unrelated nodes is affected

by temporal uncertainty A temporal observation where none of the

above uncertainties holds is a linear observation. The signature of

the reaction is in fact a linear observation.

An uncertain observation O implicitly incorporates several candi-

date signatures, where each candidate is determined by selecting one

label from each node in N without violating the temporal constraints

imposed by the precedence relationships.

Assumption 1. Let O be the temporal observation relevant to a sig-

nature S . Among the candidate signatures of O is S .

Based on Assumption 1, all candidate signatures but one are spu-

rious. However, the mode in which the signature S degrades to an

observation O is, generally speaking, nondeterministic and, there-

fore, unpredictable, thereby making it impossible to ascribe O to S .

The signature of a reaction may incidentally have not degraded. In

such a case, the temporal observation is completely certain (linear

observation).

Example 1. Assume that a DES reaction generates the signature

abc. Shown on the left in Fig. 1 is the corresponding completely cer-

tain (and therefore necessarily linear) observation whose only can-

didate signature is abc. This is the observation considered by the

diagnosis process in case no degradation has occurred. If, instead,

a degradation has occurred, the resulting temporal uncertain obser-

vation may be, for instance, the one displayed in the center of the

same figure, O2 D .N2; L2; A2/, where N2 = fN 01; : : : ; N 0

4g, and

L2 D fa; b, c; d; �g. Node N 01 incorporates the first observable la-

bel, namely a. Then, either N 02 or N 0

3 follows, each of which in-

volves two candidate labels, where � is null. The last generated node

is N 04, with a and � being the final candidate labels. This means that

the observer cannot devise the reciprocal emission order of the ob-

servable events relevant to nodes N 02 or N 0

3 since, for instance, the

difference between their time tags was less than the synchronization

error between the clocks of the two distinct channels transmitting

such events from the system to the observer. Moreover, possibly ow-

ing to the observer’s limited discrimination ability, the observer does

not know whether there is actually an event relevant to N 02 or only

noise. Likewise, the observer cannot discriminate which is the actual

label relevant to N 03. Finally, node N 0

4 is due to noise on the trans-

mission channel (in fact no a label was generated after the first one),

however the observer does not know whether what was received is

pure noise or label a instead. Based on the content of each node and

on the partial temporal relationships among nodes, it is easy to show

that kO2k includes the candidate signatures ac, ad , abc, abd , aca,

ada, acb, adb, abca, abda, acba, adba, each of which is obtained

Figure 2. Isp.O1/ (left) and two-step construction of Isp.O2/ (right).

by selecting one label for each node without violating the temporal

constraints, where the null label � is removed. Notice that this ob-

servation is actually a degradation of the given signature abc since

kO2k includes the signature itself.

3 SUBSUMPTION

In similarity-based diagnosis of DESs [11], it is essential to under-

stand whether the solution of the diagnosis problem }0 at hand can

be supported by the knowledge yielded for solving a previous (differ-

ent) diagnosis problem }, with the latter being stored in a knowledge

base. Among other constraints, reuse of } can be exploited only if

the observations O0 and O relevant to }0 and }, respectively, are

linked by a subsumption relationship,

O c O0 (2)

namely, only if O subsumes O0. O subsumes O

0 if and only if the

set of candidate signatures of O includes all the candidate signatures

of O0. The subsumption relationship is defined in terms of regular-

language containment, relevant to the corresponding index spaces.

The index space of an observation O, namely Isp.O/, is a determin-

istic automaton with the property that its regular language is the set

of candidate signatures of O, namely Lang.Isp.O// D kOk [11].

Therefore, O subsumes O0 if and only if

Lang.Isp.O// � Lang.Isp.O0//: (3)

The reason why observation subsumption supports reuse can be

roughly explained as follows. The solution of } yields an automaton

�, a sort of diagnoser based on O, where each state is marked by a set

of diagnoses and each transition is marked by a label in L � f�g. The

regular language of � is the subset of the signatures relevant to O

that comply with the model of the system, namely, Lang.�/ � kOk.

The same applies to a new problem }0 relevant to O0. However, if

O c O0, that is, kOk � kO

0k, then Lang.�/ � Lang.�0/. In other

words, � contains all the signatures of �0. This allows the diagnosis

engine to reuse � in order to generate �0 based on O0. The advantage

stems from the fact that such an operation is far more efficient than

generating �0 from scratch, which would require heavy, low-level

model-based reasoning.

26

Example 2. Suppose that a diagnosis problem inherent to obser-

vation O2 displayed in the center of Fig. 1 has to be solved. Let

assume that the portion of the knowledge base inherent to the con-

sidered DES does not include an observation equal to it, while it in-

cludes observation O1 D .N1; L1; A1/, displayed on the right of the

same figure, where N1 D fN1; : : : ; N5g, and L1 D fa; b; c; d; f; �g.

Then, a subsumption check has to be performed to ascertain whether

O1 subsumes O2. This could be done by building the deterministic

automaton generating the language of either observation, that is, the

index space. This is a two-step process, as illustrated for observa-

tion O2 on the right of Fig. 2: first a nondeterministic automaton is

drawn from the observation graph, then the equivalent deterministic

automaton is built [8]. The deterministic automaton generating the

language of observation O1 is displayed on the left of Fig. 2. It is

easy to check that Lang.Isp.O1// � Lang.Isp.O2//, therefore O1

subsumes O2.

4 COVERAGE

Checking observation subsumption by regular-language containment

may be prohibitive in real applications. In order to cope with this

complexity, an alternative checking-technique, based on the notion

of coverage was proposed in [12], where it was proven that coverage

entails subsumption, that is, it is a sufficient condition for subsump-

tion.

Definition 1. (COVERAGE) Let O D .N ; L; A/ and O0 D

.N 0; L0; A

0/ be two temporal observations, where N D

fN1; : : : ; Nng and N 0 D fN 01; : : : ; N 0

n0g. We say that O covers O0,

written

O D O0 (4)

if and only if there exists a subset NN of N , with NN D f NN1; : : : ; NNn0 g

having the same cardinality as N 0, such that, denoting N � D�

N � NN�

, we have:

� (�-coverage): 8N 2 N� .� 2 kN k/;

� (Logical coverage): 8i 2 Œ1 :: n0� .k NNik � kN 0ik/;

� (Temporal coverage): For each path NNi Ý NNj in O such that bothNNi and NNj are in NN , and all (if any) intermediate nodes of the

path are in N� , we have N 0

i� N 0

jin O

0.

Example 3. With reference to the observations in Fig. 1, it is

easy to show that O1 D O2. Assume the subset of N1 beingNN1 D fN2; N1; N4; N5g. Hence, N

�1 D fN3g. Clearly, �-coverage

holds, as � 2 kN3k. Logical coverage holds too, as kN2k � kN 01k,

kN1k � kN 02k, kN4k � kN 0

3k, and kN5k � kN 04k. It is easy to

check that temporal coverage occurs. For instance, for hN1; N3; N5i,

where N3 2 N�1 , we have N 0

2 � N 04 in O2.

5 RELAXATION

Relaxation transforms an observation by relaxing its logical and tem-

poral constraints.

Definition 2. (RELAXATION) Let O D .N ; L; A/ and O0 D

.N 0; L0; A0/ be two temporal observations. We say that O is a re-

laxation of O0, written

O # O0 (5)

iff O can be obtained from O0 by the application of a (possibly empty)

sequence of the relaxation operators �, � , and ˛, defined as follows:

Figure 3. Temporal relaxation.

� Logical relaxation (�): the logical content of a node is extended

with a set of labels.

� Temporal relaxation (� ): a temporal constraint is removed by

the following actions: (1) an arc N 7! N 0 is deleted, (2) for each

parent node Np of N , an arc Np 7! N 0 is inserted, and (3) for

each child node Nc of N 0, an arc N 7! Nc is inserted.

� Augmentation (˛): a new node N is inserted, where kN k © f�g,

and possibly connected with other nodes in such a way that no new

temporal constraint is generated between the previous nodes.2

Operators � and ˛ do not alter the existing temporal constraints

of O0, the former for it affects the logical content only, the latter by

definition. A doubt may arise about operator � : does it change the

existing temporal constraints among the nodes of O0? The answer,

provided by Proposition 1 below, is that it only removes one temporal

constraint between a pair of nodes while leaving all the other ones

unchanged. In this sense, � is the finest-grained temporal relaxation

operator.

Example 4. Shown in Fig. 3 is the effect of the application of the

temporal-relaxation operator � applied to the observation on the left,

where the temporal precedence between N and N 0 is removed. Ac-

cording to the definition, and as outlined on the right of the figure,

the removal of the arc N 7! N 0 is accompanied with the insertion of

four arcs, two arcs from the parents of N to N 0, namely N1 7! N 0

and N2 7! N 0, and two arcs from N to the children of N 0, namely

N 7! N3 and N 7! N4. These allows the relaxed observation to

keep all the (implicit) temporal constraints other than N 7! N 0.

Proposition 1. Let O D �.O0/, where N 7! N 0 is the arc removed

from O0 by � . Then, 8.Ni ; Nj / in O0, .Ni ; Nj / ¤ .N; N 0/, Ni �

Nj in O0, we have Ni � Nj in O.

Example 5. With reference to Fig. 3 and Example 4, it is easy to

check that each temporal constraint in the observation outlined on

the left (other than N 7! N 0) is preserved within the observation

outlined on the right (the result of temporal relaxation), such as, for

instance, N2 � N4, or N1 � N 0, as claimed by Proposition 1. By

contrast, such constraints would have been implicitly removed if, af-

ter the removal of N 7! N 0, we had not inserted the additional arcs

required by � .

Example 6. Consider observations O1 and O2 displayed in Fig. 1.

We show that O1 can be generated by a relaxation expression applied

2 Formally, for each pair of nodes N1 and N2 where N1 ¤ N and N2 ¤ N ,if N1 � N2 in the augmented graph, then N1 � N2 in the original graphtoo.

27

Figure 4. Relaxed observations O� (left), O� (center), and O˛ (right).

to O2 as follows:

O1 D ˛.�.�1.�2.�3.�4.O2////// (6)

where �1 is the logical relaxation by extending kN 01k with fbg, and,

similarly, �2, �3, and �4 extend kN 02k, kN 0

3k, and kN 04k with fag,

fbg, and fbg, respectively. After the application of �1, the intermedi-

ate resulting observation, namely O�, is shown on the left of Fig. 4.

Note how the topology of O2 is preserved, while the logical content

of each node has been extended. Then, � is the temporal relaxation

of O� by removing the temporal constraint between N 01 and N 0

2, re-

sulting in the new observation O� displayed on the center of Fig. 4.

Based on the definition of temporal relaxation, we should insert an

arc from each parent of N 01 to N 0

2, which is not applicable, as N 01

has no parents. Besides, we should also insert an arc from N 01 to N 0

4,

but this would violate the third condition on the temporal precedence

relationship assumed for observations, which preserve the observa-

tion graph from redundant arcs (in fact, N 01 � N 0

4 by means of the

intermediate node N 03). Finally, augmentation ˛ is applied to O� by

inserting a new node N 0˛ between N 0

2 and N 04, where kN 0

˛k D ff; �g.

This yields the precedence relationships N 02 � N˛ � N 0

4, thereby

preserving all the temporal constraints in O� , as required. The result-

ing observation, namely O˛ , is displayed on the right of Fig. 4. Note

how O˛ coincides in fact with O1 (right of Fig. 1). In other words,

O1 is a relaxation of O2.

Theorem 1. Relaxation entails subsumption:

O # O0 H) O c O

0: (7)

Example 7. On the one hand, in Example 6 we have shown that O1

is a relaxation of O2, where O1 and O2 are displayed in Fig. 1. On

the other, in Example 2 we have also shown that O1 subsumes O2.

This is consistent with Theorem 1.

Note 1. Relaxation is stronger than subsumption:

O c O0 6) O # O

0: (8)

Example 8. To be convinced of the claim of Note 1, it suffices to

show an example in which subsumption holds while relaxation does

not. Consider observations O and O0 displayed in Fig. 5. Notice how,

unlike O, O0 does not force any temporal constraint between its two

nodes. Incidentally, both observations involve just one candidate sig-

nature, namely S D aa. Thus, since kOk D kO0k D faag, both

observations subsume each other, in particular O c O0. However, it

is clear that O is not a relaxation of O0.

Theorem 1 and Note 1 offer evidence that relaxation is only a suf-

ficient condition for subsumption, not a necessary one. However,

based on experimental results, if relaxation does not hold, it is un-

likely for subsumption to hold.

Figure 5. Observations O (left) and O0 (right).

Theorem 2. Relaxation is equivalent to coverage:

O # O0 ” O D O

0: (9)

Theorem 2 allows us to test relaxation by coverage. An algorithm

for testing coverage is provided in [5].

Corollary 2.1. Coverage entails subsumption:

O D O0 H) O c O

0: (10)

Note 2. Coverage is stronger than subsumption:

O c O0 6) O D O

0: (11)

Note 2 is a consequence of Theorem 2 and Note 1.

Although, generally speaking, relaxation is not equivalent to sub-

sumption, such an equivalence holds when what is subsumed is a

linear observation.

Proposition 2. Let O0 be a linear observation. Then,

O c O0 H) O # O

0: (12)

With O0 linear, a corollary of Proposition 2 and Theorem 2 is the

equivalence of subsumption, coverage, and relaxation.

Corollary 2.1. Let O0 be a linear observation. Then,

O c O0 ” O # O

0 ” O D O0: (13)

A final corollary concerns the nature of the temporal observation

with respect to the relevant signature.

Corollary 2.2. Let O be the degradation of a signature S , that is, let

O be a temporal observation of S that complies with Assumption 1.

Then, O is a relaxation of S , and, vice versa, a relaxation of S is a

degradation of S .

Example 9. To support the claims of Proposition 2.1 and its two

Corollaries, we will consider observations O` and O2 in Fig. 1. Ex-

ample 1 has shown that kO2k includes the only signature of the linear

observation O`, therefore we can conclude that O2 c O`. Now we

will show that O2 # O`, as O2 can be generated by a relaxation

expression applied to O` as follows:

O2 D �.�c.�b.˛.O`//// (14)

where augmentation ˛ is applied to O` by inserting a new node, say

Nd , whose logical content is fa; �g, as a child of Nc ; �b and �c

extend kNbk and kNck with f�g and fdg, respectively; � removes

the temporal constraint between Nb and Nc , which requires deleting

arc Nb 7! Nc and inserting two new arcs, namely Na 7! Nc and

Nb 7! Nd . The resulting observation equals O2, thus O2 # O`.

28

It can be checked that O2 D O`, where O` D .N`; L`; A`/,

with N` D fNa; Nb; Ncg, and O2 D .N2; L2; A2/. AssumeNN2 D fN 0

1; N 02; N 0

3g and N �2 D fN 0

4g. Clearly, �-coverage holds,

as � 2 kN 04k. Logical coverage holds too, as kN 0

1k � kNak,

kN 02k � kNbk, and kN 0

3k � kNck. Temporal coverage trivially

occurs as no path in O2 has N 04 as an intermediate node. Thus, we

conclude that O2 D O`. All these conclusions are consistent with

Proposition 2 and Corollary 2.1. Given that, as shown in Example 1,

O2 is a degradation of O`, they are also consistent with Corollary 2.2.

6 CONCLUSION

This paper, which is theoretical in nature, deals with the notion of

an uncertain DES observation. If in previous works by the authors,

an uncertain observation was the output of an undefined uncertainty

function [12] as applied to the certain sequence of events generated

by a DES, now this non-deterministic function has been formally

substantiated.

Three relaxation operators have been introduced which are the for-

mal counterpart of the physical effects that makes what is observed

different from what has been generated.

Roughly, the logical operator corresponds to the superposition of

noise to an event transmitted from the DES to the observer, so that

the observer cannot univocally detect which event was actually gen-

erated out of a set of possible events. The same operator corresponds

also the observer inability to exactly recognize a specific event, ow-

ing to limited discrimination capabilities.

The temporal operator can account for several effects, for instance,

it corresponds to synchronization errors among the clocks of distinct

channels that convey events to the observer, so as the observer can-

not uncover the relative emission order of every pair of events; it

corresponds also to the transmission of the events to the observer

by means of distinct channels in a scenario wherein no time-tag is

available, etc.

Finally, the augmentation operator corresponds to pure noise

transmitted on a communication channel from the DES to the ob-

server, where the observer is unable to distinguish whether what it

has received is just noise or an event.

These three operators are the causes of two kinds of uncertainty af-

fecting observations: both the logical and the augmentation operator

cause logical uncertainty, while the temporal operator causes tempo-

ral uncertainty.

An outcome of the paper (Corollary 2.2) is that the application

of such operators to a sequence of certain events generated by a

DES (signature) produces an uncertain observation that fulfills the

assumption always made in previous works by the authors as to the

considered uncertain observation within a diagnostic problem

However, the relaxation operators are not only the generators of

an uncertain observation: they can be applied also to an existing un-

certain observation, thus obtaining a still more uncertain observation.

While delving into this possibility, the authors found out that:

� The relaxation of an uncertain observation subsumes such an ob-

servation;

� Relaxation and coverage are equivalent notions.

The first point provides a rationale for interpreting conclusions drawn

(and proven) by previous research as to similarity-based diagnosis

[11]. Based on the second point, since coverage is a sufficient con-

dition for observation subsumption, so is relaxation. Therefore, in

principle, subsumption checking could be performed as a relaxation

checking.

Checking whether an observation is a relaxation of another one

means checking whether the former can be obtained starting from the

latter by applying the relaxation operators: this is actually a planning

problem. The convenience of such a check from the computational

viewpoint is an interesting topic for future research.

Research is ongoing to state the conditions under which an ob-

servation is subsumed by another that does not cover/relaxes it.

Roughly, this may occur if the subsumed observation includes a set

(at least) of temporally unrelated nodes all having the same logical

content. If a preprocessing is performed that detects and removes

such a temporal uncertainty, a new observation is obtained that has

the same extension as the former one and whose subsuming observa-

tions necessarily cover/relax it. Performing subsumption checking on

this equivalent observation is promising to enhance the effectiveness

of similarity-based diagnosis of DESs.

REFERENCES

[1] D. Brand and P. Zafiropulo, ‘On communicating finite-state machines’,

Journal of ACM, 30(2), 323–342, (1983).

[2] C.G. Cassandras and S. Lafortune, Introduction to Discrete Event Sys-

tems, volume 11 of The Kluwer International Series in Discrete Event

Dynamic Systems, Kluwer Academic Publisher, Boston, MA, 1999.

[3] L. Console, C. Picardi, and M. Ribaudo, ‘Process algebras for systems

diagnosis’, Artificial Intelligence, 142(1), 19–51, (2002).

[4] R. Debouk, S. Lafortune, and D. Teneketzis, ‘Coordinated decentral-

ized protocols for failure diagnosis of discrete-event systems’, Journal

of Discrete Event Dynamic Systems: Theory and Application, 10, 33–

86, (2000).

[5] A. Ducoli, G. Lamperti, E. Piantoni, and M. Zanella, ‘Coverage tech-

niques for checking temporal-observation subsumption’, in Eighteenth

International Workshop on Principles of Diagnosis – DX’07, pp. 59–

66, Nashville, TN, (2007).

[6] E. Fabre, A. Benveniste, S. Haar, and C. Jard, ‘Distributed monitoring

of concurrent and asynchronous systems’, Journal of Discrete Event

Dynamic Systems, 15(1), 33–84, (2005).

[7] A. Grastien, M.O. Cordier, and C. Largouet, ‘Incremental diagnosis of

discrete-event systems’, in Sixteenth International Workshop on Prin-

ciples of Diagnosis – DX’05, pp. 119–124, Monterey, CA, (2005).

[8] J.E. Hopcroft, R. Motwani, and J.D. Ullman, Introduction to Automata

Theory, Languages, and Computation, Addison-Wesley, Reading, MA,

third edn., 2006.

[9] G. Lamperti and M. Zanella, ‘Diagnosis of discrete-event systems from

uncertain temporal observations’, Artificial Intelligence, 137(1–2), 91–

163, (2002).

[10] G. Lamperti and M. Zanella, ‘Dynamic diagnosis of active systems with

fragmented observations’, in Sixth International Conference on Enter-

prise Information Systems – ICEIS’2004, pp. 249–261, Porto, P, (2004).

[11] G. Lamperti and M. Zanella, ‘Flexible diagnosis of discrete-event sys-

tems by similarity-based reasoning techniques’, Artificial Intelligence,

170(3), 232–297, (2006).

[12] G. Lamperti and M. Zanella, ‘On monotonic monitoring of discrete-

event systems’, in Eighteenth International Workshop on Principles of

Diagnosis – DX’07, pp. 130–137, Nashville, TN, (2007).

[13] J. Lunze, ‘Diagnosis of quantized systems based on a timed discrete-

event model’, IEEE Transactions on Systems, Man, and Cybernetics –

Part A: Systems and Humans, 30(3), 322–335, (2000).

[14] Y. Pencole and M.O. Cordier, ‘A formal framework for the decentral-

ized diagnosis of large scale discrete event systems and its application

to telecommunication networks’, Artificial Intelligence, 164, 121–170,

(2005).

[15] R. Su and W.M. Wonham, ‘Global and local consistencies in distributed

fault diagnosis for discrete-event systems’, IEEE Transactions on Au-

tomatic Control, 50(12), 1923–1935, (2005).

29

The Concept of Entropy by means of Generalized Ordersof Magnitude Qualitative Spaces1

Llorenc Rosello and Francesc Prats and Monica Sanchez2 and Nuria Agell 3

Abstract. A new concept of generalized absolute orders of magni-tude qualitative spaces is introduced in this paper. The new structuremakes it possible to define sets of qualitative labels of any cardinal-ity, and is consistent with the classical structure of qualitative spacesof absolute orders of magnitude and with the classical interval alge-bra. In addition, the algebraic structure of these spaces ensures initialconditions for adapting measure theory to a qualitative environment.This theory provides the appropriate framework in which to intro-duce the concept of entropy and, consequently, the opportunity tomeasure the gain or loss of information when working within qual-itative spaces. The results obtained are significant in terms of situ-ations which arise naturally in many real applications when dealingwith different levels of precision.

1 INTRODUCTION

Qualitative Reasoning (QR) is a subarea of Artificial Intelligence thatseeks to understand and explain human beings’ ability for qualitativereasoning [6], [11]. The main objective is to develop systems thatpermit operating in conditions of insufficient numerical data or inthe absence of such data. As indicated in [22], this could be due toboth a lack of information as well as to an information overload.

A main goal of Qualitative Reasoning is to tackle problems in sucha way that the principle of relevance is preserved; that is to say eachvariable has to be valued with the level of precision required [7].It is not unusual for a situation to arise in which it is necessary towork simultaneously with different levels of precision, depending onthe available information, in order to ensure interpretability of theobtained results. To this end, the mathematical structures of Ordersof Magnitude Qualitative Spaces (OM) were introduced.

The wordinformationappears constantly in QR. However, its mean-ing is as yet undefined within a qualitative context. The implicit andexplicit use of the term and concept addresses the need to define and,perhaps paradoxically, to quantify them.

In this work it is presented a way of measuring the amount of infor-mation of a system when using orders of magnitude descriptions to

1 This work has been partly funded by MEC (Spanish Ministry of Educationand Science) AURA project (TIN2005-08873-C02). Authors would like tothank their colleagues of GREC research group of knowledge engineeringfor helpful discussions and suggestions.

2 Polytechnical University of Catalonia, Barcelona, Spain email:[email protected],[email protected],[email protected]

3 Esade, Ramon Llull University, Barcelona, Spain email: [email protected]

represent it. Taking into account that the entropy can be used to mea-sure the information, this work is intended to be a first step towardsthis measure by means of orders of magnitude qualitative spaces.

The concept of entropy has its origins in the nineteenth century, par-ticularly in thermodynamics and statistics. This theory has been de-veloped from two aspects: the macroscopic, as introduced by Carnot,Clausius, Gibbs, Planck and Caratheodory and the microscopic, de-veloped by Maxwell and Boltzmann [15]. The statistical concept ofShannon’s entropy, related to the microscopic aspect, is a measure ofthe amount of information [18],[2].

In order to define the concept of information within the QR frame-work, this paper adapts the basic principles of Measure Theory [8],[5] to give OM a structure in which to define the concept of entropy,and, consequently, the concept of information.

Section 2 defines the concept of generalized absolute orders of mag-nitude qualitative spaces. In Section 3, the algebraic structure of thesespaces is analyzed in order to ensure initial conditions in which toadapt the Measure Theory. A measure and the concept of entropy inthe generalized absolute orders of magnitude spaces are given in sec-tion 4 and 5 respectively. The paper ends with several conclusionsand outlines some proposals for future research.

2 GENERALIZED ABSOLUTE ORDERS OFMAGNITUDE QUALITATIVE SPACES S

∗g

Order of magnitude models are an essential piece among the theoret-ical tools available for qualitative reasoning about physical systems([10], [19]. They aim at capturing order of magnitude commonsense([21]) inferences, such as used in the engineering world. Order ofmagnitude knowledge may be of two types: absolute or relative. Theabsolute order of magnitudes are represented by a partition ofR,each element of the partition standing for a basic qualitative class. Ageneral algebraic structure, called Qualitative Algebra or Q-algebra,was defined on this framework ([9]), providing a mathematical struc-ture which unifies sign algebra and interval algebra through a con-tinuum of qualitative structures built from the rougher to the finestpartition of the real line. The most referenced order of magnitudeQ-algebra partitions the real line into 7 classes, corresponding tothe labels: Negative Large(NL), Negative Medium(NM), NegativeSmall(NS), Zero(0), Positive Small(PS), Positive Medium(PM) andPositive Large(PL). Q-algebras and their algebraic properties havebeen extensively studied ([13], [22])

31

Order of magnitude knowledge may also be of relative type, in thesense that a quantity is qualified with respect to another quantityby means of a set of binary order-of-magnitude relations. The semi-nal relative orders of magnitude model was the formal system FOG([14]), based on three basic relations, used to represent the intuitiveconcepts of ”negligible with respect to” (Ne), ”close to” (Vo) and”comparable to” (Co), and described by 32 intuition-based inferencerules. The relative orders of magnitude models that were proposedlater improved FOG not only in the necessary aspect of a rigorousformalisation, but also permitting the incorporation of quantitativeinformation when available and the control of the inference process,in order to obtain valid results in the real world ([12], [3], [4]).

In ([20], [22]) the conditions under which an absolute orders of mag-nitude and a relative orders of magnitude model are consistent isanalysed and the constraints that consistency implies are determinedand interpreted.

This paper proposes a further step towards the generalization of qual-itative orders of magnitude. This generalization makes it possible todefine orders of magnitude as either a discrete or continuous set oflabels, providing the theoretical basis on which to develop a MeasureTheory in this context.

Definition 1 Let X be a non-empty set,I a subset ofR, andB : I → P(X) an injective function. Then eachB(t) = Bt ⊂ Xis a generalized basic label onX and the setS of generalized basiclabels onX is

S = {Bt | t ∈ I}.

Note that ift 6= t′, thenBt 6= Bt′ .

Definition 2 If i, j ∈ I , with i < j, thegeneralized non-basic label[Bi, Bj) is defined by

[Bi, Bj) = {Bt | t ∈ I, i ≤ t < j}.

In the casei = j ∈ I , the convention[Bi, Bi) = {Bi} will be used.If necessary,[Bi, Bi) = {Bi} can be identified with the basic labelBi.

Definition 3 If i ∈ I , thegeneralized non-basic label[Bi, B∞) isdefined by

[Bi, B∞) = {Bt | t ∈ I, i ≤ t}.

Note thatB∞ is a symbol, not a basic label.

Definition 4 The set ofGeneralized Orders of MagnitudeS∗g is:

S∗g = {∅} ∪ {[Bi, Bj) | i, j ∈ I, i ≤ j} ∪ {[Bi, B∞) | i ∈ I}.

In this definition ofS∗g the basic labelBi has been identified with the

singleton{Bi}.

It is important to remark that the functionB : I → P(X) deter-mines the elements ofS and S

∗g , and the cardinal of the setI ⊂ R

determines the cardinal ofS and therefore the cardinal ofS∗g .

Theclassical orders of magnitude qualitative spaces[22] verifies theconditions of the generalized model that has just been introduced.This model are build from a set of ordered basic qualitative labelsdetermined by a partition of the real line.

Let X be the real interval[a1, an), and a partition of this set givenby {a2, . . . , an−1}, with a1 < a2 < . . . < an−1 < an. The set ofbasic labels is

S = {B1, . . . , Bn−1},

where, for1 ≤ i ≤ n − 1, Bi is the real interval[ai, ai+1).The set of indexes isI = {1, 2, . . . , n − 1}.

a1 a2 an−1 an. . .

B1 Bn−1

Figure 1. Classical aualitative labelsSn

For1 ≤ i < j ≤ n − 1 the non-basic label[Bi, Bj) is:

[Bi, Bj) = {Bi, Bi+1, . . . , Bj−1},

and it is interpreted as the real interval[ai, aj).

For1 ≤ i ≤ n − 1 the non-basic label[Bi, B∞) is:

[Bi, B∞) = {Bi, Bi+1, . . . , Bn−1},

and it is interpreted as the real interval[ai, an).

The complete universe of description for the Orders of MagnitudeSpace is the set

Sn = { [Bi, Bj) | Bi, Bj ∈ S , i ≤ j} ∪ { [Bi, B∞) | Bi ∈ S},

which is called the absolute orders of magnitude qualitative spacewith granularityn, also denotedOM(n). In this case,S∗

g = {∅} ∪Sn .

There is a partial order relation≤P in Sn “to be more precise than”,given by:

L1 ≤P L2 ⇐⇒ L1 ⊂ L2.

The least precise label is denoted by? and it is the label[B1, B∞),which corresponds to the interval[a1, an).

BiB1 ... BnBj

?

p

r

e

c

i

s

i

o

n

... ...

[Bi,Bj)

a

b

s

t

r

a

c

t

i

o

n

.

Figure 2. The spaceSn

32

This structure permits working with all different levels of precisionfrom the label ? to the basic labels.

In some theoretical works, orders of magnitude qualitative spacesare constructed by partitioning the whole real line(−∞, +∞) in-stead of a finite real interval[a1, an). However, in most real worldapplications involved variables do have a lower bounda1 and an up-per boundan, and then values less thana1 or greater thanan areconsidered as outliers and they are not treated like any other.

The classical sign algebraS = {−, 0, +} was the first absolute or-ders of magnitude space considered by the QR community. It cor-responds to the caseS = {B−1 = (−∞, 0), B0 = {0}, B1 =(0, +∞)}. The sign algebra is obtained via a partition of the realline given by an unique landmark0. The classical orders of magni-tude qualitative spaces are built from partitions via a set of landmarks{a2, . . . , an−1}, and the classical interval algebra is built from thefinest partition of the real line whose landmarks are all real numbers.

It is important to remark the significance of the presented mathemat-ical formalism in the sense that it permits to lump together a familyof S

∗g forming a continuum from the sign algebraS = {−, 0, +} to

the interval algebra corresponding toS = R.

3 THE MEASURE SPACE (P(X),Σ( S∗g ), µ∗

)

To introduce the classical concept of entropy by means of qualitativeorders of magnitude spaces, Measure Theory is required. This the-ory seeks to generalize the concept of “length”, “area”and “volume”,understanding that these quantities need not necessarily correspondto their physical counterparts, but may in fact represent others. Themain use of the measure is to define the concept of integration for or-ders of magnitude spaces. First, it is necessary to define the algebraicstructure on which to define a measure.

Definition 5 A class of setsℑ is called asemi-ringif the followingproperties are satisfied:

1. ∅ ∈ ℑ.2. If A, B ∈ ℑ, thenA ∩ B ∈ ℑ.3. If A, B ∈ ℑ, A ⊂ B, then ∃n ∈ N, n ≥ 1 and

∃D1, D2, . . . , Dn such thatA = D0 ⊂ D1 ⊂ . . . ⊂ Dn = B,with Dk − Dk−1 ∈ ℑ,∀k ∈ {1, . . . , n}.

Proposition 1 S∗g is a semi-ring.

Proof:

1. ∅ ∈ S∗g by definition.

2. If [Bi, Bj), [Bk, Bl) ∈ S∗g , it is trivial to check that[Bi, Bj) ∩

[Bk, Bl) ∈ S∗g , taking into account the relative position between

the real intervals[i, j) and[k, l).Analogously, in the case of intersections[Bi, Bj) ∩ [Bk, B∞) or[Bi, B∞) ∩ [Bk, B∞).

3. If [Bi, Bj), [Bk, Bl) ∈ S∗g such that[Bi, Bj) ⊂ [Bk, Bl), then

two cases are considered:

(a) If Bk = Bi or Bl = Bj , it suffices to takeD0 = [Bi, Bj) andD1 = [Bk, Bl).

(b) Otherwise, takeD0 = [Bi, Bj), D1 = [Bi, Bl) andD2 =[Bk, Bl).

The cases[Bi, Bj) ⊂ [Bk, B∞) and [Bi, B∞) ⊂ [Bk, B∞)are proved in a similar way.

Definition 6 A classA of subsets of a non-empty setX is calledanalgebrawhen it contains the finite unions and the complements ofits elements. If finite unions are replaced by countable unions, it iscalled aσ-algebra.

The smallestσ-algebra that containsS∗g ⊂ P(X) is called theσ-

algebra generated byS∗g , denoted byΣ( S

∗g ).

Definition 7 LetX be a non-empty set andC ⊂ P(X), with∅ ∈ C.A measure onC is an applicationµ : C → [0, +∞] satisfying thefollowing properties:

1. µ(∅) = 0.2. For any sequence(En)∞n=1 of disjoint sets ofC such that

∪+∞n=1En ∈ C, then

µ(

+∞[

n=1

En) =

+∞X

n=1

µ(En).

Any measureµ on the wholeP(X), when it is restricted toS∗g , gives

a measure onS∗g .

Definition 8 Let µ be a measure onS∗g . Theouter measureon an

arbitrary subsetA of X is defined by:

µ∗(A) = inf{X

k∈N

µ([Bsk, Btk

)), A ⊂[

k∈N

[Bsk, Btk

)}.

Caratheodory theorem [8] assuresµ∗ of definition 7 is a measureon Σ( S

∗g ), and(P(X), Σ( S

∗g ), µ∗) is called a measure space. It is

proved that, sinceS∗g is a semi-ring,µ∗

| S∗g= µ.

In this measure space an integration with respectµ∗ can be defined.Because of the fact thatµ∗

| S∗g= µ, in any integration onS

∗g the

measureµ∗ can be replaced byµ.

4 ENTROPY BY MEANS OF S∗g

Once the integration inS∗g has been defined, entropy can then be

considered. To introduce the concept of entropy by means of qual-itative orders of magnitude, it is necessary to consider the qualita-tivization function between the set to be qualitatively described andthe space of qualitative labels,S

∗g .

33

To simplify the notation, let us express with a calligraphic letter theelements inS

∗g ; thus, for example, elements[Bi, Bj) or [Bi, B∞)

shall be denoted asE .Let Λ be the set that represents a magnitude or a feature that is qual-itatively described by means of the labels ofS

∗g . SinceΛ can repre-

sent both a continuous magnitude such as position and temperature,etc., and a discrete feature such as salary and colour, etc.,Λ could beconsidered as the range of a function

a : I ⊂ R → Y,

whereY is a convenient set. For instance, ifa is a room temperatureduring a period of timeI = [t0, t1], Λ is the range of temperaturesduring this period of time. Another example can be considered whenI = {1, . . . , n} andΛ = {a(1), . . . , a(n)} aren number of peoplewhose eye colour we aim to describe. In general,Λ = {a(t) =at | t ∈ I}.

The process of qualitativization is given by a function

Q : Λ → S∗g ,

whereat 7→ Q(at) = Et = minimum label (with respect to the in-clusion⊂) which describesat, i.e. the most precise qualitative labeldescribingat. All the elements of the setQ−1(Et) are ”representa-tives” of the labelEt or “are qualitatively described” byEt. They canbe considered qualitatively equal.

The functionQ induces a partition inΛ by means of the equivalencerelation:

a ∼Q b ⇐⇒ Q(a) = Q(b).

This partition will be denoted byΛ/ ∼Q, and its equivalence classesare the setsQ−1(Q(aj)) = Q−1(Ej), ∀j ∈ J ⊂ I . Each of theseclasses contains all the elements ofΛ which are described by thesame qualitative label.

Definition 9 Letµ be a measure onS∗g such that

Z

[

i∈I

{Bi}dµ = 1.

The entropyH with respect the partitionΛ/ ∼Q is the integral:

H(Λ/ ∼Q) = −

Z

Q(Λ)

log µ dµ, (1)

whereQ(Λ) is the set of labels mapped by Q (logarithms are to thebase 2).The expression (1) can be written as:

H(Λ/ ∼Q) = −X

j∈J

log(µ(Ej))µ(Ej). (2)

As in most definitions of entropy, it gives a measure of the amount ofinformation. In Definition 9 entropy can be interpreted as the mea-sure of the amount of information that provides the knowledge ofΛby means ofQ.

Nevertheless, the inner features of the orders of magnitude structureconsidered introduce some differences between the entropy definedin (1) and the entropy defined by Rokhlin [15] and Shannon [18], ascan be seen in the following example:

Example 1 Suppose thatQ maps each element ofΛ to the samelabel E ∈ S

∗g ; then the induced partitionΛ/ ∼Q contains only

one class equal toΛ and the entropy defined in equation (1) isH(Λ/ ∼Q) = −µ(E) log µ(E). In the classical interpretation ofthe entropy, the knowledge aboutΛ induced by this particularQ willlead to an entropy equal to zero, because in the given situation it isunderstood that this trivial partition ofΛ provides no information atall. On the contrary, in the approach that has been presented in thispaper, althoughQ map the whole set to the same label it could givea certain information aboutΛ: the intrinsic information provided bythe measure of the label itself.

Two different measures that show this fact are considered in the fol-lowing examples. On the one hand, the first differs from Shannon’sclassical interpretation of entropy as noted in Example 1: althoughQ map each element ofΛ to the same labelE ∈ S

∗g entropy is not

equal to zero . On the other, the entropy corresponding to Example 3behaves like the classical interpretation of Shannon and Rokhlin, inthe sense just discussed. Example 2 takes into account the lengths ofthe intervals corresponding to the labels, and Example 3 is related tothe cardinality of the set of representatives of each label.

Example 2 Let us define a particular measureµ on {∅} ∪ Sn asfollows:For the basic labelsBi = [ai, ai+1), whith i = 1, . . . , n − 1, let

µ(Bi) =ai+1 − ai

an − a1.

This measure is proportional to the knowledge of imprecision aboutthe magnitude and it is normalized with respect to the “basic” knownrange given by the lengthan − a1. For non-basic labels the measureis, for i, j = 1, . . . , n − 1, i < j:

µ([Bi, Bj)) =

j−1X

k=i

µ(Bk) =aj − ai

an − a1,

andfor i = 1, . . . , n − 1:

µ([Bi, B∞)) =

n−1X

k=i

µ(Bk) =an − ai

an − a1.

Elements ofΛ represented by quite precise labels will provide a big-ger contribution to entropyH than those who are represented by lessprecise labels. Considering the particular case in whichQ maps allthe elements ofΛ to the same label:Q(Λ) = {E}, thenΛ/ ∼Q= ΛandH(Λ/ ∼Q) = −µ(E) log(µ(E)) 6= 0.

Example 3 Another interpretation of the entropy defined in equa-tion (1) is obtained by defining another measureµ over{∅ ∪ Sn asfollows: For eachEt ∈ {∅} ∪ Sn ,

µ(∅) = 0, µ(Et) = card(Q−1(Et))/card(Λ).

This case recovers the classical interpretation of Shannon andRokhlin in the sense that ifQ maps all the elements ofΛ to the samelabel, then the partition does not give information ofΛ because theentropy isH(Λ/ ∼Q ) = −1 · log 1 = 0. Moreover, the entropyreaches its maximum when different elements ofΛ are mapped todifferent labelsEt ∈ Sn , i.e., whenQ is an injective map fromΛonto Sn . This maximum isH(Λ/ ∼Q ) = log(cardΛ).

34

Example 4 This last example is presented to show why it is nec-essary n some practical problems the mathematical formalism de-veloped in this paper:in this paper. Actually, the best way to describethe evolution of a function is by means of the derivative. The measuretheory provides the mathematical framework to do so under certainconditions (these conditions are not explained in this paper but canbe found in [5]), a functionν defined on a measure space(X, σ, µ)can be derived respect to the measureµ, i.e. there exist a functionfsuch thatf = dν/dµ. The case below falls within the developmentframe of the AURA research project, which sets out to adapt soft-computing techniques to the study of the financial rating tendenciesby using qualitative reasoning. The main goal of the project is to usethese techniques to extract knowledge and allow prognosis. The rat-ing is an attempt to measure the financial risk of a given company’sbond issues. The specialized rating agencies, such as Standard&Poor’s, classify firms according to their level of risk, using both quan-titative and qualitative information to assign ratings to issues. Learn-ing the tendency of the rating of a firm therefore requires the knowl-edge of the ratios and values that indicate the firms’ situation and,also, a deep understanding of the relationships between them and themain factors that can modify these values. The processes employedby these agencies are highly complex and are not based on purelynumeric models. Experts use the information given by the financialdata, as well as some qualitative variables, such as the industry andthe country or countries where the firm operates, and, at the sametime, they forecast the possibilities of the firm’s growth, and its com-petitive position. Finally, they use an abstract global evaluation basedon their own expertise to determine the rating. Standard& Poor’sratings are labelledAAA,AA,A, BBB, BB, B,CCC, CC, C andD. From left to right these rankings go from high to low credit qual-ity, i.e., the high to low capacity of the firm to return debt.

The problem of classifying firms by using their descriptive variableshas already been tackled by several authors [1]. In [16] and [17] itis analyzed the variables that influence variations in ratings and howthis influence is expressed, but not the speed of rating tendencies(i.e., how “fast” or “slow” ratings change) by using orders of mag-nitude descriptions. The particular evolution of the rating of a givenfirm and its prediction from the previous rating and the values of itspresent financial ratios is currently being studied. The mathematicsinvolved use the measure theory and the entropy concept. In orderto simplify the notation and to give a glimpse at the problem, theone dimensional case is considered: letB1 = D, B2 = C, B3 =CC, . . . , B10 = AAA and let beS10 the absolute orders of mag-nitude space considered for describing the rating depending only onone variablex ∈ R. As mentioned before and in the references [16]and [17] the rating is a subjective valuation of several experts, so thebest way to describe it is by means of a functionR : R → S10.Therefore the rating tendency is a derivative of a function definedin a non-euclidean space. Moreover, the rating evolution sometimesneither increase nor decrease because for example in ax1 ∈ R isR(x1) = [B2, B4) andR(x1 + ǫ) = [B1, B5) whereǫ > 0. Whathappens betweenx1 andx1 + ǫ is that precision on rating is lost, orin other words, the entropy betweenx1 andx1 + ǫ has increased:if H(R(x)) = −

R

Qlog µdµ, thendH(R(x))/dµ = − log µ. This

example is an sketch of an application of the theory on the dynam-ics in S

∗g spaces, and its applications on financial problems. Further

development is needed and we are working on it.

5 CONCLUSION AND FUTURE WORK

This paper introduces the concept of entropy by means of absoluteorders of magnitude qualitative spaces. This entropy measures theamount of information of a system when using orders of magnitudedescriptions to represent it.

In order to define the concept of entropy within Qualitative Reason-ing framework, this paper adapts the basic principles of MeasureTheory to give the space of absolute orders of magnitude the nec-essary structure. With the presented structure, we obtain a family ofqualitative spaces forming a continuum from the sign algebra to theclassical interval algebra.

From a theoretical point of view, future research could focus on twolines. On the one hand, it could focus on the comparison of the givenentropy with the macroscopic concept of Caratheodory entropy. Onthe other hand, the adaptation of Measure Theory provides the the-oretical framework in which developing a rigorous analytical studyof functions between orders of magnitude spaces. The continuity anddifferentiability of these functions will allow the dynamical study ofqualitatively described processes.

Within the framework of applications, this work and its relatedmethodology will be orientated towards the modelization and theresolution of financial and marketing problems. Regarding finan-cial problems, the concept of entropy will facilitate the study of theevolution and variation of the financial ratings. On the other hand,entropy as a measurement of coherence and reliability is useful ingroup decision-making problems arising from retail marketing ap-plications.

Moreover, the introduced entropy will allow defining a conditionalentropy in this framework, which in turn will allow considering theRokhlin distance to be used in decision-making problems of rankingand selection of alternatives.

REFERENCES

[1] J.M. Ammer and N Clinton, ‘Good news is no news? the impact ofcredit rating changes on the pricing of asset-backed securities’, Techni-cal report, Federal Reserve Board, (July 2004).

[2] Thomas M. Cover and Joy A. Thomas,Elements of Information Theory,Wiley Series in Telecomunications, 1991.

[3] P. Dague, ‘Numeric reasoning with relative orders of magnitude’.AAAI Conference, Washington, (1993).

[4] P. Dague, ‘Symbolic reasoning with relative orders of magnitude’. 13thIJCAI, Chambery, (1993).

[5] G.N. Folland,Real Analysis: Modern Techniques and Their Applica-tions, Pure and Applied Mathematics: A Wiley-Interscience Series ofTexts, Monographs, and Tracks, John Wiley & Sons, Inc, 1999.

[6] K. Forbus,Qualitative Reasoning, CRC Hand-book of Computer Sci-ence and Engineering, CRC Press, 1996.

[7] Kenneth Forbus, ‘Qualitative process theory’,Artificial Intelligence, 24,85–158, (1984).

[8] Paul R. Halmos,Measure Theory, Springer-Verlag, 1974.[9] IJCAI. The orders of Magnitude Models as Qualitative Algebras, num-

ber 11th, 1989.[10] J. Kalagnanam, H.A. Simon, and Y. Iwasaki, ‘The mathematical bases

for qualitative reasoning’,IEEE Expert., (1991).[11] B. Kuipers, ‘Making sense of common sense knowledge’,Ubiquity,

4(45), (January 2004).[12] M.L. Mavrovouniotis and G. Stephanopoulos, ‘Reasoning with orders

35

of magnitude and approximate relations’. AAAI Conference, Seattle,(1987).

[13] A. Missier, N. Piera, and L. Trave, ‘Order of magnitude algebras: asurvey’,Revue d’Intelligence Artificielle, 3(4), 95–109, (1989).

[14] O Raiman, ‘Order of magnitude reasoning’,Artificial Intelligence, (24),11–38, (1986).

[15] V.A. Rokhlin, ‘Lectures on the entropy of eeasure ereserving eransfor-mations’,Russian Math. Surveys, 22, 1 – 52, (1967).

[16] Llorenc Rosello, Nuria Agell, Monica Sanchez, and Francesc Prats,‘Qualitative induction trees applied to the study of financial rating’, inArtificial Intelligence Research and Development, eds., Monique Po-lite, Thierry Talbert, Beatriz Lopez, and Joquim Melendez, pp. 47 – 54.IOS Press, (2006).

[17] Llorenc Rosello, Nuria Agell, Monica Sanchez, and Francesc Prats,‘Learning financial rating tendencies with qualitative trees’, in21st In-ternational Workshop on Qualitative Reasoning, ed., Chris Price, pp.142 – 146, (2007).

[18] Claude E. Shannon, ‘A mathematical theory of communication’,TheBell System Technical Journal, 27, 379 – 423, (1948).

[19] P. Struss, ‘Mathematical aspects of qualitative reasoning’,AI in Engi-neering, 3(3), 156–169, (1988).

[20] L. Trave-Massuyes, F. Prats, M. Sanchez, and N Agell, ‘Consistent rela-tive and absolute order-of-magnitude models’. 16th International Work-shop on Qualitative Reasoning, (2002).

[21] L. et al. Trave-Massuyes,Le raisonnement qualitatif pour les sciencesde l’ingenieur, Ed. Hermes, 1997.

[22] Modeles et raisonaments qualitatifs, eds., Louise Trave-Massuyes andPhilippe Dague, Hermes Science (Parıs), 2003.

36

Model-based Testing using Quantified CSPs: A MapMartin Sachenbacher and Stefan Schwoon1

Abstract. Testing is the process of stimulating a system with in-puts in order to reveal hidden parts of the system state. In this pa-per, we consider finding input patterns to discriminate between dif-ferent, possibly non-deterministic models of a technical system, aproblem that was put forward in the model-based diagnosis litera-ture. We analyze this problem for different types of models and testswith different discriminating strength. We show how the variants canbe uniformly formalized and solved using quantified CSPs, a game-theoretic extension of CSPs. The results of the paper are (1) a map ofthe complexity of different variants of the testing problem, (2) a wayto compute discriminating tests using standard algorithms instead ofad-hoc methods, and (3) a starting point to extend testing to a richerclass of applications, where tests consist of stimulation strategies in-stead of simple input patterns.

1 IntroductionAs the complexity of technical devices is growing, methods and toolsto automatically check such systems for the absence or presence offaults become increasingly important. Diagnosability asks whether acertain fault can ever go undetected in a system due to limited observ-ability. It has been shown how this question can be framed and solvedas a satisfiability problem [7, 13]. Testing instead asks whether thereexist inputs (test patterns) to stimulate a system, such that a givenfault will always lead to observable differences at the outputs. Forthe domain of digital circuits with deterministic outputs, it has alsobeen shown how this question can be framed and solved as a satisfi-ability problem [10, 6].

In this paper, we consider constraint-based testing for a broaderclass of systems, where the models need not be deterministic. Thereare several sources for non-determinism in model-based testing oftechnical systems: in order to reduce the size of a model – for exam-ple, to fit it into an embedded controller [16, 14] – it is common toaggregate the domain of continuous system variables into discrete,qualitative values such as ’low’, ’medium’, ’high’, etc. A side-effectof this abstraction is that the resulting models can no longer be as-sumed to be deterministic functions, even if the underlying systembehavior was deterministic. Another source is the test situation itself:even in a rigid environment such as an automotive test-bed, there areinevitably some variables or parameters that cannot be completelycontrolled while testing the device.

Notions of testing for non-deterministic models have been in-troduced in various areas. In the field of model-based reasoningwith logical (constraint-based) system descriptions, Struss [15] in-troduced the problem of finding so-called definitely discriminatingtests (DDTs), which asks whether there exist inputs that can unam-biguously reveal or exclude the presence of a certain fault in a sys-

1 Technische Universitat Munchen, Institut fur Informatik, Boltzmannstraße3, 85748 Garching, Germany, email: {sachenba,schwoon}@in.tum.de

tem, even if there might be several possible outputs for a given input.[15] provided a characterization of this problem in terms of relational(constraint-based) models, together with an ad-hoc algorithm to com-pute DDTs. Later work [9] extended the idea to systems modeled asautomata, which, using a fixed bound of time steps, are unfolded intoconstraint networks such that the former algorithm can be applied.Generating DDTs is a problem of considerable practical importance;the framework was applied to real-world scenarios from the domainof railway control and automotive systems [9]. In the field of au-tomata theory, [1, 5] have studied the analogous problem of gener-ating distinguishing sequences, which asks whether there exists aninput sequence for a non-deterministic finite state machine, such thatbased on the generated outputs, one can unambiguously determinethe internal state of the machine.

In this paper, we give an overview and establish connections be-tween these different notions of the testing problem, with a gener-alized form of constraint models serving as the glue. We show howthe different variants can be conveniently formulated using quanti-fied CSPs (QCSPs), an extension of CSPs to multi-agent (adversar-ial) scenarios. This leads to three contributions: first, we provide anoverview of the complexity landscape of model-based testing for dif-ferent combinations of discriminating strength and model types. Forexample, we observe that the problems of finding possibly discrim-inating tests and finding definitely discriminating tests for logicalmodels [15] have the same worst-case complexity, which is how-ever less than those of finding distinguishing sequences for automatamodels. Second, we map the various test generation problems toQCSP formulas, which, instead of devising ad-hoc algorithms as in[15, 9], enables to use off-the-shelf solvers in order to effectivelycompute tests. Third, we show that our QCSP (adversarial planning)formulation of testing can be straightforwardly extended to problemsthat require complex test strategies instead of simple input patterns,and thus go beyond the framework in [15, 9].

2 Quantified CSPs (QCSPs)In a constraint satisfaction problem (CSP), all variables are (implic-itly) existentially quantified; we wish to find an assignment for eachof the variables that satisfies all constraints simultaneously. Quanti-fied CSPs (QCSPs) are a generalization of CSPs that allow a subsetof the variables to be universally quantified:

Definition 1 (Quantified CSP) A QCSP φ = 〈Q,X,D,C〉 has theform

Q1x1 . . . Qmxm . C(x1, ..., xn)

where m ≤ n and C is a set of constraints over the variables X ={x1, . . . , xn} with domainsD = {d1, . . . , dn}, andQ is a sequenceof quantifiers where each Qi, 1 ≤ i ≤ m, is either an existential (∃)or a universal (∀) quantifier.

37

Definition 2 (Satisfiability of QCSP) The satisfiability of a QCSPφ = 〈Q,X,D,C〉 is recursively defined as follows. If Q is emptythen φ is satisfiable iff the CSP 〈X,D,C〉 is satisfiable. If φ is of theform ∃x1Q2x2 . . . Qnxn . C then φ is satisfiable iff there exists avalue a ∈ d1 such thatQ2x2 . . . Qnxn . C∧(x1 = a) is satisfiable.If φ is of the form ∀x1Q2x2 . . . Qnxn . C then φ is satisfiable iff forevery value a ∈ d1, Q2x2 . . . Qnxn . C ∧ (x1 = a) is satisfiable.

Compared to the classical CSP framework, QCSPs have more ex-pressive power to model particular aspects of real world problems,such as uncertainty or other forms of uncontrollability in the envi-ronment. For example, in game playing, they can be used to find awinning strategy for all possible moves of the opponent.

There exist a number of solvers for quantified formulas, mostof which use variants of search and local propagation, the dom-inating algorithmic approach for SAT/CSP problems. While suchsolvers are easy to implement because they build on existing tech-nology, their performance often turns out to be not competitive withthe alternative approach of expanding the problem into a classi-cal instance (SAT/CSP) and using a SAT/CSP solver. However, ithas recently been shown [4] that more advanced algorithmic pre-processing and inference techniques, which usually do not pay off forclassical problems, often work well for quantified problems, and canmake QBF/QCSP-approaches several orders of magnitudes fasterthan classical approaches. It is therefore expected that QBF/QCSPsolvers will see significant performance improvements in the future,similar to those that SAT/CSP solvers have undergone in the past.

3 Discriminating Tests for Logical Models

We briefly review the theory of constraint-based testing of physi-cal systems as introduced in [15]. Testing attempts to discriminatebetween hypotheses about a system – for example, about differ-ent kinds of faults – by stimulating the system in such a way thatthe hypotheses become observationally distinguishable. Formally, letM =

⋃i Mi be a set of different models (hypotheses) for a sys-

tem, where each Mi is a set of constraints over variables V . LetI = {i1, . . . , in} ⊆ V be the subset of input (controllable) vari-ables, O = {o1, . . . , om} ⊆ V the subset of observable variables,andU = {u1, . . . , uk} = V −(I∪O) the remaining (uncontrollableand unobservable) variables. The goal of testing is then to find as-signments to I (input patterns) that will cause different assignmentsto O (output patterns) for the different models Mi:

Definition 3 (Discriminating Tests) An assignment tI to I is a pos-sibly discriminating test (PDT), if for all Mi there exists an assign-ment tO to O such that tI ∧Mi ∧ tO is consistent and for all Mj ,j 6= i, tI ∧Mj ∧ tO is inconsistent. The assignment tI is a definitelydiscriminating test (DDT), if for all Mi and all assignments tO to O,if tI ∧Mi ∧ tO is consistent then for all Mj , j 6= i, it follows thattI ∧Mj ∧ tO is inconsistent.

For example, consider the (simplified) system in Fig. 1. It consistsof five variables x, y, z, u, v, where x, y, z are input variables and v isan output variable, and two components that compare signals (x andy) and add signals (u and z). The signals have been abstracted intoqualitative values ’low’ (L) and ’high’ (H); thus, for instance, valuesL and H can add up to the value L or H, and so on. Assume we havetwo hypotheses about the system that we want to distinguish fromeach other: the first hypothesis is that the system is functioning nor-mally, which is modeled by the constraint set M1 = {fdiff , fadd}.

xxy u

v

fdiff

fzvfadd

x y u

L L LL H H

u z v

L L LL H L

fdiff: fadd: u z v

L L LL H L

fadd‐stuck:

H L HH H L

L H HH L LH L H

H L LH H L

H H H

Figure 1. Circuit with a possibly faulty adder.

The second hypothesis is that the adder is stuck-at-L, which is mod-eled by M2 = {fdiff , faddstuck}. Then for example, the assignmentx ← L, y ← H, z ← L is a PDT for M (it leads to the observationv = L or v = H for M1, and v = L for M2), while the assignmentx ← L, y ← H, z ← H is a DDT for M (it leads to the observationv = H for M1, and v = L for M2).

In the following, we restrict ourselves to the case where there areonly two possible hypotheses, for example corresponding to normaland faulty behavior of the system. Note that DDTs are then symmet-ric: if tI is a DDT to discriminateM1 fromM2, then it is also a DDTto discriminate M2 from M1.

3.1 Characterizing PDTs and DDTsWe sketch how for logical (state-less models), finding PDTs andDDTs can be characterized as a game played between two opponents.The first player (∃-player) tries to reveal the fault by choosing inputvalues for which the two hypotheses yield disjunct observations. Thesecond player (∀-player) instead tries to hide the fault by choosingvalues for outputs or internal variables such that the two hypothesesyield overlapping observations. In the case of PDTs, he can choosevalues only for internal variables, whereas in the case of DDTs, hecan choose values both for internal and observable variables. Boththe ∃-player and the ∀-player must adhere to the rules that they canonly choose among values that are consistent with the model of thesystem, as not all values are possible in all situations (there mightalso be additional rules for the ∃-player such that he can only chooseamong allowed inputs, but without loss of generality, we do not con-sider such restrictions here). The goal of the game is that exactly onehypothesis becomes true. Clearly, a PDT or DDT then exists if andonly if the first player has a winning strategy.

Thus, the first form of testing in Def. 3, finding PDTs, correspondsto solving a QCSP and is captured by the formula

∃i1 . . . in ∃o1 . . . om ∀u1 . . . uk .M1 → ¬M2 (1)

In analogy to (1), we can capture the second (stronger) form oftesting, finding DDTs, by the following QCSP formula:

∃i1 . . . in ∀o1 . . . om ∀u1 . . . uk .M1 → ¬M2 (2)

Note that for the case where M1 and M2 comprise only of deter-ministic functions, the quantification over the output variables ranges

38

over a single possible assignment, and thus (1) and (2) will have thesame solutions (PDTs and DDTs become equivalent).

The two problems of finding a PDT and a DDT can be embeddedinto the polynomial time hierarchy:

Proposition 1 (Complexity of PDTs and DDTs) The problem offinding PDTs and DDTs is ΣP

2 -complete.

Because the complexity class ΣP2 is believed to lie between NP and

PSpace, this means that the problem of finding tests for logical mod-els is more complex than solving CSPs, but less complex than theproblem of finding tests for automata models (see Sec. 4).

The QCSP formulation allows us to use standard QCSP/QBFsolvers in order to actually compute tests (see Sec. 6), as opposedto devising special algorithms as in [15, 9].

4 Discriminating Tests for Automata ModelsIn this section, we extend the notion of hypotheses (models) to bediscriminated from the case of logical (state-less) models to the moregeneral case of dynamic models whose state can change over time, asfor instance used in NASA’s Livingstone [17] or MIT’s Titan model-based system [16]. This means that we are no longer searching fora single assignment to input variables, but rather for a sequence ofinputs over different time steps. The following two definitions areadapted from [7]:

Definition 4 (Plant Model) A (partially observable) plant is a tupleP = 〈x0, X, I, δ, O, λ〉, where X, I,O are finite sets, called thestate space, input space, and output space, respectively, x0 ∈ X isthe start state, δ ⊆ X × I × X is the transition relation, and λ ⊆X ×O is the observation relation.

For technical convenience, we henceforth assume that in all ourplants δ and λ are complete, that is for every x ∈ X and i ∈ I thereexists at least one x′ such that (x, i, x′) ∈ δ and at least one o ∈ Osuch that (x, o) ∈ λ.

The intuitive meaning of a plant is as follows:X is the set of statesthat the plant can assume, and the state is not revealed to the observer.When the plant is in state x, input i will cause the state to changefrom x to x′ provided that (x, i, x′) ∈ δ. Moreover, it can emit theobservable output o provided that (x, o) ∈ λ.

We write δ(x, i, x′) for (x, i, x′) ∈ δ, and λ(s, o) for (x, o) ∈ λ.Note that a plant need not be deterministic, that is, the state after atransition may not be uniquely determined by the state before thetransition and the input. Likewise, a plant state may be associatedwith several possible observations.

Definition 5 (Feasible Trace) Let P = 〈x0, X, I, δ, O, λ〉 be aplant, and σ = i1, i2, . . . , ik ∈ I∗ be a sequence of k inputs andρ = o0, o1, . . . , ok ∈ O∗ be a sequence of k+1 outputs. Then (σ, ρ)is a feasible trace of P iff there exists a sequence σ = x0, x1, . . . , xk

of states such that δ(xj−1, ij , xj) for all 1 ≤ j ≤ k and λ(xj , oj)for all 0 ≤ j ≤ k.

A plant represents a hypothesis about the actual behavior of thesystem under test. Given two such hypotheses P1, P2 we are inter-ested in determining which of the hypotheses is true. To this end,our aim is to stimulate the system under test using a sequence of in-puts, and observe the output sequence; if we find that the observedoutput can be generated by one plant but not by the other, we knowwhich hypothesis is correct. In this sense we can extend the notion ofdiscriminating tests (Def. 3) from static systems to dynamic systems(plants):

Definition 6 (Discriminating Test Sequences) Given two plantsP1 = 〈x0, X, I, δ, O, λ〉 and P2 = 〈y0, Y, I, η, O, µ〉, a sequenceof inputs σ ∈ I∗ is a possibly discriminating test sequence (PDTS),if there exists a sequence of outputs ρ ∈ O∗ such that (σ, ρ) is afeasible trace of P1 but not of P2. The sequence σ is a definitely dis-criminating test sequence (DDTS) for P1 and P2, iff for all sequencesof outputs ρ, it holds that if (σ, ρ) is a feasible trace P1 then it is nota feasible trace of P2.

Notice that, due to our assumptions about completeness, for ev-ery input sequence σ there exist sequences ρ, τ such that (σ, ρ) isa feasible trace of P1 and (σ, τ) is a feasible trace of P2. PDTSsand DDTSs are equivalent to the notion of weak and strong tests asdefined in [5]: like PDTs and DDTs, a PDTS is a sequence that mayreveal a difference between two hypotheses, whereas a DDTS is a se-quence that will necessarily do so. In the case of deterministic plants,PDTSs and DDTSs coincide. Again, DDTSs are symmetric: a DDTSto discriminate P1 from P2 is also a DDTS to discriminate P2 fromP1.

For example, Fig. 2 shows two plants P1 and P2 with I = {L,H}and O = {0,1}. The input sequence σ = L,L is a PDTS for P2, P1,because, for example, 0,1,0 is a possible output sequence of P2 butnot of P1. The sequence σ′ = H,H is a DDTS for P2, P1, because theonly possible output sequence 0,0,0 of P2 cannot be produced by P1.

LL H

L,H LL

L,H

x0 x1L,H

0 1

x0 x1L

0 1x2

H L

x20L,H

Figure 2. Two plants P1 (left) and P2 (right).

4.1 Characterizing PDTSs and DDTSs

We give QCSP formulas that encode the problem of finding PDTSsand DDTSs with a length less or equal to k. Using QCSPs, feasibletraces of length k of a plant can be captured as follows: a sequenceof inputs and outputs is feasible, iff there exists a sequence of statessuch that for any two consecutive states x, x′ along the sequence, therespective input i and output o must be consistent with the transitionrelation δ and the observation relation λ:

φ(i1, . . . , ik, o0, . . . , ok, X, δ, λ) ≡ ∃x0, . . . , xk ∀x, x′, i, o .([∨k−1

j=0 (x = xj) ∧ (x′ = xj+1) ∧ (i = ij+1)]→ δ(x, i, x′))

∧ ([∨kj=0(x = xj) ∧ (o = oj)]→ λ(x, o)) (3)

From (3), we can construct a QCSP formula that encodes the prob-lem of finding a PDTS with a maximum path length of k:

∃i1, . . . , ik ∃o0, . . . , ok . φ(i1, . . . , ik, o0, . . . , ok, X, δ, λ)

∧¬φ(i1, . . . , ik, o0, . . . , ok, Y, η, µ) (4)

39

Extending on (4), the following QCSP formula captures DDTSswith a maximum path length of k:

∃i1, . . . , ik ∀o0, . . . , ok . φ(i1, . . . , ik, o0, . . . , ok, X, δ, λ)

→ ¬φ(i1, . . . , ik, o0, . . . , ok, Y, η, µ) (5)

We compare this to the approach in [9], which is based on un-rolling automata into a constraint network using k copies of the tran-sition relation and the observation relation, and then applying testmethods for logical models as discussed in Sec. 3. The advantageof the QCSP-based encoding (4,5) is that for any k, it requires onlya single copy of the transition relation and the observation relation,which are the biggest components in most automata model specifi-cations. Thus, the size of the formula will grow much more moder-ately with the number of time steps k than the constraint networkin [9]. However, it is still open to what extent current QCSP/QBFsolvers can exploit this more compact encoding of the test genera-tion problem, and turn it into actual performance improvements (seealso Sec. 6).

As for the complexity, for non-deterministic finite-state machinesit has been shown that the problem of uniquely identifying its ini-tial state from its input and output behavior is PSpace-complete [1].This problem is equivalent to the problem of designing a sequenceof inputs that allows to unambiguously distinguish among two non-deterministic finite-state machines (with known initial states), andtherefore equivalent to the problem of finding DDTSs:

Proposition 2 The problem of finding DDTSs is PSpace-complete.

To our knowledge, the complexity of finding PDTSs is still un-known, but it is likely that this problem is also PSpace-complete.

5 Adaptive TestingAs discussed above, the QCSP (game-theoretic) framework is usefulto compactly express, analyze and solve different variants of (known)model-based testing problems. However, in addition, it can also serveas a starting point to tackle new classes of problems that are closer tothe practice of testing. Recall that in Def. 3, tests are assumed to con-sist of (complete) assignments to controllable variables I . Actually,looking closely, there are two assumptions underlying this definition,namely that i) testing is performed as a two-step process where onefirst sets the inputs and then observes the outputs, and ii) the control-lable variables characterize all relevant causal inputs to the system.In the following, we seek to relax these two assumptions.

Relaxing the first assumption means to extend testing from theproblem of finding input assignments to the problem of finding adap-tive tests, where input variables can be set depending on the valuesof observed output variables. Such an adaptive sequence is in facta strategy that describes which values the input variables must begiven in response to the values of observed variables (represented,for example, as a decision tree). Generating such adaptive strategiesgoes beyond the theory in [15], which assumed that tests consist ofassignments (patterns) for the input variables, but it is possible in theQCSP framework. For logical models, adaptive tests can be capturedusing the following modified QCSP formula (assuming, without lossof generality, that the number of input variables equals the numberof output variables):

∃i1 ∀o1 . . .∃in ∀on ∀u1 . . . uk .M1 → ¬M2 (6)

While the non-adaptive version of DDTs (Sec. 3.1) is ΣP2 -

complete, the adaptive version (6) is harder to compute (PSpace-complete). For the case of (non-deterministic) automata models, we

get a similar picture: it has been shown [1] that finding such adaptivedistinguishing sequences is in ExpTime, and therefore even harderthan the problem of finding DDTSs. Surprisingly, for deterministicautomata models, the problem is polynomial and therefore easierthan the DDTS problem [11]. For model-based testing, this leadsto two interesting insights: first, since the class of adaptive tests(observation-dependent inputs) generalizes the class of non-adaptivetests (observation-independent inputs), from the two classes beingdifferent it follows that both for logical (constraint-based) modelsand for automata models, adaptive tests are strictly more power-ful in the sense that an adaptive test might exist even if a non-adaptive test does not exist. Second, the more general form of adap-tive (observation-dependent) testing is not just more powerful, butfor deterministic (or nearly deterministic) models it is even compu-tationally preferable over non-adaptive testing.

As already noted in the introduction, relaxing the second assump-tion (controllable variables characterize all relevant causal inputs tothe system) is often a practical necessity: during testing, even in ahighly controlled environment such as an automotive test-bed, theremight be variables or parameters that influence the system’s behav-ior, but whose values cannot be completely controlled. For logicalmodels, this scenario of testing under limited controllability can becaptured using a modification of (2). Let I be partitioned into inputvariables Ic = {i1 . . . is} that can be controlled (set during testing),and input variables Inc = {is+1 . . . in} that can be observed but notcontrolled. Then a definitely discriminating test exists iff the follow-ing formula is satisfiable:

∀is+1 . . . in ∃i1 . . . is ∀o1 . . . om∀u1 . . . uk .M1 → ¬M2 (7)

Again, this problem is strictly harder than the DDT problem(Sec. 3.1). Also note again that while solutions to (1) and (2) aresimply assignments to the values of the input variables, solutions to(7) are in general more complex and correspond to a strategy or pol-icy that states how the values of the controllable variables Ic must beset depending on the values of the non-controllable variables Inc. Toillustrate this, consider again the example in Fig. 1, but assume thatvariable x can’t be controlled. According to Def. 3, no DDT exists inthis case, as the possible observations for v will always overlap forthe two hypothesesM1 andM2. However, there exists a test strategyto distinguish M1 from M2, which consists of setting y dependingon the value of x: choose input y ← H, z ← H if x = L, and chooseinput y ← L, z ← H if x = H. Again, generating test for such sys-tems with limited controllability goes beyond the theory in [15], butit is possible in the QCSP framework.

We are currently working on merging the two sources of non-determinism in testing (non-deterministic behavior of the system andlimited controllability of the system) into one common frameworkfor QCSP-based adaptive testing.

6 Prototypic Implementation of QCSP-basedTesting

We have conducted preliminary experiments of QCSP-based testgeneration with the solvers Qecode [3] and sKizzo [2] (since thepresent version of Qecode does not allow one to extract solutionsfrom satisfiable instances, we transform the instance into QBF anduse sKizzo to extract solutions). So far, we have implemented sev-eral examples of non-adaptive and adaptive test generation for logicalmodels (Sec. 3), and a small example of non-adaptive test generationfor automata models (Sec. 4). However, at the moment these exam-

40

ples are still too small for a meaningful performance comparison ofour approach (non-adaptive case only) to the approach in [15, 9].

Figure 3 shows solutions generated from (2) and (7) for the exam-ple in Fig. 1. The solutions are represented in the form of BDDs withcomplemented arcs (see [2]), where ¬x stands for x ← L, x standsfor x ← H, etc. The left-hand side of the figure shows the strategy(in this case, a simple set of assignments) that is generated if vari-ables x, y, z are specified as controllable (input) variables, whereasthe right-hand side of the figure shows the strategy when only y, zare controllable (in this case, y must be set depending on the value ofx). No solution (definitely discriminating test strategy for the fault)is found if only z is assumed to be controllable.

¬x y y zz

x1

1

Figure 3. Test strategies generated for the example in Fig. 1.

7 Discussion and Future WorkWe reviewed an existing theory [15] of testing for physical systems,which defines a weaker (PDTs) and a stronger form (DDTs) of testinputs, and showed how it can be framed as QCSP solving. For thefirst time, we give precise results on the complexity of this prob-lem (in between NP and PSpace). Furthermore, we showed how as-sumptions in this theory about the complete controllability of systeminputs can be relaxed and lead to a strictly more powerful class oftests, where inputs are intelligently set in reaction to observed val-ues. Such test strategies go beyond the test pattern approach of theexisting theory, but they can be captured in the QCSP framework. Wealso extended the QCSP-based formulation of testing to the case ofplants modeled as non-deterministic automata.

While there exist approaches that solve non-deterministic testingproblems using classic constraint solvers [9] and model checkers [5],we believe that the QCSP-based representation can be advantageousfor several reasons:

• First, as noted in Sec. 4, the QCSP encoding is quite compact.While it is not yet clear if this theoretical advantage can indeedbe capitalized by current solver technology, there are at least hints[4] that it can lead to performance improvements as more sophis-ticated techniques are added to these solvers.

• Second, because QCSPs are kind of a natural generalization ofCSPs, it is not too difficult to lift extensions of CSPs such as softconstraints and optimization to QCSPs. In fact, the next releaseof the QCSP solver we used for our experiments (Qecode) con-tains optimization extensions. Thus, using the QCSP-based for-mulation, it will be relatively easy to extend model-based testingin order to generate, for instance, cost-optimal test strategies orprobabilistic test strategies that most likely discriminate fault hy-potheses.

• Third, existing methods to combat search space complexity byautomated abstraction of constraints can be straightforwardly ex-tended from CSPs to QCSPs and thus be adapted to the contextof model-based testing with limited effort. Based on our previouswork in this direction [14, 12] and related work in [8], we planto devise an abstraction-refinement method for constraint-basedtesting of hybrid systems.

We are also currently working on larger, more realistic examples toevaluate our QCSP-based testing approach. In particular, in the futurewe seek to complement passive verification tools [7] for embeddedautonomous controllers [16] with a capability to generate test strate-gies that can actively reveal faults.

ACKNOWLEDGEMENTSThe authors would like to thank Michael Esser, Paul Maier, and PeterStruss for useful comments.

REFERENCES[1] Rajeev Alur, Costas Courcoubetis, and Mihalis Yannakakis, ‘Distin-

guishing tests for nondeterministic and probabilistic machines’, in Pro-ceedings ACM Symposium on Theory of Computing, pp. 363–371,(1995).

[2] Marco Benedetti, ‘skizzo: A suite to evaluate and certify qbfs’, in Pro-ceedings CADE-05, (2005).

[3] Marco Benedetti, Arnaud Lallouet, and Jrmie Vautard, ‘Qcsp madepractical by virtue of restricted quantification’, in Proceedings IJCAI-07, pp. 38–43, (2007).

[4] Marco Benedetti and Hratch Mangassarian, ‘Experience and per-spectives in qbf-based formal verification’, Journal on Satisfiability,Boolean Modeling and Computation (JSAT), (2008). to appear.

[5] Sergiy Boroday, Alexandre Petrenko, and Roland Groz, ‘Can a modelchecker generate tests for non-deterministic systems?’, ElectronicNotes in Theoretical Computer Science, 190(2), 3–19, (2007).

[6] Sebastian Brand, ‘Sequential automatic test pattern generation by con-straint programming’, in Proceedings CP-01 Workshop on Modellingand Problem Formulation, (2001).

[7] Alessandro Cimatti, Charles Pecheur, and Roberto Cavada, ‘Formalverification of diagnosability via symbolic model checking’, in Pro-ceedings IJCAI-05, pp. 363–369, (2003).

[8] E. Clarke, A. Fehnker, Z. Han, B.H. Krogh, O. Stursberg J. Ouaknine,and M. Theobald, ‘Abstraction and counterexample-guided refinementin model checking of hybrid systems’, Journal of Foundations of Com-puter Science, 14(4), 583–604, (2003).

[9] Michael Esser and Peter Struss, ‘Fault-model-based test generation forembedded software’, in Proceedings IJCAI-07, pp. 342–347, (2007).

[10] Tracy Larrabee, ‘Test pattern generation using boolean satisfiability’,IEEE Transactions on Computer-Aided Design of Integrated Circuitsand Systems, 11(1), 4–15, (1992).

[11] David Lee and Mihalis Yannakakis, ‘Testing finite-state machines:State identification and verification’, IEEE Transactions on Computers,43(3), 306–320, (1994).

[12] Paul Maier and Martin Sachenbacher, ‘Constraint optimization and ab-straction for embedded intelligent systems’, in Proceedings CPAIOR-08, (2008). to appear.

[13] J. Rintanen and A. Grastien, ‘Diagnosability testing with satisfiabilityalgorithms’, in Proceedings of IJCAI-07, pp. 532–537, (2007).

[14] Martin Sachenbacher and Peter Struss, ‘Task-dependent qualitative do-main abstraction’, Artificial Intelligence, 162(1–2), 121–143, (2005).

[15] Peter Struss, ‘Testing physical systems’, in Proceedings AAAI-94, pp.251–256, (1994).

[16] B.C. Williams, M. Ingham, S. Chung, and P. Elliott, ‘Model-based pro-gramming of intelligent embedded systems and robotic space explor-ers’, Proceedings of the IEEE Special Issue on Modeling and Design ofEmbedded Software, 91(1), 212–237, (2003).

[17] B.C. Williams and P. Nayak, ‘A model-based approach to reactive self-configuring systems’, in Proceedings of AAAI-96, pp. 971–978, (1996).

41

Model-Based SystemsModel-Based Systems The Model-Based Systems (MBS) paradigm refers to a...

Documents

Transcript of Model-Based SystemsModel-Based Systems The Model-Based Systems (MBS) paradigm refers to a...