Formative Evaluation of Ontologies for Information Agents
Transcript of Formative Evaluation of Ontologies for Information Agents
-
8/4/2019 Formative Evaluation of Ontologies for Information Agents
1/6
FORMATIVE EVALUATION OF ONTOLOGIES FOR INFORMATION
AGENTS
Muthukkaruppan Annamalai
Faculty of Computer Science and Software Engineering, MARA University ofTechnology, 40450 Shah Alam, Selangor, Malaysia ([email protected])
ABSTRACT
Information agents require knowledge about a particular
domain to gather useful information for humans. This
knowledge should ideally be stored externally in
ontologies. An application developer either builds or
simply adopts and extends existing domain ontologies to
describe the applications resources in meaningful ways
to information agents. In order to realise this potential,
the domain knowledge should be explicitly representedwithin `good ontologies. The question is how do we
know if an ontology being evolved is good, i.e., how do we
know if an ontology is appropriate for particular need? If
not, what is lacking in it and how it can be enhanced to
make it suitable for use? The answer emerging in the
knowledge engineering literature is ontology evaluation.
The evaluation of an ontology is crucial for its successful
use and reuse, yet it is acknowledgingly a difficult issue to
tackle. Up to now few works have been performed to this
end. This paper addresses the issue of formative
evaluation (validation and verification) of the ontologiesbeing developed for information agents. In particular, it
critically reviews the existing criteria based evaluation
practices to advance a more feasible set of criteria,
making additional distinctions about the aspects of the
ontology that came under the purview of each of the
evaluation criterion.
1. INTRODUCTION
Information agents can search and gather specific pieces
of information from the distributed knowledge sources in
a networked environment. The ability of an information
agent to perform this task effectively rests with the
knowledge it holds about the domain. This adaptable
knowledge must be described in a way that the
information agents can use during search and retrieval,
and hence warrants an ontology.
An ontology is a representation vocabulary,
specialised in some domain or subject matter. The
ontology defines a set of concepts for representing
specific facts in an instance of a domain (Chandrasekaran
and Josephson 1999). The emphasis is on the
conceptualisation of the conceptual terms in the
vocabulary intended to be captured. Consequently, the
notion that ontology is an explicit specification ofconceptualisation in a shared domain as proposed by
Gruber (1993) is the widely cited definition in artificial
intelligence (AI) and information systems (IS). Ontology
serves as a primary source for structuring and annotating
knowledge content, and subsequently provides machine-
readable semantic knowledge about the information in the
knowledge sources to the agent. A shared ontology
additionally facilitates a knowledge community to
exchange information in a domain.
A good domain ontology represents the modelled
world closely through consistently and coherentlyspecified definitions of concepts and relationships that
hold among them. It directs an information agent to infer
precise information from the knowledge contents that are
conceptually linked to this ontology. In contrast, an
ontology containing erroneous and incomplete definitions
does not provide a reliable semantic basis for applications
that depend on it. Consequently, a formative evaluation
of an ontology can help to identify its inherent
deficiencies that can be addressed while the ontology is
being developed to produce an effective conceptualisation
of the knowledge in the domain of discourse.
Unfortunately, the research and development onshared domain ontologies for information agents is still
largely confined to the academic environment. Since
researchers are more concerned about the technology that
utilises an ontology, passable ontologies are seen as
sufficiently meeting their current research needs. As
such, evaluating the goodness of domain ontologies has
not been a primary interest of researchers. This is evident
from the very small number of papers that have been
written about ontology evaluation so far. However, we
believe this lackadaisical attitude is set to change with the
advent of agents to facilitate knowledge enhanced search
and information retrieval. This will spur the production
of good domain ontologies for a wide range of application
domain. Formative evaluation plays a key function in the
development of a good ontology. The evaluation
corroborates the reliability of the ontology being built,
i.e., whether the conceptual definitions in the ontology are
well-founded. If not, the evaluation effort prompts us to
consider what is needed to produce an effective
conceptualisation that could adequately represent the
knowledge in the domain of discourse.
A domain ontology is often designed with the
potential of reuse in various applications in a domain area.
Therefore, a seemingly time saving strategy to create a
new ontology for describing a specific situation in adomain is by simply adopting an existing one. However,
-
8/4/2019 Formative Evaluation of Ontologies for Information Agents
2/6
in reality the domain specific knowledge needed by an
agent varies from application to application. Some
applications require more or less detailed knowledge
about a domain than others. As such, the evaluation of
the selected ontology is a precursory step to decide
whether that ontology is suitable for adoption. The
formative evaluation will expose the ontology's strengthsand weaknesses, and suggests the modifications and
extensions that are necessary to make the ontology
amenable to use.
In principle, an ontology should be evaluated on the
syntax, structure and semantics of its conceptual
definitions. The syntactic correctness of the definitions
could be checked using syntax analysers incorporated
within editors, parsers and validators. There has been
considerable work developing such tools. Discussing
them is beyond the scope of this paper, but information
about some of these tools can be found at ontology
language websites such as http://www.daml.org/tools/tools.html/ and http://www-ksl-svc.stanford.edu/ .
During the development of an ontology, we are mainly
concerned with the evaluation of the structure and
semantics of the conceptualised entities. In this paper, we
present a set of criteria to assess the structural and
semantic adequacy of a domain ontology in the context of
its applications in a domain area. In Section 2, we give a
brief description about ontologies for information agents.
In Section 3, we critically review the existing criteria
based evaluation practices to propose a more feasible set
of ontology evaluation criteria. The evaluation of
ontologies with respect to the prescribed set of evaluation
criteria is reviewed in Section 4. Finally, the concluding
section we summarise the contribution of this paper and
point to the direction of future work.
2. ONTOLOGIES FOR INFORMATION AGENTS
An ontology for information agents is a library of related
concepts, explicitly defined and formally organised into
subclasses, as a way of structuring and defining the
meaning of the represented conceptual terms. In general,
the ontology specifies classes of entities (concepts) in the
domain, relationships between these concepts and
properties attributed to them. Rigorously structuredontologies also describe functions and axioms (rules)
associated with these concepts to further constrain their
interpretation. In the simplest case, however, an ontology
is structured as a hierarchy of concepts related by
subsumption (subtype-supertype) relationships, reflecting
a taxonomy of conceptualisation.
3. RELATED WORK AND DISCUSSION
The importance of ontology evaluation is recognised in
existing literature on ontology development. At the same
time, it is commonly acknowledged that evaluating thequality of an ontology is a difficult issue to tackle.
Gruninger and Fox (1995) and, Uschold and King
(1995) highlight the evaluation of the reliability of a
developed ontology. The former uses a set of competency
questions to evaluate the suitability of an ontology, while
in the latter, the ontology is evaluated against its
requirement specification.
The importance of ontology evaluation is alsostressed by Fernandez et. al. (1997), except that the
evaluation is to be carried out as part of the ontology
development process; this paper shares a similar view.
A more serious effort to evaluate ontologies was
initiated by Gomez-Perez (1996). Drawing inspiration
from evaluation of knowledge systems, she emphasises
both validation and verification of the ontology content
are two important aspects of ontology evaluation. In
retrospect, the origin of these terms can be traced to
evaluation of software (Boehm 1981), the precursor for
knowledge systems. Validation and verification are
associated with checking the appropriateness and thecorrectness of the developed software, respectively.
Adapted to the evaluation of ontologies, validation checks
the suitability or appropriateness of the ontology being
designed, and verification checks the correctness and the
adequacy of the ontology.
The ontology is evaluated against its frame of
reference to ensure that it appropriately and adequately
satisfies its purpose of design. Without such frames of
references, it is difficult to check if the ontology is in
compliance with its purpose of design, i.e., conforms to
the understanding of what it is to be used for, and
competently satisfies its needs and requirements. The
common frames of references are requirement
specification, competency questions and, knowledge and
information resources in the subject domain.
Subsequently, the researchers in ontology
development have recommended numerous evaluation
criteria, on which the technical evaluation of the
ontologies can be based in order to guide the development
of ontologies having requisite quality of design. In
essence, the evaluation is a subjective means to check the
compliance of the ontology with respect to certain desired
properties in ontologies.
From the perspective of modelling in AI, Gruber
(1995) advocated the use of criteria such as clarity,coherence, extensibility, minimal encoding bias, and
minimal ontological commitment, which serve as
precedent for the design of an ontology. These notions
reverberate in contemporary ontology development
methodologies. For example, Uschold (1996) prescribed
that an ontology is evaluated using criteria as clarity,
consistency and reusability, which in retrospect, a set of
design criteria. In the same vein, Fox and Gruninger
(1998) suggested the use of criteria as perspicuity,
precision, generality, granularity, minimality
(conciseness), expressiveness and competence. Gomez-
Perez (1999, 2001) proposed consistency, completeness,conciseness and expansibility as worthy evaluation
criteria.
-
8/4/2019 Formative Evaluation of Ontologies for Information Agents
3/6
The modelling of knowledge in AI is strongly similar
to the modelling of the entities and relationships in the
conceptual schema. From the perspective of IS, the
entity-relationship model alludes to design qualities as
correctness, completeness, understandability, simplicity,
integration and implementability (Moody and Shanks
1994). The aforementioned design qualities cover muchof the same kind of ground as the evaluation criteria
mentioned in the previous paragraph.
It is obvious that the above researchers tend to
distinguish different set of criteria based on different
concerns in the design and evaluation of an ontology. In
spite of that, the quality demanded of some criteria
overlap with many others. We cite the following as
examples: Coherence and consistency correspond with
correctness; Clarity extends along with understandability,
perspicuity and expressiveness; Granularity and precision
accede to competency; Minimal ontological commitment,
generality and reusability are conformable criteria.However, accepting all of the above as valid design
criteria will give rise to conflict. For example, the
admittance of the level of expressiveness offered by
rigorous formalism for proper interpretation of defined
concepts, clashes with a simultaneous call for simplicity.
Likewise, there is a friction between clarity and
conciseness. The discord between reusability of the
modelled knowledge and its usability is another matter to
take issue.
We deem it is essential to bring together a
consolidated set of evaluation criteria that are compatible
with each other. Of the utmost necessity is the conceptual
definitions must be semantically consistent and coherent
with the modelled area of knowledge, and the definitions
must be consistent with each other, i.e., none of the
conceptual definitions in the ontology contradicts with the
other. Since, global consistency is not easy to enforce, we
should at least strive to achieve local consistency, i.e., an
ontology that is consistent with the view of a particular
user-community. An effective ontology must be
adequately complete to satisfy its needs and requirements.
An ontology void of unnecessary details is free from
ambiguity and redundancy, and additionally facilitates its
checking and maintenance. Because ontology
development is an evolutionary process, the design mustallow for incremental modifiability, i.e., it must be able to
be monotonically extended, conserving the existing
conceptualisation. Therefore, we regard consistency and
coherence, completeness, conciseness and, extensibility
and expansibility as important evaluation criteria of
ontologies.
A useful ontology should be sufficiently expressive,
whose concepts are defined at the appropriate level of
details and is adequately competent in meeting its purpose
of design. Representing the conceptualised model with
minimal encoding bias is yet another practical design
criteria. Hence, we set down expressibility, minimalencoding bias and competency as additional evaluation
criteria, which brings about the following discussion on
the impending design trade-offs.
On the trade-off between simplicity and
expressibility, we concur with the general view that
ontologies should be kept simple so that they are easy to
implement that encourages its reuse (Staab 2002).
Nevertheless, we will still argue that an ontology must besufficiently expressive to appropriately constrain the
possible interpretation of the conceptual terms that meets
its original purpose of design.
We also concur with Gruber that an ontology should
as far as possible be represented independent of symbol-
level encoding. Representing an ontology at an
intermediate level apart from its impending
implementation allows for the specification of
conceptualisation that is not severely restricted by
particular language limitation.
In relation to competency, there is a trade-off
between reusability and usability of an ontology. Themodelling for usability is directed by a purposive
mechanism, so that a defined concept can be applied more
directly to specify related knowledge content. On the
other hand, the modelling for reusability emphasises
generically represented vocabulary that is applicable
across many domains in a variety of situations.
Ontologies developed following the latter fashion of
design assert minimal ontological commitments, thus
calling for another design decision to make the necessary
extension and rework before they can be usefully applied.
Further more, the generic terms in the extended ontology
tend to cloud the conceptualisation with extraneous
knowledge that may be of little or no interest to the
knowledge community. Therefore, we argue that working
towards adequate ontological commitment to support the
desired competency should be viewed as an integral
aspect of ontology design, such that the ontology is at
disposal for use.
Hence, we propose consistency and coherence,
completeness, conciseness, extensibility and expansibility,
competency, expressiveness and minimal encoding bias as
a credible set of ontology evaluation criteria for
information agents.
4. ONTOLOGY EVALUATION CRITERIA
In this section, we discuss the evaluation of ontologies
with respect to the prescribed set of ontology evaluation
criteria; and making additional distinctions about the
aspects of the ontology that came under the purview of
each of the evaluation criterion.
4.1. Consistency and Coherence
The most important criteria in ontology evaluation are to
verify the consistency and coherence of the conceptual
definitions in the ontology. The verification checks onthe correctness and relevance of the concepts and the
properties associated with them. It involves two levels of
-
8/4/2019 Formative Evaluation of Ontologies for Information Agents
4/6
analyses. Firstly, we analyse the structural integrity of
individual conceptual definitions to check if each concrete
concept in the ontology corresponds to a specific entity in
the modelled area of domain knowledge, and the
definition is logically consistent with the intended
conceptualisation. We also check to ensure that each
individual assertion (instantiated concept) in the ontologyis true. Secondly, we analyse the definitions in the
ontology as a whole, which includes the abstract concepts.
Collectively, the definitions in the ontology must also be
consistent and coherent with each other, such that the
definitions in the ontology do not contradict with one
another.
4.1.1. Individual Conceptual Integrity
An ontology is comprised of a restricted set of concepts
related to an area of knowledge. For each concept, a set
of properties, i.e., attributes and relations linking it toother concepts are defined to facilitate its interpretation.
Information is used to crystallise a concrete concept's
structure with the appropriate semantic content such that
an instantiated concept can carry specific information
about the entity it represents.
The conceptual integrity verifies the individual
conceptual definitions with respect to the information
used to articulate a concept's structure. We also check to
ensure that there is no contradiction in the interpretation
of a concrete concept with respect to the entity it
represents.
4.1.2 Collective Consistency
Collective consistency analyses the coherence of the
definitions in the ontology by verifying the relationships
that bind the concepts. We verify the specialisation,
generalisation and equivalence relationships depicted in
subsumption hierarchical classification, the commonly
used scheme for structuring concepts in ontologies. We
also review the association or cross relationships that exist
between the concepts in the ontology, as well as the
reciprocal relationships that complement the cross
relationships such as inverse and transitive relationships.
The idea is to check whether the logical relationshipsdefined in the ontology intuitively reflect the
dependencies between their corresponding entities (both
concrete and abstract) in the modelled area of knowledge.
Like the concept structures, the relations can also be
structured in the ontology. A relation can be defined as a
specialised or generalised form of another relation; or it
can be specified as equivalent-to or inverse-of another
relation. In such case, the relational dependencies ought
to be analysed separately.
4.2. Completeness
If consistency and coherence criteria are used to verify the
correctness and relevancy of the concepts and the
properties defined in the ontology, the completeness
criterion checks whether the ontology has covered all the
relevant concepts and properties. Gomez-Perez (1999)
goes a step further by stating that an ontology is deemed
complete if each definition in the ontology is complete
with respect to the real world. This additional requirement
is voiced in a philosophical tone and demands that eachdefinition in the ontology is analysed with respect to its
coverage of the knowledge in the real world -- a
requirement that is hard to accomplish. Ontologies are by
nature incomplete; and we uphold the view that a domain
ontology is not a representational mechanism to answer
arbitrary questions about the domain.
Based on the above argument, our completeness
evaluation will go as far as to assess the functional
adequacy of the ontology in the context of its use and the
purpose of its design. The functional needs are stipulated
in the requirement specification and described by a pre-
generated set of competency questions. So, when we sayan ontology is complete, we actually mean the ontology is
functionally complete with respect to the area of
knowledge implied by its reference framework, rather
than ontologically complete with respect to the real world.
We will see later that a functionally complete ontology is
the basis for evaluating the competence of an ontology.
A functionally complete ontology meets all the
requirement specification of the modelling needs and/ or
is able to characterise the answers of all the competency
questions using the terms defined in the ontology. The
competency questions play an eminent role in the
completeness analysis of the modelled purposive domain
knowledge. The ontology needs to support not only the
statement of the answer, but also the derivation of the
answer for each of the competency questions. The
incompleteness of an ontology becomes evident when
either the requirement specifications are not satisfied or
the ontology cannot express the answer(s) to a
competency question posed to it. To address the
deficiency in the ontology, the missing concepts and the
properties have to be accounted for.
4.3. Conciseness
In contrast to the verification of completeness, theverification of conciseness checks to see whether there are
extraneous definitions in the ontology. This analysis
seeks to identify and remove redundant definitions that
are present in the ontology. In general, unnecessary and
unwanted definitions do not add value to the ontology.
Obvious redundancies are redefinition of existing
concepts in an ontology. Particularly, redefinition of
equivalent concepts using separate set of properties must
not be allowed because they are susceptible to differing
interpretation and use. The redefinitions that introduce
polysemous concepts can seriously impair the reliability
of the ontology. We believe such redundancy could beavoided by questioning the justification behind the
inclusion of each definition in the ontology. A more
-
8/4/2019 Formative Evaluation of Ontologies for Information Agents
5/6
demanding task is to check for inferred redundancies in
the ontology, i.e., whether a result can be deduced in more
than one way.
The point is to remove concepts and properties whose
presence in the ontology cannot be justified.
4.4. Extensibility and Expansibility
While an ontology carving a specific area of knowledge
can be shaped as a separate entity, it must be easy to
extend the ontology and expand its conceptual definitions.
The idea is to allow for incremental modification,
specialisation and adaptation of the ontology without
having to revise the existing definitions.
Extensibility checks to ensure that the concepts are
defined and organised in the manner that facilitate future
extension of the ontology. Expansibility checks if the
ontology allows expansionary modification such as
addition of new conceptual definitions and expansion ofexisting definitions without degenerating its present state
of being.
Clearly, the formalism in which an ontology is
represented is a tangible factor determining its
extensibility and expansibility. For example, the
hierarchically organised, objectified concept definitions
constructed using Frames provides a framework for
ordered representation that are easy to read and change,
and are also able to scale more easily as compared to say,
logical rules (Grosso et. al. 1999).
4.5. Competency
The competency criterion is specifically used to assess
whether the concepts in the ontology are defined with the
requisite level of detailness. The check on competency
helps to ensure that an ontology is capable of supporting
the purpose of its design. In general, the concepts in such
ontologies ought to be represented at a finer level of
granularity. The level of details that must be captured by
a conceptual definition is however dependent upon the
needs and requirements that arise in order to competently
utilise the modelled knowledge.
As a general rule, a competent ontology must also be
functionally complete. The detail representation offersdirect assistance for immediate adoption of the ontology.
So, in addition to the verification of completeness with
respect to the ontology's frame of reference, we also need
to verify the competency of an ontology by checking on
its ability to express relevant content of the knowledge
artefacts.
4.6. Expressiveness and Minimal Encoding Bias
As point out in Section 2, an ontology for information
agents must be specified with a certain degree of
formality. The ontology must be expressed using aformalism that is easy to read and understand so as to
facilitate its evaluation, maintenance, implementation and
uptake. In other words, the ontologies must be accessible
to both human and formal tools.
Although expressiveness and minimal encoding bias
are design decisions that is often factored into the
ontology representation formalism, an intermediate, semi-
formal representation of the conceptual definitions that is
as far as possible independent of particular symbol-levelencoding offers greater vantage in the design and
development of ontologies for information agents.
5. CONCLUSION
The formative evaluation of an ontology for information
agent encompasses validation and verification of the
ontology content to corroborate the reliability of the
ontology.
The validation of the ontology is necessary to ensure
that a `right' ontology is being built. For this, we propose
to validate the ontology against its reference frames.Common frames of references are requirement
specification, competency questions and, shared
knowledge and information resources in the domain of
discourse on which the development of the ontology was
based.
The verification of the ontology content is necessary
to ensure that the ontology is built `right'. For this, we
propose to check on the compliance of the ontology with
respect to a feasible set of ontology evaluation criteria,
namely consistency and coherence, completeness,
conciseness, extensibility and expansibility, competency,
expressiveness and minimal encoding bias.
At present, ontology evaluation is performed
manually. The question we ask is can suitable diagnostic
tools be developed to facilitate the formative evaluation of
ontologies. In addition to checking on the logical
correctness of the ontological definitions, it is interesting
to study to what extent completeness evaluation can be
automated. For instance how can we gather, formalise
and pose competency questions to ontologies using query-
answering tools in an organised manner? Conversely,
could we make it possible for ontology development tools
to systematically generate the competency questions that
it is capable of answering, which can be easily verified by
a knowledge engineer? Such supporting tools can largelyalleviate the burden of competency evaluation.
ACKNOWLEDGEMENT
The author expresses special thanks and gratitude to Prof.
Leon Sterling of The University of Melbourne for
inspiring the research on ontology and for his comments
on earlier writings on ontology design and evaluation.
-
8/4/2019 Formative Evaluation of Ontologies for Information Agents
6/6
REFERENCES
Boehm, B., 1981. Software Engineering Economics. Prentice-Hall. NJ.
Chandrasekaran, B and Josephson, J. R., 1999. What AreOntologies, and Why Do We Need Them? In IEEEIntelligent Systems, January/February 1999.
Fernandez, M., Gomez-Perez, A and Juristo, N., 1997.Methontology: From Ontological Art Towards OntologicalEngineering. In Proceedings of AAAI97 SpringSymposium on Ontological Engineering, StanfordUniversity, CA.
Fox, M and Gruninger, M., 1998. Enterprise Modelling. In AIMagazine, 19(3).
Gomez-Perez, A., 1996. A Framework to Verify KnowledgeSharing Technology. In Expert Systems with Application,11(4).
Gomez-Perez, A., 1999. Evaluation of Taxonomic Knowledgeon Ontologies and Knowledge-Based Systems. InProceedings of KAW99 Workshop on KnowledgeAcquisition for Knowledge-Based Systems, Alberta.
Gomez-Perez, A., 2001. Evaluation of Ontologies. InInternational Journal of Intelligent Systems, 16(3).
Grosso, W., Eriksson, H., Fergerson, R., Gennari, J., Tu, S andMusen, M. Knowledge Modeling at the Millennium (TheDesign and Evolution of Protege-2000). In Proceedings of
KAW99 Workshop on Knowledge Acquisition, Modellingand Management, Alberta.
Gruber, T., 1993. A Translation Approach to PortableOntologies. In Knowledge Acquisition, 5(2).
Gruber, T., 1995. Toward Principles for the Design ofOntologies Used for Knowledge Sharing. In InternationalJournal of Human Computer Studies, 43(5).
Gruninger, M and Fox, M., 1995. Methodology for the Designand Evaluation of Ontologies. In Proceedings of IJCAI95Workshop on Basic Ontological Issues in KnowledgeSharing, Montreal.
Moody, D and Shanks, S., 1994. What Makes a Good DataModel? Evaluating the Quality of Entity RelationshipModels. In Proceedings of ER94 Conference on Entity-Relationship Approach. Springer.
Staab, S., 2002. Trends and Controversies: Ontologies KISSESin Standardization. In IEEE Intelligent Systems,March/April 2002.
Uschold, M., 1996. Building Ontologies: Towards a UnifiedMethodology. In Proceedings of Expert Systems96Conference of the British Computer Society SpecialistGroup on Expert Systems, Cambridge.
Uschold, M and King, M., 1995. Towards a Methodology forBuilding Ontologies. In Proceedings of IJCAI95Workshop on Basic Ontological Issues in KnowledgeSharing, Montreal.