Experimenting with eXtreme Design

Eva Blomqvist, Valentina Presutti,Enrico Daga, and Aldo Gangemi STLab, ISTC-CNR, Rome, Italy

EKAW 2010Lisbon (Portugal) – 2010-10-12

Outline

• Background on Content ODPs• Problem – Research Questions• XD Tools and XD methodology• Experiment setup• Results of analysis

– Confirming previous results– XD Tools– XD Methodology

• Conclusions and future work

Background - ODPs

• It is difficult to construct good quality ontologies!– Lack of guidelines– Best-practices exist but how can we communicate them?

• Classes of problems in ontology design that can be solved by applying common solutions=> Ontology Design Patterns (ODPs)

• Content ODPs (CPs) – small ontologies with explicit documentation of design

rationales – modeling “good practices”

• An ontology can be seen as a…– Composition of CPs +– Dependencies +– Expansion

Background – ODPs (cont.)

• A Content ODP (CP) is always associated with requirements, usually expressed using Competency Questions (CQs)

• Example: InformationRealizationWhat are the physical realizations of this information object? What information objects are realized by this physical object?

Background – ODPs (cont.)

• Collected at http://www.ontologydesignpatterns.org• Currently: Logical-, Architectural-, Content-, Re-

engineering-, Alignment-, and Lexico-syntactic ODPs

Problem

• Overall question: What are the benefits of ODPs in ontology engineering? (Initial focus on Content ODPs)

• Study conducted in 2008-09 (reported at K-CAP’09)– Are Content ODPs perceived as useful?

• Yes!

– Are the ontologies constructed using Content ODPs ‘better’, in some modelling quality sense?

• Coverage of problem decreased (slower?) but major improvement in usability aspects, and fewer common mistakes

3. Are the tasks given to the participants solved faster when using Content ODPs?

• Not really, rather slower (too little experience?)

– How do participants use the Content ODPs provided, and what support would be beneficial?

• How to find and select ODPs? How to reuse them? Tools?

Problem (cont.)

• Since the previous study, we have developed:– The eXtreme Design (XD) methodology– XD Tools

• Summary of research questions:– Can we confirm the results from the previous study

(Questions 1-4)? + How is modularity affected?

– Does XD Tools support the process of reusing CPs?

– Does the XD methodology support the process of reusing CPs, and does it affect any of the aspects from the previous study (e.g., time, quality)?

Related Work

• Ontology Engineering methodologies– Classical methodologies are similar to software

engineering methodologies– Few focus on collaboration, and very few are pattern-

based – (Clark & Porter, 1997) and (Maas & Janzen, 2009)

– Most are not evaluated, just motivated through… • Example use case• Theoretical comparison

• Pattern-based tools– Mainly for Logical ODPs (e.g. support from OPPL,

and templates and wizards in Protégé)– Tool evaluation mostly as benchmarking/comparison

XD Tools

• Plugin for Eclipse-based OE environments (TopBraid Composer, NeOn Toolkit, etc.)

• Main parts– Registry browser– XD Selector– Specialization wizard– Annotation dialogue– XD Analyzer

XD Tools

Browse, search, and get

Content ODPs

Analyze your ontology against

good practices and patternsSpecialize, compose, annotate

ODPs and ontologies

XD Methodology

• An agile approach to ontology engineering• Two sets

– The problem space, i.e., the actual modeling issues (local problems)

– The solution space, i.e., reusable modeling solutions• Test-driven: unit tests on modules – integration

tests on overall ontology• Task-focused: solve only and exactly the

requirements at hand• Divide-and-conquer (design pairs) – collaboration

and integration are crucial aspects

XD Methodology (cont.)

…Step 5 - Select a coherent set of CQs, treating one

modelling issue, for development iteration.Step 6 - Match the CQs to CPs, candidate CPs for reuse

are identified.Step 7 - Select CPs to use: best fit to local problem

without unnecessary overhead.Step 8 - Reuse (import, specialize) and integrate

(compose, extend) selected CPs. Step 9 - Test (unit tests, e.g., through SPARQL queries)

and fix. Then start the next iteration.…

Experiment Setup

• 2 sessions, 35 participants in total (inexperienced, mainly master and PhD students, and junior researchers)

• Controlled settings: same tools, same training and instructions, same tasks, same patterns

• Procedure:

• One difference: Session 2 – XD Tools more mature

Experiment Setup (cont.)Task 1:• Context

The national association for promotion of theater in Italy wants to set up a web-based system for keeping track of details about theater productions and the actors at different theaters. In order to support reasoning about the productions, the system should be based on an ontology. …

• Story: theater productions During each year a number of theatre festivals are held in cities around Italy. In January 2007 a festival called “Roma Loves Shakespeare” took place in Rome. Two different productions of “The Merchant of Venice” participated, one from a theatre in Pisa and the other from a theatre institute in Venice, featuring an ensemble of university art students. Other plays were Othello and a Midsummer Night’s Dream …

• Competency questions (CQs) and contextual statements of theater production – When did a certain theatre festival take place? – Where did a certain festival take place?– What plays could be seen during a certain theatre festival? – In what city is a certain theatre located? – In what country is a certain city located? – What play is the basis of a certain production? – …Contextual statements: 9. A production has exactly one premier.

Evaluation of Results

• Analysis of questionnaire responses• Evaluation of the constructed ontologies

(Mainly functional and usability evaluations – see K-CAP’09 paper)

– Coverage of problem: terminological coverage and task coverage

– Usability: e.g., presence of labels, comments, inverse relations, disjointness, level of axiomatization

– Modelling mistakes: incomplete solutions, solutions with shortcomings

– Pattern usage: implicit and explicit

Results – Confirming previous conclusions?

1. Are CPs perceived as useful by the participants?– Confirmed – Increase for second session: Due to XD Tools?

• Are the ontologies constructed using CPs ‘better’, in some modelling quality sense?

– Coverage: Reduction of terminological coverage is no longer detected – Due to XD Tools?

– Usability: Confirmed – Most prominent improvement!

• Are the tasks solved faster when using CPs?– With tool support: no longer slower!

• What common modelling ‘mistakes’ can be identified, when not using patterns and when using CPs?

– Decrease in occurrence of most frequent mistakes confirmed (44% average decrease) - Same types of mistakes

Results - Modularity

• Do CPs increase the modularity of ontologies?– Task 1: no ontologies are modularized– Task 2: the ontologies contain on average 7.5

modules– Conclusion: Since the participants choose to

reuse the CPs as OWL-modules, rather than ideas for solutions, this inherently introduces modularity

Results – XD Tools

• How well does XD Tools support……finding CPs to reuse?…specializing CPs?

• Does XD Tools introduce overhead?

Results – XD Tools

• How well does XD Tools support……finding CPs to reuse?…specializing CPs?

Results – XD Tools

• How well does XD Tools support……finding CPs to reuse?…specializing CPs?

• Does XD Tools introduce overhead?

Results – XD methodology

• Is the XD methodology perceived as useful by the participants?

• Is the XD methodology a ‘natural’ way to work with CPs?

• Are the ontologies constructed using the XD methodology ‘better’, in some modelling quality sense, than the ontologies constructed ‘only’ using CPs?

• Are the tasks solved faster when using the XD methodology, compared to ‘only’ using CPs?

• What common modelling ‘mistakes’ can be identified and are they different from the ones noted when ‘only’ using CPs?

Results – XD methodology

• Is the XD methodology perceived as useful by the participants?– 94% claimed to follow the XD methodology closely

Results – XD methodology

• Is the XD methodology perceived as useful by the participants?

• Is the XD methodology a ‘natural’ way to work with CPs?

Results – XD methodology

• Is the XD methodology perceived as useful by the participants?

• Is the XD methodology a ‘natural’ way to work with CPs?

• Are the ontologies constructed using the XD methodology ‘better’, in some modelling quality sense, than the ontologies constructed ‘only’ using CPs?– Small increase from Task 2 to Task 3 (could be due to

maturation)

• Are the tasks solved faster when using the XD methodology, compared to ‘only’ using CPs?– Small increase – however, mistakes seem to be found more

easily, hence some parts seem to be done faster!

Results – XD methodology

• …• What common modelling ‘mistakes’ can be identified

and are they different from the ones noted when ‘only’ using CPs?– Continued decrease in occurrence of most frequent

mistakes from Task 2 to 3 (15% average decrease)

– Two types of errors decrease significantly more than the others:

• N-ary relations – decrease by 64%

• Missing datatype properties – decrease by 46%

– Conclusion: XD is test-driven – structured unit testing leads to finding certain types of mistakes

Future Work

• Currently ongoing– Development of the integration and collaboration part

of the XD methodology– Explorative experiments – 1 session performed

• From experimenting on the iteration of the design pairs to experimenting on the complete development team

• Things to be considered– Collaboration between pairs – integration pair?– Integration testing and refactoring– Information feedback loop– …

• Planned experiment– Comparative study of XD and other methodologies

Future Work (cont.)

• Additional experimental work– Other domains and tasks– New target groups – experts– Other types of patterns

• Presentation and representation issues– Analysing future ontologies (when patterns are more

wide-spread)• XD Tools

– Shortcuts for expert users to reduce overhead– Improve search functionalities

• XD methodology– Elaborate collaboration and integration aspects

Conclusions

• Confirmed conclusions– Content ODPs are perceived as useful– They improve mainly the usability aspects of the

ontologies, but also reduces the occurrence of a number of common ‘mistakes’

– Development is neither faster nor slower

• XD Tools– Supports the use of CPs – no longer slower– Perceived as useful, although adds some overhead

• XD methodology– Perceived as useful, and it is a natural way of working with

CPs (descriptive and pragmatic)– Main advantage seems to be testing – drastically

decreases some common ‘mistakes’

Thank you!

Questions?