DiLEO a domain ontology for the evaluation of digital libraries

dbis/dlib/ionio

2

structure

- Introduction- On the evaluation of digital libraries

- Modeling the evaluation of digital libraries- An ontological representation of the digital library evaluation

domain- Ontologies- DiLEO presentation

in short

- an ontology- for comparing instances- for the support of digital library evaluation planning

3

motive

- e development of a schema for the description and the comparison of evaluation instances.

- Outmost aim is to cover the disagreement among evaluation models through a structured and formal meta-model.

“the lack of globally accepted abstract evaluation models and methodologies can be counterbalanced by collecting, publishing and analyzing current research activities” [Führ et al., 2007]

- At the same time to develop a digital library evaluation planning tool.

“Every evaluation should begin with a well-crafted plan. Writing an evaluation plan is not just common sense. It is an essential roadmap to successful evaluation!” [Reeves et a., 2003].

4

on the evaluation of digital librariespart a

modeling evaluation

- We do not refer to digital library evaluation models, but to the modeling of the process itself.

- Five main works:- In Tefko Saracevic’s classi"cation [2004]- In the Evaluation Computer [Kovacs & Micsik, 2004]- In the PRET A Rapporter framework [Blandford et al., 2007]- In the 5SQual model [Gonçalves et al., 2007] - In the Zachman Framework

6

Saracevic’s Classi"cation

- A classi"cation of evaluation studies according to:- what elements have been evaluated (Constructs)- which were the goals, the perspectives and so on (Context)- which were the perspectives that interested us (Criteria)- how the evaluation was conducted (Methodology)

7

Saracevic’s Classi"cation

- Divided the evaluation studies to these proposing evaluation models and to these reporting results of evaluation initiatives.

- Saracevic formed the concept of Context to encapsulate all high-level questions, such as why one evaluates, what is his/her target, etc.

- At the same time he developed a category to classify studies according to what was evaluated (Constructs) and two categories (Criteria, Methodology) to classify the studies according to how these are conducted.

8

the evaluation computer

- A faceted classi"cation of different views, which synthesize an instance of an evaluation or an ‘evaluation atom’.

- A calculation of the distance between two ‘atoms’ in a space.

9

the evaluation computer

- Five facets:- System- Content- Organization- User- Evaluation

10

PRET A Rapporter

- An evaluation framework that emphasized on the context of work. - According to the authors the framework holds features that assist

the planning of an evaluation.- e framework structures the evaluations according to:

- the purpose of evaluation- the resources and the constrains- the ethical considerations- the data gathering- the analysis of data- the reporting of "ndings

11

PRET A Rapporter

- PRET A Rapporter is moving case study-wise. In practice this means it focuses on particular dimensions in each of the three indicative studies that presents.- a formative evaluation of a system- a comparative evaluation of two interfaces of the same database- a qualitative study of a system in actual use.

12

5SQual

- e model is based on the well-known framework for the description of the digital libraries 5S (Streams, Structures, Spaces, Scenarios, & Societies).

- e model de"nes some dimensions (criteria) that correspond to constituting elements of the digital libraries.

- e authors refer to a series of studies, where these criteria are applied on digital libraries, such as ACM DL, CITIDEL and NDLTD.

13

14

Digital library concept Quality dimension 5S Concepts

Digital object Accessibility Societies (actor), Structures (metadata speci"cation), Streams + Structures (structured streams)

Pertinence Societies (actor), Scenarios (task)

Preservability Streams, Structures (structural metadata), Scenarios (process (e.g., migration)

Relevance Streams + Structures (structured streams), Structures (query), Spaces (Metric, Probabilistic, Vector)

Similarity Same as in relevance, Structures (citation/link patterns)

Signi"cance Structures (citation/link patterns)

Timeliness Streams (time), Structures (citation/link patterns)

Metadata speci"cation Accuracy Structure (properties, values)

Completeness Structure (properties, schema)

Conformance Structure (properties, schema)

Collection Completeness Structure (collection)

Catalog Completeness Structure (collection)

Consistency Structure (collection)

Repository Completeness Structure (collection)

Consistency Structure (catalog, collection)

Services Composability See Extensibility, reusability

Efficiency Streams (time), Spaces (operations, contraints)

Effectiveness See Pertinence, Relevance

Extensibility Societies + Scenarios (extends, inherits_from, rede"nes)

Reusability Societies + Scenarios (includes, reuses)

Reliability Societies + Scenarios (uses, executes, invokes)

the Zachman framework

- Zachman Framework is a framework for enterprise architecture, developed by John Zachman, IBM, early 1980.

- e framework re%ects a formal and high-level structured view of an organization. A taxonomy for the organization of structural elements of the organization under the lens of speci"c perspectives.

- It classi"es and organizes in a two-dimensional space all the concepts needed to be homogeneous and to express different planning perspectives.- According to the participants (alternative perspectives).- According to processes (questions).

15

16

WhatData

HowProcess

WhereLocation

Who Worker

WhenTiming

WhyMotivation

Scope [Planner]

CoreBusiness

Concepts

MajorBusiness

Transformations

BusinessLocations

PrincipalActors

Business Events

Mission& Goals

Business Model[Owner]

Fact Model TasksBusiness

ConnectivityMap

Work%owModels

BusinessMilestones

PolicyCharter

System Model[Evaluator]

DataModel

BehaviorAllocation

Platform &Communication

sMap

BRScriptsState

TransitionDiagrams

RuleBook

Technology Model

[Evaluator]

RelationalDatabaseDesign

ProgramSpeci"cations

Technical Plat-form &

Commu- nications Design

Procedure &Interface

Speci"cations

Work Queue & Scheduling

Designs

RuleSpeci"cations

Detail representation

[Evaluator]

DatabaseSchema

Source Code Network Procedures &

InterfacesWork Queues & Schedules

RuleBase

Functioning Bus[Evaluator]

OperationalDatabase

OperationalObject Cod

OperationalNetwork

OperationalProcedures &

Interfaces

OperationalWork Queues& Schedules

OperationalRules

the Zachman framework

an ontological representation of the digital library evaluation domain

part b

why an ontology?

- Formal models that help us:- understand a domain of knowledge; in this case the domain of

digital library evaluation.- to structure a knowledge base to collate different instances; in

this case instances portraying evaluations of digital libraries.- to infer a logical development; in this case to assist digital

library evaluation planning.

18

why an ontology?

- e previous schemas are located vertically in speci"c research areas. For example the PRET A Rapporter framework has a HCI view of things or the 5SQual examines the dimension of quality.

- ey de"ne concepts (constituents), either of the digital libraries, or of the evaluation, but not their in-between relationships.- e purpose is to use the ontology relationships and to

highlight the links between the concepts and to semantically strengthen them.

- It has the potential to express paths, which will reveal alternative or complementary concepts and threads.

19

ontologies

- We use elements such as:– classes (representing concepts, entities, etc.)– relationships (linking the concepts together)– functions (constraining the relationships in particular ways)– axioms (stating true facts)– instances (re%ecting examples of reality)

20

engineering process

- DiLEO is the result of some process:- Literature review and study

- selecting the proper concepts- continuously exploring the proper relationships

- Expressed in OWL- Validation

- through discussion and practice in the “Exploring perspectives on the evaluation of digital libraries” tutorial in ECDL 2010.

- through a focus group with "eld researchers.

21

a typical presentation of an evaluation

- Development in OWL with Protégé Ontology Editor- http://protege.stanford.edu/

22

the higher levels

Dimensionseffectiveness, performance measurement, service quality, technical excellence, outcomes assessment

Subjects

Objects

Characteristics

Levelscontent level, processing level, engineering level, interface level, individual level, institutional level, social level

Goalsdescribe, document, design

Research Questions

Dimensions Typeformative, summative, iterative

hasDimensionsType

isAffecting / isAffectedBy

isCharacterizing/isCharacterizedBy

isFocusingOnisAimingAt

isOperatedByisOperating

isDecomposedTo

23

isCharacterizing/isCharacterizedBy

the lower levels

Activityrecord, measure, analyze, compare, interpret, report, recommend

MeansComparison studies, expert studies, laboratory studies, "eld studies, logging studies, surveys

Factorscost, infrastructure, personnel, time

Means Typesqualitative, quantitative

Instrumentsdevices, scales, software, statistics, narrative items, research artifacts

Findings

Criteriaspeci"c aims, standards, toolkits

Metricscontent initiated, system initiated, user initiated

Criteria Categories

isSupporting/isSupportedBy

hasPerformed/isPerformedIn

hasSelected/isSelectedIn

hasMeansType

isMeasuredBy/isMeasuring

isUsedIn/isUsing

isGrouped/isGrouping

isSubjectTo

isDependingOn

isReportedIn/isReporting

24

connection of the levels

Dimensionseffectiveness, performance measurement, service quality, technical excellence, outcomes assessment

SubjectsLevelscontent level, processing level, engineering level, interface level, individual level, institutional level, social level

Research Questions Activityrecord, measure, analyze, compare, interpret, report, recommend

MeansComparison studies, expert studies, laboratory studies, "eld studies, logging studies, surveys

Findings

Objects

Metricscontent initiated, system initiated, user initiated

isAddressing

isAppliedTohasConstituent/isConstituting

hasInitiatedFrom

25

Relations Domain RangeisCitedIn / inverse: isCiting Appellations/study identi"er (AP/

stid)Appellations/study reference (AP/strf )

Constraints: max cardinality=1Constraints: max cardinality=1Constraints: max cardinality=1

hasDimensionsType Dimensions (D) Dimensions Type (DT)

Constraints: min cardinality=1, ∃ (formative ∪ summative ∪ iterative)Constraints: min cardinality=1, ∃ (formative ∪ summative ∪ iterative)Constraints: min cardinality=1, ∃ (formative ∪ summative ∪ iterative)

isAffecting inverse: isAffectedBy Dimensions (D) Level (L)

min cardinality =1, ∃ (content level ∪ engineering level ∪ processing level ∪ interface level ∪ individual level ∪ institutional level ∪ social level)min cardinality =1, ∃ (content level ∪ engineering level ∪ processing level ∪ interface level ∪ individual level ∪ institutional level ∪ social level)min cardinality =1, ∃ (content level ∪ engineering level ∪ processing level ∪ interface level ∪ individual level ∪ institutional level ∪ social level)hasConstituent / inverse: isConstituting

Dimensions (D) Activities (A)

Constraints: min cardinality =1, ∃ (record ∪ measure ∪ analyze ∪ compare ∪ interpret ∪ report ∪ recommend)Constraints: min cardinality =1, ∃ (record ∪ measure ∪ analyze ∪ compare ∪ interpret ∪ report ∪ recommend)Constraints: min cardinality =1, ∃ (record ∪ measure ∪ analyze ∪ compare ∪ interpret ∪ report ∪ recommend)isSupporting / inverse: isSupportedBy

Instruments (I) Activities (A)

Constraints: min cardinality =1, ∃ (record ∪ measure ∪ analyze ∪ compare ∪ interpret ∪ report ∪ recommend)Constraints: min cardinality =1, ∃ (record ∪ measure ∪ analyze ∪ compare ∪ interpret ∪ report ∪ recommend)Constraints: min cardinality =1, ∃ (record ∪ measure ∪ analyze ∪ compare ∪ interpret ∪ report ∪ recommend)

relationships

- Some of the forty (40) relationships

26

use of ontology

- We use threads of the ontology — paths — to express explicitly a process or a requirement. For example:

- Activities/analyze - isPerformedIn - Means/logging studies- hasMeansType - Means Type/quantitative

27

Activityrecord, measure, analyze, compare, interpret, report, recommend

MeansComparison studies, expert studies, laboratory studies, "eld studies, logging studies, surveys

Means Typesqualitative, quantitative

isPerformedIn hasMeansType

use of ontology

- Level/individual level - isAffectedBy - Dimensions/performance measurement - isFocusingOn - Objects/usage of content/usage of data - isOperatedBy - Subjects/human agents - isCharacterizedby - Characteristics/experience

- ... isCharacterizedby - Characteristics/discipline- ... isCharacterizedby - Characteristics/age

28

Dimensionseffectiveness, performance measurement, service quality, technical excellence, outcomes assessment

Subjectssystem agents, human agents

Objectsusage of content: usage of data, usage of metadata

Characteristicsage, count, discipline, experience, profession,

Levelscontent level, processing level, engineering level, interface level, individual level, institutional level, social level

isAffectedBy

isFocusingOnisOperatedBy

isCharacterizedby

instances

- Entry of instances in Protégé.

29

query examples

- We ask the knowledge base by issuing SPARQL queries- Assuming that we want to plan an evaluation with log "les.- During the evaluation planning we are interested in knowing

which were the research questions of relevant studies.- To mine this information from the knowledge base we need to

submit a SPARQL query.

30

SPARQL querySELECT DISTINCT ?Research_QuestionsInst ?MeansWHERE { ?Research_QuestionsInst a<Research_Questions>. ?Dimensions a<Technical_Excellence>. ?Activity a <Record>. ?Means a <Logs>. ?Research_QuestionsInst<isBelongingTo> ?Dimensions. ?Dimensions<hasConstituent> ?Activity. ?Activity<isPerformedIn> ?Means }

query examples

- the query and the answers will have this form:

31

answersthe research questions (in the "rst column) from two studies (wm2008c and

nzdl2000) that used log "les (in second column).

sources

- more on DiLEO:- G. Tsakonas & C. Papatheodorou (2011). “An ontological representation of

the digital library evaluation domain”. Journal of the American Society of Information Science and Technology 62(8), 1577–1593.

- related readings are located in:- http://www.mendeley.com/groups/731821/dileo/

32