Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson
description
Transcript of Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson
![Page 1: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/1.jpg)
Towards a Quality Model for Digital Libraries
DELOS Evaluation WorkshopPadova, Italy Oct. 4, 2004
Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. WatsonVirginia Tech, Blacksburg, VA 24061 [email protected] http://fox.cs.vt.edu/talks
![Page 2: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/2.jpg)
Acknowledgements (Selected)
Sponsors: ACM, Adobe, AOL, CAPES, CONACyT, DFG, IBM, NLM, NSF (IIS-9986089, 0086227, 0080748, 0325579; DUE-0121679, 0136690, 0121741, 0333601), OCLC, VTLS
VT Faculty/Staff: Debra Dudley, Weiguo Fan, Gail McMillan, Manuel Perez, Naren Ramakrishnan, …
VT Students: Yuxin Chen, Shahrooz Feizabadi, Nithiwat Kampanya, S.H. Kim, Aaron Krowne, Bing Liu, Ming Luo, Paul Mather, Fernando Das Neves, Unni. Ravindranathan, Ryan Richardson, Rao Shen, Ohm Sornil, Hussein Suleman, Ricardo Torres, Wensi Xi, …
![Page 3: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/3.jpg)
Special thanks Norbert Fuhr, Maristella Agosti, other
organizers and support team, DELOS, EU Last time in Padova and Venice was 1972 Have been frustrated by lack of interest in
DL evaluation for several years How can we move DL into a science without
agreement on evaluation? Would be happy to help connect this with
IEEE TCDL, NSDL, and other efforts
![Page 4: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/4.jpg)
Outline
Major Points of the Presentation Introduction to the 5S View of DLs
Informal Definition of DLs Formal Definition of DLs DL Ontology
Defining a Quality Model for DLs Quality and the Information Life Cycle An XML Log Standard for DLs Conclusions and Future Work
![Page 5: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/5.jpg)
Major Points of the Presentation New formalization for digital libraries: 5S Formalization of traditional (and new?)
measures within our 5S framework Contextualization of these measures within
the information life cycle, and some data Reminder of work on DL logging, in hopes
this can be refined and adapted/adopted Encourage wider sharing of tools,
collections, beyond original intent (e.g., INEX, ETDs)
![Page 6: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/6.jpg)
Outline
Major Points of the Presentation Introduction to the 5S View of DLs
Informal Definition of DLs Formal Definition of DLs DL Ontology
Defining a Quality Model for DLs Quality and the Information Life Cycle An XML Log Standard for DLs Conclusions and Future Work
![Page 7: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/7.jpg)
DL Services/Activities Taxonomy
BrowsingCollaboratingCustomizingFilteringProviding accessRecommendingRequestingSearchingVisualizing
AnnotatingClassifyingClusteringEvaluatingExtractingIndexing
MeasuringPublicizing
RatingReviewing (peer)
SurveyingTranslating (language)
ConservingConverting
Copying/ReplicatingEmulatingRenewing
Translating (format)
AcquiringCataloging
Crawling (focused)DescribingDigitizingFederatingHarvestingPurchasingSubmitting
PreservationalCreational
AddValue
Repository-Building
Information SatisfactionServices
Infrastructure Services
![Page 8: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/8.jpg)
Requirements Analysis Design Implementation Test
5S 5SLOO ClassesWorkflow Components
DLEvaluation
5SGraph 5SLGenFormalTheory/Metamodel
DL XMLLog
5S Framework and DL Development
![Page 9: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/9.jpg)
5SLGen: Automatic DL Generation
5S Meta
Model5SLGraph
DL Expert
DL Designer
5SL DL
Model
5SLGen
Practitioner
Researcher
TailoredDL
Services
Teacher
componentpool
ODLSearch,ODLBrowse,ODLRate,ODLReview,
…….
Requirements (1) Analysis (2)
Implementation (4)
Design (3)
![Page 10: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/10.jpg)
![Page 11: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/11.jpg)
Outline
Major Points of the Presentation Introduction to the 5S View of DLs
Informal Definition of DLs Formal Definition of DLs DL Ontology
Defining a Quality Model for DLs Quality and the Information Life Cycle An XML Log Standard for DLs Conclusions and Future Work
![Page 12: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/12.jpg)
Informal 5S Definitions: DLs are complex systems that
help satisfy info needs of users (societies) provide info services (scenarios) organize info in usable ways (structures) present info in usable ways (spaces) communicate info with users (streams)
![Page 13: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/13.jpg)
![Page 14: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/14.jpg)
Background: The 5S ModelStreams
Scenarios
Societies
Structures
Spaces
Static /Passive
Dynamic /Active
![Page 15: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/15.jpg)
5SsModels Examples Objectives
Stream Text; video; audio; image Describes properties of the DL content such as encoding and language for textual material or particular forms of multimedia data
Structures Collection; catalog; hypertext; document; metadata; organization tools
Specifies organizational aspects of the DL content
Spaces Measure; measurable, topological, vector, probabilistic
Defines logical and presentational views of several DL components
Scenarios Searching, browsing, recommending,
Details the behavior of DL services
Societies Service managers, learners, teachers, etc.
Defines managers, responsible for running DL services; actors, that use those services; and relationships among them
![Page 16: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/16.jpg)
Metamodels For “typical digital library”
Minimal DL Starts with digital object (e.g., born digital)
For scientific digital library, educational DL, cultural heritage DL, e-Gov DL, … Ex.: archaeological DL - ETANA-DL Starts with real object
![Page 17: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/17.jpg)
Digital Objects (DOs) Born digital Digitized version of “real” object
Is the DO version the same, better, or worse? Decision for ETDs: structured + rendered
Surrogate for “real” object Not covered explicitly in metamodel for a
minimal DL Crucial in metamodel for archaelogy DL
![Page 18: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/18.jpg)
Metadata Objects (MDOs) MARC Dublin Core RDF IMS OAI (Open Archives Initiative) Crosswalks, mappings Ontologies Topics maps, concept maps
![Page 19: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/19.jpg)
Repository Also called: digital rep., digital asset rep.,
digital object rep., institutional repository Stores and maintains digital objects (assets) Provides external interface for Digital
Objects: Creation, Modification, Access Enforces access policies Provides for content type disseminations
Adapted from Slide by V. Chachra, VTLS
![Page 20: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/20.jpg)
Other Key Definitionscoll, catalog, service, archive,
(minimal) DLSee Gonçalves et al. in April
2004 ACM Transactions on Information Systems (TOIS)
![Page 21: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/21.jpg)
Scope: see abstract Minimal DL: catalog, collection, digital object,
metadata specification, repository, and services Quality dimensions: accessibility, accuracy,
completeness, composability, conformance, consistency, effectiveness, efficiency, extensability, impact factor, pertinence, preservability, relevance, reliability, reusability, significance, similarity, and timeliness.
Measurement characteristics: response time (with regard to efficiency), cost of migration (with respect to preservability), and number of service failures (to assess reliability)
![Page 22: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/22.jpg)
Outline
Major Points of the Presentation Introduction to the 5S View of DLs
Informal Definition of DLs Formal Definition of DLs DL Ontology
Defining a Quality Model for DLs Quality and the Information Life Cycle An XML Log Standard for DLs Conclusions and Future Work
![Page 23: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/23.jpg)
The 5S Formal Model A digital library is a 10-tuple (Streams, Structs, Sps,
Scs, St2, Coll, Cat, Rep, Serv, Soc) in which: Streams is a set of streams, which are sequences
of arbitrary types (e.g., bits, characters, pixels, frames);
Structs is a set of structures, which are tuples, (G, ), where G= (V, E) is a directed graph and : (V E) L is a labeling function;
Sps is a set of spaces each of which can be a measurable, measure, probability, topological, metric, or vector space.
![Page 24: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/24.jpg)
The 5S Formal Model (2) Scs = {sc1, sc2, …, scd} is a set of scenarios where
each sck = <e1k({p1k}), e2k({p2k}), …, ed_kk({pd_kk})> is a sequence of events that also can have a number of parameters {pik}. Events represent changes in computational states; parameters represent specific locations in a state and respective values.
St2 is a set of functions : V Streams ( ) that associate nodes of a structure with a pair of natural numbers (a, b) corresponding to a portion of a stream.
![Page 25: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/25.jpg)
The 5S Formal Model (3) Coll = {C1, C2, …, Cf} is a set of DL collections where
each DL collectionCk = {do1k, do2k, …, dof_kk} is a set of digital objects.
Each digital object do is a tuple (h, SM, ST, StructuredStreams) whereh is a handle,SM is a set of streams,ST is a set of structural metadata specifications,StructuredStreams is a set of StructuredStream
functions defined from the streams in SM set and from the structures in the ST set.
![Page 26: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/26.jpg)
The 5S Formal Model (4)
Cat = {DMC_1, DMC_2, …, DMC_f} is a set of metadata catalogs for Coll where each metadata catalog DMC_k = {(h, msshk)}, and msshk = {mshk1, mshk2, …, mshkn_hk} is a set of descriptive metadata specifications. Each descriptive metadata specification mshki is a structure with atomic values (e.g., numbers, dates, strings) associated with nodes.
A repository Rep = {(Ci, DMC_i)} (i=1 to f) is a set of pairs (collection, metadata catalog) It is assumed there exists operations to manipulate
them (e.g., get, store, delete).
![Page 27: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/27.jpg)
The 5S Formal Model (5) Serv = {Se1, Se2, …, Ses} is a set of services where each service Sek = {sc1k,
.., scs_kk} is described by a set of related scenarios.
Soc = (C, R) where C is a set of communities and R is a set of relationships among communities. SM = {sm1, sm2, …, smj}, and Ac = {ac1, ac2, …, acr } are two such communities where the former is a set of service managers responsible for running DL services and the latter is a set of actors that use those services. Being basically an electronic entity, a member smk of SM
distinguishes itself from actors by defining or implementing a set of operations {op1k, op2k, …, opnk} smk. Each operation opik of smk is characterized by a triple (nik, sigik, impik), where nik is the operation’s name, sigik is the operation’s signature (which includes the operation’s input parameters and output), and impik is the operation’s implementation. These operations define the capabilities of a service manager smk.
![Page 28: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/28.jpg)
The 5S Formal Model (6) Soc = (C, R) where C is a set of communities and R is a set of
relationships among communities. SM = {sm1, sm2, …, smj}, and Ac = {ac1, ac2, …, acr } are two such communities where the former is a set of service managers responsible for running DL services and the latter is a set of actors that use those services. Being basically an electronic entity, a member smk of SM
distinguishes itself from actors by defining or implementing a set of operations {op1k, op2k, …, opnk} smk. Each operation opik of smk is characterized by a triple (nik, sigik, impik), where nik is the operation’s name, sigik is the operation’s signature (which includes the operation’s input parameters and output), and impik is the operation’s implementation. These operations define the capabilities of a service manager smk.
![Page 29: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/29.jpg)
BackgroundStreams
text
audio
image
video do mss
R
C DMcIc
Se
Sc
e
SM
Ac
op
Scenarios
Societies
Top
Pr
Metric
Measurable
Measure
Structures
Spaces
Vec
ms
![Page 30: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/30.jpg)
Background: 5S and DL formal definitions and compositions (April 2004 TOIS)
5S
structures (d.10)streams (d.9) spaces (d.18) scenarios (d.21) societies (d. 24)
structural metadataspecification(d.25)
descriptive metadataspecification(d.26)
repository(d. 33)
collection (d. 31)
(d.34)indexingservice
structured stream (d.29)
digitalobject (d.30)
metadata catalog (d.32)
browsingservice
(d.37)
searchingservice (d.35)
digital library(minimal) (d. 38)
services (d.22)
sequence (d. 3)
graph (d. 6)function (d. 2)
measurable(d.12), measure(d.13), probability (d.14), vector (d.15), topological (d.16) spaces
event (d.10)state (d. 18)
hypertext(d.36)
sequence (d. 3)
transmission(d.23)
relation (d. 1) language (d.5)
grammar (d. 7)
tuple (d. 4)*
![Page 31: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/31.jpg)
Reducing confusion, misnaming A “document” is a stream, with a
superimposed or externally understood structure, along with a use scenario. Structures: grammatical, rhetorical, markup
This could help us better address “Semi-structured information” “Unstructured information”
![Page 32: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/32.jpg)
Glossary: Concepts in the Minimal DL and Representing SymbolsConcept Symbol Digital object do Metadata specification ms Set of metadata specifications mss Collection C Catalog DMC Repository S Event e Scenario Sc Services Se Actor Ac Service Manager SM Operation op Society Soc
![Page 33: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/33.jpg)
Outline
Major Points of the Presentation Introduction to the 5S View of DLs
Informal Definition of DLs Formal Definition of DLs DL Ontology
Defining a Quality Model for DLs Quality and the Information Life Cycle An XML Log Standard for DLs Conclusions and Future Work
![Page 34: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/34.jpg)
Digital Library Formal OntologyStreams
text
audio
image
video do mss
R
C DMc
describes
stores
is_version_of
Ic
Se
Sc
e
extendsreuses
SM
Ac
opexecutes
participates_in
recipient
runs
Scenarios
Societies
inherits_from/includes
association
uses
Top
Pr Metric
Measurable
Measure
describes
employsproduces
employsproduces
employsproduces
Structures
Spaces
Vec
belongs_to
contains
ms
is_ais_a
precedeshappens_before
is_a
redefinesinvokes
contains
contains
![Page 35: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/35.jpg)
Searching Browsing
Ic
Acquiringinterests/needs
query anchor
universalcollection
C
DMC
Indexing
Society
ActorDescribingCataloguing
Linking
Hypertext
Infra-structure Services(fundamental)
Information Satisfaction Services(fundamental)
handle
{doi, i I}
Submitting
AuthoringDigitizing
doi
mskj
doj
p
p p
p
e
e
describes
p
p
p
e
e
p
e
pe
e
e
Requesting
e e
p
has
contains
![Page 36: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/36.jpg)
SearchingBrowsing
queryanchor
Society
actor
C, {doi, i I}
Recommending Filtering Binding Visualizing Expanding query
user model/expr query/category {doj, j J}
{dor, r R} {dof, f F}
biuk
InformationSatisfaction Services
spj query’
fundamental
Rating Training
Infrastructure
Services (Add_Value)
composite
Requesting
handle
p pp
e e e{(doi, acj, rij), i I, j }
p
e
e
p p p p p
e e
classCt
e ee e
e
p
e
Indexing
IC
p
e
transformer
e
![Page 37: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/37.jpg)
Outline
Major Points of the Presentation Introduction to the 5S View of DLs
Informal Definition of DLs Formal Definition of DLs DL Ontology
Defining a Quality Model for DLs Quality and the Information Life Cycle An XML Log Standard for DLs Conclusions and Future Work
![Page 38: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/38.jpg)
Defining Quality in Digital Libraries What’s a “good” digital Library?
Central Concept: Quality! Hypotheses of this work:
Formal theory can help to define “what’s a good digital library” by: Proposing and formalizing new quality measures
for DLs Formalizing traditional measures within our 5S
framework Contextualizing these measures within the
Information Life Cycle
![Page 39: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/39.jpg)
Defining Quality in Digital librariesDL Concept Dimensions of Quality Digital object Accessibility
Pertinence Preservability Relevance Similarity Significance Timeliness
Metadata specification Accuracy Completeness Conformance
Collection Completeness Impact Factor
Catalog Completeness Consistency
Repository Completeness Consistency
Services Composability Efficiency Effectiveness Extensibility Reusability Reliability
![Page 40: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/40.jpg)
Defining Quality in Digital Libraries
Structure of this part of presentation For each quality metric
Discussion about the metric Meaning, use, etc.
Definition of numerical measure Example of Use
![Page 41: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/41.jpg)
Digital Objects: Accessibility A digital object is accessible by an DL
actor or patron, if it exists in the collections of the DL, the repository is able to retrieve the object, and:
1) an overly restrictive rights management property of a metadata specification does not exist for that object; or
2) if it exists, the property does not restrict access to the particular society to which the actor belongs or to that actor in particular.
![Page 42: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/42.jpg)
Digital Objects: Accessibility Accessibility acc(dox, acy) of digital object dox
to actor acy is: 0, if there is no collection C in the DL so that dox C;
otherwise acc(dox, acy) = z struct_streams(dox) rz(ac_y))/|struct_streams(dox)|, where: rz(acy) is a rights management rule defined as an indicator
function: 1, if
z has no access constraints; or z has access constraints and acy cmz, where cmz Soc(1) is
a community that has the right to access z; and 0, otherwise
![Page 43: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/43.jpg)
Digital Objects: Accessibility VT ETD Collection
Fi rst Letter of Author’ s Name
Unrestri cted Restri cted Mi xed Degree of accessi bi l i ty for users not on the VT communi ty
A 164 50 5 mi x(0. 5, 0. 5, 0. 167, 0. 1875, 0. 6)
B 286 102 3 mi x(0. 5, 0. 5, 0. 13)
C 231 108 7 mi x (0. 11, 0. 5, 0. 5, 0. 5, 0. 33, 0. 09, 0. 33)
D 159 54 2 mi x(0. 875, 0. 666)
E 67 26 1 mi x(0. 5)
F 88 39 2 mi x(0. 375, 0. 09)
G 166 72 2 mi x(0. 666, 0. 5)
H 225 91 3 mi x(0. 66, 0. 5, 0. 235)
I 20 8 1 mi x(0. 5)
J 84 36 2 mi x(0. 5, 0. 6)
K 166 69 2 mi x(0. 5, 0. 5)
L 189 68 6 mi x(0. 153, 0. 33, 0. 5, 0. 5, 0. 94)
M 299 115 9 mi x(0. 5, 0. 5, 0. 5, 0. 041, 0. 5, 0. 5, 0. 5, 0. 117, 0. 5)
![Page 44: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/44.jpg)
Digital Objects: AccessibilityN 74 16 1 mix(0.8)
O 45 19 2 mix(0.5, 0.125)
P 172 71 3 mix(0, 0, 0.33)
Q 13 6 0 mix = none
R 158 71 3 mix(0.66, 0.5, 0.5)
S 398 159 8 mix(0.66, 0.5, 0.5, 0.6, 0.33, 0.66, 0.33, 0.6)
T 111 49 1 mix(0.13)
U 9 7 0 mix = none
V 63 20 0 mix = none
W 191 81 5 mix (0.5, 0.22, 0.38, 0.875, 0.5)
X 11 5 0 mix = none
Y 38 9 3 mix(0.5, 0.5, 0.125)
Z 47 17 2 mix(0.5, 0.5)
All 3474 1368 73
![Page 45: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/45.jpg)
Digital Objects: Pertinence
Let Inf(doi) represent the ``information'' (not physical) carried by a digital object or any of its (metadata) descriptions, IN(acj) be the information need of an actor and Contextjk be an amalgam of societal factors which can impact the judgment of pertinence by acj at time k. These include among others, time, place, the
actor's history of interaction, task in hand, and a range of other factors that are not given explicitly but are implicit in the interaction and ambient environment.
![Page 46: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/46.jpg)
Digital Objects: Pertinence
Let's define two sub-communities of actors, users and external-judges Ac, as: users: set of actors with an information need who use
DL services to try to fulfill/satisfy that need external-judges: set of actors responsible for determining
the relevance of a document to a query.
Let's also constrain that a member of external-judges can not judge the relevance of a document to a query representing her own information need, i.e., at the same point in time users external-judges = .
![Page 47: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/47.jpg)
Digital Objects: Pertinence The pertinence of a digital object to a user acj
is an indicator function Pertinence(doi, acj): Inf(doi) IN(acj) Contextjk defined as: 1, if Inf(doi) is judged by acj to be informative
with regards to IN(aci) in context Contextjk;
0, otherwise
![Page 48: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/48.jpg)
Preservability
Fidelity Obsolescence
Depends on Depends on
Process Source format
Target format
Cost
Software Hardware Evaluation StorageIdentification Training …
Digital Objects: Preservability Factors in Preservability
![Page 49: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/49.jpg)
Digital Objects: Preservability Preservability(doi, dl) =
(fidelity of migrating(doi,formatx, formaty),
obsolescence(doi, dl)).
fidelity(doi, formatx, formaty) = 1/ distortion(p(formatx, formaty))
obsolescence(doi, dl) = cost of converting/migrating
object within the context of the specific dl
![Page 50: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/50.jpg)
Digital Objects: Relevance
Relevance (doi,q) 1, if doi is judge by external-judge to be relevant to q0, otherwise
Relevance Estimate Rel(doi,q) = doi
dj / |doi| |q|
Objective, public, social notion Established by a general consensus in the field, not
subjective, private judgment by an actor with an information need
![Page 51: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/51.jpg)
Digital Objects: Similarity
reflect the relatedness between two or more digital objects
Used in many services (e.g., classification, find similar, etc)
![Page 52: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/52.jpg)
Digital Objects: Similarity Metrics
Content-based Cosine(di, dj)
doi dj
/ |doi| |doj
|
Bag-of-words(di,dj) |W(di) W(dj)| / |W(di)|
Okapi(di,dj) (see draft)
![Page 53: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/53.jpg)
Digital Objects: Similarity Metrics
Citation-based Co-citation
cocit(di,dj) = |Pdi Pdj| /max P
Bibliographic coupling bibcoup(di, dj) = |Cdi Cdj|/ max Cd
Amsler Amsler(di, dj) =|(Pdi Cdi) (Pdj Cdj)| / max P
Cd
![Page 54: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/54.jpg)
Digital Objects: SimilarityHighest degree of cocitation Publication Year A unified lattice model for static analysis of programs by construction or approximation of fixpoints
4th ACM SIGACT-SIGPLAN 1977
Active messages: a mechanism for integrated communication and computation
19th annual int. symposium on Computer architecture
1992
Improving direct-mapped cache performance by the addition of a small fully-associative cache and prefetch buffers
17th annual international symposium on Computer Architecture
1990
Computer programming as an art CACM 1974
The SPLASH-2 programs: characterization and methodological considerations
22nd annual international symposium on Computer architecture
1995
ATOM: a system for building customized program analysis tools
ACM SIGPLAN '94 1994
Analysis of pointers and structures Proceedings of the conference on Programming language design and implementation
1990
Revised report on the algorithmic language scheme | ACM SIGPLAN Notices (Issue) 1986
The directory-based cache coherence protocol for the DASH multiprocessor
17th annual international symposium on Computer Architecture
1990
![Page 55: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/55.jpg)
Digital Objects: SimilarityHighest degree of bibliographic coupling publication date
Query evaluation techniques for large databases CSUR 1993
Compiler transformations for high-performance computing CSUR 1994
On randomization in sequential and distributed algorithms CSUR 1994 External memory algorithms and data structures: dealing with massive data CSUR 2001
A schema for interprocedural modification side-effect analysis with pointer aliasing TOPLAS 2001
Complexity and expressive power of logic programming CSUR 2001
Computational geometry: a retrospective ACM symposium on Theory of computing 1994
Research directions in object-oriented database systems ACM SIGACT-SIGMOD-SIGART symposium
Cache coherence in large-scale shared-memory multiprocessors: issues and comparisons CSUR 1993
![Page 56: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/56.jpg)
Digital Objects: Similarity Distributions
Figure 3(a) Figure 3(b)
![Page 57: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/57.jpg)
Digital Objects: Similarity Application: Automatic classification with
kNN
Evidence Macro F1 (30%) Abstract_BagOfWords 0.195 Abstract_Cosine 0.343 Abstract_Okapi 0.339 Bib_Coup 0.347 Amsler 0.412 Co-citation 0.273 Title_BagOfWords 0.492 Title_Cosine 0.525 Title_Okapi 0.525
![Page 58: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/58.jpg)
Digital Object: Timeliness
(current time or time of last freshening) – time of the latest citation, if object is ever cited
age = (current time or time of last freshening) – (creation time or publication time) , if object is never cited
Time of last freshening = time of the creation/publication of most recent object in the collection to which doi belongs
![Page 59: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/59.jpg)
Digital Objects: Timeliness ACM Digital Library
0
1000
2000
3000
4000
5000
6000
7000
8000
Timeliness 0 1 2 3 4 5 6 7 8 9 10
No. of Documents 5165 7264 5162 4209 2716 2120 1698 1554 1372 1357 1019
1 2 3 4 5 6 7 8 9 10 11
![Page 60: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/60.jpg)
Metadata Specifications and Metadata Format: Completeness
Refers to the degree to which values are present in the description, according to a metadata standard. As far as an individual property is concerned, only two situations are possible: either a value is assigned to the property in question, or not.
Metric Completeness(msx) = 1 - (no. of missing attributes in
msx/ total attributes of the schema to which msx
conforms)
![Page 61: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/61.jpg)
Metadata Specifications and Metadata Format: Completeness OCLC NDLTD Union catalog
00. 10. 20. 30. 40. 50. 60. 70. 80. 9
1
GWUD LSU
VTET
D
MIT
UBC
PHYS
NET
VTIN
DIV
VAND
ERBI
LT
NCSU
USAS
K
PITT HKU
HUMB
OLT
OCLC
BGMY
U
DRES
DEN
VIEN
NA
GATE
CH
ETSU USF
MUEN
CHEN
UTEN
N
CCSD
WATE
RLOO
NSYS
U
LAVA
L
UPSA
LLA
CALT
ECH
UCL
WagU
niv
![Page 62: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/62.jpg)
Metadata Specifications and Metadata Format: Conformance
An attribute attxy of a metadata specification msx is
conformant to a metadata format/standard if: it appears at least once, if attxy is marked as
mandatory, and; its value is from the domain defined for attxy; it does not appear more than once, if it is not
marked as repeatable. Metric
Conformance(msx) = ((attribute attxy of msx)
degree of conformance of attxy)/ total attributes).
![Page 63: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/63.jpg)
Metadata Specifications and Metadata Format: Conformance Based on ETD-MS
0. 75
0. 8
0. 85
0. 9
0. 95
1
GW
UD
LSU
VTET
D
MIT
UBC
PHYS
NET
VTIN
DIV
VAN
DER
BILT
NC
SU
USA
SK
PITT HKU
HU
MBO
LT
OC
LC
BGM
YU
DR
ESD
EN
VIEN
NA
GAT
ECH
ETSU
USF
MU
ENC
HEN
UTE
NN
CC
SD
WAT
ERLO
O
NSY
SU
LAVA
L
UPS
ALLA
CAL
TEC
H
UC
L
Wag
Uni
v
![Page 64: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/64.jpg)
Collection, Metadata Catalog, and Repository: Collection Completeness
A complete DL collection is one which contains all the pertinent existing digital objects.
Metric completeness(Cx) = |Cx| /| “ideal collection”|
![Page 65: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/65.jpg)
Collection, Metadata Catalog, and Repository: Collection Completeness
Collection Degree of Completeness
ACM Gui de 1
DBLP 0. 652
CITIDEL(DBLP + ACM + NCSTRL + NDLTD- CS) 0. 467
IEEE- DL 0. 168
ACM-DL 0. 146
ACM Guide
Journal (articles) 256527
Proceeding (papers) 299850
Book (chapters) 107870
Theses 46098
Tech. Reports 25081
Bibliographies 2
Plays 1
735429
![Page 66: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/66.jpg)
Catalog Completeness/Consistency Completeness(DMC)=
1 – (no. of do’s without a metadata specification/size of the described collection)
Consistency(DMC)=
0, if there is at least one set of metadata specifications assigned to more than one digital object
1, otherwise
![Page 67: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/67.jpg)
Repository Completeness and Consistency
Completeness (Rep) = Number of collections in the repository/ideal number of
collections
Consistency(Rep) = 1, if the consistency of all the repositories’ catalogs with
respect to their described collection is 1
0, otherwise
![Page 68: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/68.jpg)
Services: Efficiency/ Effectiveness Effectiveness
Very common measures: Precision, Recall, F1, 10-precision, R-Precision
Other services may have different measures: e.g., Recommending, etc.
Efficiency (duration of a service event): Let t(e) be the time of an event e, eix and efx be the first
and the last event of service sex . The efficiency of service sex is defined as:
Efficiency(sex) = t(efx) - t(eix)
![Page 69: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/69.jpg)
Services: Extensibility and Reusability
A service Y reuses a service X if the behavior of Y incorporates the behavior of X.
A service Y extends a service X if it subsumes the behavior of X and potentially includes additional subflows
of events.
![Page 70: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/70.jpg)
Services: Extensibility and Reusability (2) Metrics
Macro-Reusability(Serv) = ( reused(sei), sei
Serv)/ |Serv|, where reused is a indicator function defined as : 1, if smj, sej reuses si;
0, otherwise. Micro-Reusability(Serv) = ( LOC(smx) *
reused(sei), smx SM, sei Serv, sex runs sei )/
|LOC(sm), sm SM|, where LOC corresponds to the number of lines of code of a service manager
![Page 71: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/71.jpg)
Services: Extensibility and Reusability
Service Component
Based
LOC for implementing
service
LOC reused from
component
Total LOC
Searching – Back-end Yes - 1650 1650
Search Wrapping No 100 - 100
Recommending Yes - 700 700
Recommend Wrapping No 200 - 200
Annotating – Back-end Yes 50 600 600
Annotate Wrapping No 50 - 50
Union Catalog Yes - 680 680
User Interface Service No 1800 - 1600
Browsing No 1390 - 1390
Comparing (objects) No 650 - 650
Marking Items No 550 - 550
Items of Interest No 480 - 480
Recent Searches/Discussions
No 230 - 230
Collections Description No 250 - 250
User Management No 600 - 600
Framework Code No 2000 - 2000
Total 8280 3630 11910
Macro-Reusability = 3/16 = 0.187Micro-Reusability = 3630 / 11910 = 0.304
![Page 72: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/72.jpg)
Services: Reliability Def: 1 – no. of failures/no. of accesses Failure is an event that
was supposed to happen in a scenario but did not;
did happen, but did not execute some of its operations
did happen, where the operations were executed, but the results were not the expected ones.
![Page 73: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/73.jpg)
Services: Reliability CITIDEL (NSDL collection – computing/IT)
CITIDEL service No. of failures/no. of accesses Reliabil ity
searchi ng 73/ 14370 0. 994
browsi ng 4130/ 153369 0. 973
requesti ng (getobj ect) 1569/ 318036 0. 995
structured search 214/752 0. 66
contri buti ng 0/ 980 1
![Page 74: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/74.jpg)
Outline
Major Points of the Presentation Introduction to the 5S View of DLs
Informal Definition of DLs Formal Definition of DLs DL Ontology
Defining a Quality Model for DLs Quality and the Information Life Cycle An XML Log Standard for DLs Conclusions and Future Work
![Page 75: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/75.jpg)
Information Life Cycle
![Page 76: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/76.jpg)
AuthoringModifying
OrganizingIndexing
Storing
Archiving
NetworkingAccessing
Filtering
Creation
DistributionUtilization
Significance
Similarity
Pertinence
AccuracyCompletenessConformance
Seeking
SearchingBrowsingRecommending
Relevance
Timeliness
Accessibility
Accessibility
Believability
Inactive
Active
Discard
RetentionMining
Semi-Active
Preservability
Timeliness
Preservability
Describing
Quality and the Information Life Cycle
![Page 77: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/77.jpg)
Outline
Major Points of the Presentation Introduction to the 5S View of DLs
Informal Definition of DLs Formal Definition of DLs DL Ontology
Defining a Quality Model for DLs Quality and the Information Life Cycle An XML Log Standard for DLs Conclusions and Future Work
![Page 78: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/78.jpg)
XML Log Standard for DLs: Pubs1. Marcos André Gonçalves, Ganesh Panchanathan, Unnikrishnan
Ravindranathan, Aaron Krowne, Edward A. Fox, Filip Jagodzinski, and Lillian Cassel. The XML Log Standard for Digital Libraries: Analysis, Evolution, and Deployment. Proc. JCDL'2003, Third Joint ACM / IEEE-CS Joint Conf. on Digital Libraries, May 27-31, 2003, Houston, 312 - 314
2. Marcos André Gonçalves, Ming Luo, Rao Shen, Mir Farooq Ali, and Edward A. Fox. An XML Log Standard and Tool for Digital Library Logging Analysis. In Proc. Research and Advanced Tech. for Digital Libraries, 6th European Conf., ECDL 2002, Rome, Sep. 16-18, 2002, eds. Maristella Agosti and Constantino Thanos, LNCS 2458, Springer, pp. 129-143.
![Page 79: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/79.jpg)
XML Log Standard for DLs, QualityDL Concept Dimensions of Quality Log can be used to
measure? Digital object Accessibility
Pertinence Preservability Relevance Similarity Significance Timeliness
No Yes No Yes No No No
Metadata specification Accuracy Completeness Conformance
No No No
Collection Completeness Impact Factor
No No
Catalog Completeness Consistency
No No
Repository Completeness Consistency
No No
Services Composability Efficiency Effectiveness Extensibility Reusability Reliability
No Yes Yes No No Yes
![Page 80: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/80.jpg)
Outline
Major Points of the Presentation Introduction to the 5S View of DLs
Informal Definition of DLs Formal Definition of DLs DL Ontology
Defining a Quality Model for DLs Quality and the Information Life Cycle An XML Log Standard for DLs Conclusions and Future Work
![Page 81: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/81.jpg)
Conclusions and Future Work “Study of User Quality Metrics for Metasearch
Retrieval Ranking”: new grant in 2004 IMLS NLG led by Martin Halbert of Emory University
Development of more usage-oriented measures Current measures are mostly system-oriented
Development of Quality ToolKit (5SQual) for DL managers with following features: Mapping tool to map local log format to standard
XML Log format Components to implement all measures Visualization of data and measures Broken into several logical pieces to be used in the
different phases of the information life cycle
![Page 82: Edward A. Fox, Marcos André Gonçalves, Baoping Zhang, Layne T. Watson](https://reader035.fdocuments.us/reader035/viewer/2022062314/568146de550346895db4156c/html5/thumbnails/82.jpg)
Questions/Discussion?