Long-term preservation aspects in the eSciDoc project

Natasa Bulatovic

Max-Planck Digital Library

bulatovic@mpdl.mpg.de

eSciDoc: Background information

Joint Project between Max Planck Society and FIZ Karlsruhe

Funded by German Federal Ministry for Education and Research (BMBF) until Mid 2009 Additional substantial own efforts by both partners Both Partners committed themselves to ongoing efforts until 2011

eSciDoc project landscape

Diversity of MPG supported by a service infrastructure

21.04.23

● People

● Librarians, researchers, developers, general public

● Data

● Publications, research data, supplementary material etc.

● Processes

● Various institutes apply various workflows supporting their data management

eSciDoc: what it addresses and provides?

Organizational aspects Business processes: introspection into the “research information and

research data life-cycle” Modelling of the “research workflows” (usage, scenarios, use cases, process

worfklows) “SOA” is not only technical infrastructure - it is driven by organization-wide

processes and requirements

Technical aspects service infrastructure instead of silos applications solutions to visualize, publish, relate and manage data easy composition of services and data mashups and repurposing

Both organizational and technical activities facilitate the growth of the organization and the dissemination, accessibility and easy reuse of the research results across disciplines

eSciDoc.PubMan: Management of publications

eSciDoc.Virr: Management of digitized textual resources

eSciDoc.FACES: Management of image collections

Challenges

And facts for eSciDoc project related to the digital preservation and long-term archiving

Which data are managed?

Support for variety of data (publications, old manuscripts, microscopic images, patents, datasets)

Abstraction: Data structure

Abstraction in data modelling and implemented data services (items, containers).xml

Specialization through content models gives contextual information on what data it is (publication, scanned book page, lexical resource, collection of “digital things”)

http://colab.mpdl.mpg.de/mediawiki/ESciDoc_Logical_Data_Model

Atomistic model

Atomistic model - items

21.04.23 Seite 12

Atomistic model - containers

21.04.23 Seite 13

Communities and workflows

Core workflow

Support for variety of workflows for different user-communities

Community specific workflow

Persistent identification

Each resource and resource component (metadata, files) must be persistently identified

Users may decide what to identify (resource, resource version)

Users may configure when to assign the persistent identifier on a repository level

Keep all persistent identifiers of the resource (if such existed before the resource has been created in the system)

Different communities decide what to identify and when in a different manner and at a different state of the completeness of their data

Persistent identification system

But which PID system to choose out of many ? Handle System (CNRI - Corporation for National Research Initiatives) Digital Object Identifiers - DOI (IDF – International DOI Foundation) Archival Resource Key –ARK (California Digital Library) the Uniform Resource Name - URN (IANA) National Bibliography Numbers - NBNs the persistent URL – PURL (OCLC, Online Computer Library Center) the Open URL

Most important is the organizational commitment for PID and PID infrastructure, PIDs are rather organizational then technical effort

EPIC – European Persistent Identifier Consortium (GWDG Germany, SARA Netherlands, CSC Finland, http://www.pidconsortium.eu/ )

Resource metadata

Standards and Frameworks – have to be used so data are known Dublin Core, MODS, other Where no standards exist, describe the profile in standard format Dublin Core Singapore Framework

Services, metadata quality and metadata extraction Metadata Validation service CoNE – Control of Named Entities service JHove – technical metadata extraction service Supports and important aspect: quality of data, clear identification of resources,

format migration

Interoperability – data redundancy Ensures resources are disseminated via RSS, OAI-PMH

Support for variety of data (publications, old manuscripts, microscopic images, patents, datasets) and their metadata

Metadata standards and frameworks

“The Singapore Framework for Dublin Core Application Profiles is a framework for designing metadata applications for maximum interoperability and for documenting such applications for maximum reusability”

http://dublincore.org/architecturewiki/SingaporeFramework/http://dublincore.org/documents/2007/06/04/abstract-model/

Metadata profiles

● DCAP based (Dublin Core Application Profile)

● DC terms (identified URIs)

● eSciDoc solution specific terms (identified by URIs)

● METS/MODS

● Publicly available

● Functional description http://colab.mpdl.mpg.de/mediawiki/ESciDoc_Application_Profiles

● Application profiles schemas (xsd) http://metadata.mpdl.mpg.de/escidoc/metadata/schemas/0.1/

● Vocabulary encoding schemas http://colab.mpdl.mpg.de/mediawiki/ESciDoc_Encoding_Schemes

● Interoperability levels

● Shared term definitions (done)

● Semantic interoperability (done)

● Description set syntactic interoperability (started)

● Description set profile interoperability (started)

Services

• CoNE – Control of named entities• Journals, Persons, Dewey Decimal Classification (3 public levels), Creative

Commons Licenses (CC licenses), ISO 639-3 Languages, MIME Types, PACS classification, Custom classifications

• “Vocabulary Encoding schemas for resources”

• Validation service• Semantic metadata validation based on Shibboleth

• If the publication is of type “Journal Article” then the “Journal” information has to be present

• Rules are preserved in XML – and are not encoded in the business logic code

• Jhove service• Technical metadata are extracted prior to binary content upload

• Technical metadata are kept together with the binary content

• Example• http://faces.mpdl.mpg.de/details/escidoc:47276

• http://r-coreservice.mpdl.mpg.de/ir/item/escidoc:47276

Data provenance

Support for event logs and version history (what happened with the content?)

PREMIS v1 created for each resource (events info)

Example: http://pubman.mpdl.mpg.de/pubman/item/escidoc:66603

Upcoming: upgrade to PREMIS v2

Software, standards, documentation

Software all used software is an open source software eSciDoc software available under CDDL license (open source)

Documentation Accessible (still a lot of work to do) Specifications and some design are available via MPDL Colab e.g.

http://colab.mpdl.mpg.de/mediawiki/PubMan_About

Standards and formats XML, XSD used for interface and service operation messages Use of metadata standards PREMIS Open formats wherever possible e.g. metadata, lexical resources TEI-XML etc. Containment: Fedora XML objects for content resources OAIS reference model – compatible

Bitstream preservation

OAIS Archive: What we will put for LTA?

•XML•XML/RDF•XSDs•Open source code•Binary content with technical metadata (XML) and calculated MD5 checksums

Lessons learned

Have we covered all aspects? They must be considered from the very start in infrastructural design and

implementation Things are getting more complicated in dynamic environment (data are changed

often)

Selection process What – all or only selected research data? When – whenever data is created or after the finalization of curration activities? Why - “institutional memory”, future research?

Data redundancy As many copies as possible Share and replicate as much as possible - redundancy does not hurt (it costs )

Organizational commitment Sustainable Long-term Technical Collaboration and cooperation with partners necessary (its not a single unit

effort)

Thank you!

www.escidoc-project.de http://colab.mpdl.mpg.de

bulatovic@mpdl.mpg.de

Resources eSciDoc project web pages, Infrastructure download http://www.escidoc-project.de MPDL collaboratory network http://colab.mpdl.mpg.de MPDL software download

http://escidoc1.escidoc.mpg.de/projects/common_services/

http://escidoc1.escidoc.mpg.de/projects/pubman/ http://escidoc1.escidoc.mpg.de/projects/faces/ http://escidoc1.escidoc.mpg.de/projects/virr/ http://colab.mpdl.mpg.de/mediawiki/ESciDoc_Admin