DuraSpace, Fedora and DuraCloud

Triangle Research Libraries Network

September, 2009

DuraSpace, Inc.

• Combined Fedora Commons, Inc. and DSpace Foundation

• 501-(c)3 private, non-profit company• 4-year project funded by Moore Foundation to

become self-sustaining• Continuing software development• Moving towards community-based software

development• Ensuring durability for a web in the clouds!

Scholarly and Scientific Collections

Preservation and Archiving

Education, Knowledge Spaces

The world we work in…

Data Curation, Linking, Publishing

blog and wiki

and more …

DuraSpace Products

• Fedora• Dspace• Akubra – storage plug-in module with

transactional file system• Mulgara – RDF indexing engine• Topaz – core semantic knowledgebase

components• DuraCloud

Solution Communities

• Community group that creates and maintains the vision for software solutions in an area

• Trying to create the conditions for collaboration

• Gather resources to create software for solution

• Coordinates development with DuraSpace technical staff

Solution Areas

• Data Curation• Open Access Publishing• Preservation and Archiving• Small Archives• Scholars’ Workbench

The Flexible Extensible Digital Object Repository Architecture

• A set of abstractions that can be used to represent different kinds of data

• A repository management system• A foundation for many information

management applications• Designed to make data “durable” over the

long term

165 Current Known Users

• Broadcasting and media – 1• Consortia – 9• Corporations – 14• Government agencies – 8• IT- Related Institutions – 10• Medical Centers and Libraries – 4• Museums and Cultural Organizations – 5• National Libraries and Archives – 16• Professional Societies – 2• Publishing - 4• Research Groups and Projects – 18• Semantic and Virtual Library Projects - 6• University Libraries and Archives - 68

Making complex digital information “durable” is a very hard problem

• The existence and meaning of content needs to be verifiable as technologies change

• A history of the changes to the encoding and state of content must be reliably provided

• A meaningful context for any unit of content may be one of many and must be sustained

• Complex resources will increasingly be dispersed across institutional boundaries.

The Fedora abstractions provide a durability framework.

• Content is “unitized” as information objects that combine data, metadata, policies, relationships and the history of the object.

• Complex digital resources are formally defined graphs of related objects.

• The public view of the content is presented as abstract behaviors.

• The web services orientation of Fedora provides the basis for repository federation.

Abstract Data M anage me nt(F e dora CommonsSe rv ice F rame work)

Preservationand Archiv ing

Solutions

Data CurationSolutions

Scholars'Repository

PublishingSolutions

TapeLib rar ie s

SAN

RAID array

DC

Persistent ID

RELS-EXT

AUDIT

1

2

n

Reserved Datastreams

Custom Datastreams

(any type, any number)

A data object is one unit of content

POLICY

Object Properties

“Data or Service Definition or Service Deployment

“A”, “I”, or “D” (Active, Inactive, Deleted)

“Any string”

“Any string”

“2007-04-30T19:59:03.000Z” (UTC, ISO8601 format)

“2007-04-30T19:59:03.000Z” (UTC, ISO8601 format)

“Any string”

System generates value

Either way

Client provides value

LEGEND

“namespace:name

PID

Object Type

State

Label

Content Model

Created Date

Last Modified Date

Owner ID

PID

Datastream Properties

Version

Any XML “NCName” unique within the object

“X”, “M”, “E”, or “R” (Inline XML, Managed,Externally Referenced, or Redirected)

“A”, “I”, or “D” (Active, Inactive, Deleted)

“true” or “false”

1 or more

System generates value

Either way

Client provides value

LEGEND

Datastream

Datastream ID

State

Control Group

Versionable

Policies

• Machine enforceable expressions of rules, what they are applied to and who they affect.

• Who is affected can be defined in different authorization sources, such as LDAP services

• Rules can be as simple as “allow” or “deny”.• Rules are applied to objects as a whole, any

datastream, or a dissemination, as well as each API call and more.

Relationships Among Objects

• Describes adjacency relationships among objects, among units of content

• RDF data of the form:

PID – typeOfRelationship – relatedObjectPID• Can used to assemble aggregations of

objects• Can build graphs of relationships to feed into

user interfaces

Optional Object Behaviors

• Data objects can have different views or transformations

• Sets of abstract behaviors that different kinds of objects can subscribe to

• Corresponding sets of services that specific objects can execute

• The business logic is hidden behind an abstraction

Pid

syste m Me ta

MO D S

JP2 0 00

T hum b S cree n Mas te rC us to m

S izeD ub linC o re MODS C itation

MODSFile

J PEG200File

ContentAccess

ContentManagement

General Im age Object

JPEG2000 Im age Object

Persistent ID(PID)

SystemMetadata

thumbnail image file

med res. image file

high res. image file

max res. image file

Persistent ID(PID)

SystemMetadata

JPEG200im age file

Service Description

Service M echanismfor General Im age Objects

ServiceM echanismfor JPEG2000 Im age Objects

get-thum bnail-sized-im age

get-m ed-sized-im age

get-h igh-res-im age

get-m ax-sized-im age

get-thum bnail-sized-im age

get-m ed-sized-im age

get-h igh-res-im age

get-m ax-sized-im age

get-sm allest-JPEG2000-size

get-m idrange-JPEG2000-size

get-h igh-res-JPEG2000-size

get-m ax-JPEG2000-size

Content Models

• Create classes of data objects• Expressed as Cmodel objects• A Cmodel object defines the number and

types of data streams for objects of that class• A Cmodel object binds to service objects to

enable appropriate behaviors to be inherited by data objects

Persistent ID (PID)

Service DefinitionMetadata

SystemMetadata

DatastreamsCmodel Object

Persistent ID (PID)

Service BindingMetadata (WSDL)

SystemMetadata

Datastreams

WebService

service contract

service

subscriptio

n

data contract

Persistent ID (PID)

RDF data

Datastreams

System Metadata

Service Mechanism Object

Service Definition Object

Persistent ID (PID)

System Metadata

Datastreams

Data Objects

Other components include:- Parameter values used by the method- Datetime stamp for earlier version

A behavior call has the form:

Object PID + SDef Name + Method Name

Objects Representing Aggregations

• Creating parent objects for complex resources

• Representing explicit collections• Representing implicit collections• Creating digital surrogates for physical

entities

Text Collections

Te xts

M o de r n E ng l i s h C o l l e c t i o n

P ag eIm ag e s

TextText Artwork Artwork

W ork W ork

Images

W ork

The Rossetti Archive

RemoteSensingDevice

Im agesDatasets

Proposal

RemoteCamera

FinalReport

A Research Project

Fedora Repository ServiceGSearch

OAI

Ingest

SimpleJMS

Fedora Framework Service Integration

More…repository publishes events

serviceslisten andconsumeevents or other messages

DuraCloudTrusted management of and access to durable digital assets

in the cloud

DuraSpaceMediating

Service

Microsoft

DuraCloud - basics• Replicate to multiple storage providers• Replicate to multiple geographic areas• Monitor and audit digital assets• Compute services in cloud next to content

• Hosted by DuraSpace not-for-profit org• Partnerships with cloud providers• “Pay for use” for services and storage• Available to run internally- open source

Use Cases:DuraCloud with Cloud Storage

• Online backup for text, images, datasets, video, audio

• Enable preservation via multiple copies, geographies, administrations

• Elastic provisioning of temporary or permanent storage for projects or jobs

Use Cases:DuraCloud with Cloud Compute

• Streaming service for video• Hosting JPEG2000 image engine• Indexing and other processing heavy jobs• Repositories in cloud• Data and text mining over open data• Optional aggregation of open data

Partners and Pilots

Selected initial cloud providers

Selected 2 initial pilot partners

Pilot use cases

• Ingest large quantity of material

• Replicate to multiple cloud platforms

• Manage replication and monitoring

• Run services

DuraCloud Timeline

• Initial open source release– summer 2009• Begin pilots – September 2009• Pilot data loading and testing – Fall 2009• Plug-ins for repository platforms – Q4 2009• Beta for repository community - Q1 2010• Pilot testing with compute services Q1 2010• Report pilot results – Q1 2010• Launch production service Q2 2010

http://www.duraspace.org/

http://www.fedora-commons.org/