ResourceSync Overview

66
Herbert Van de Sompel, Martin Klein - ResourceSync CNI Spring 2014, St. Louis, MO, March 31 ResourceSync was funded by the Sloan Foundation & JISC A Modular Framework for Web-Based Resource Synchronization Martin Klein Los Alamos National Laboratory @mart1nkle1n http://www.openarchives.org/rs #resourcesync Herbert Van de Sompel Los Alamos National Laboratory @hvdsomp ResourceSyn c

description

The slides were used to accompany an overview of the outcomes of the ResourceSync project at the 2014 Spring Membership Meeting of the Coalition for Networked Information (CNI). The launch of ResourceSync, a joint project of the National Information Standards Organization (NISO) and the Open Archives Initiative (OAI) funded by the Alfred P. Sloan Foundation, was motivated by the ubiquitous need to synchronize resources for applications in the realm of cultural heritage and research communication. After an initial problem definition and scoping phase, the project has designed, specified, and tested a framework for web-based synchronization that is based on SiteMaps, a protocol widely used by web servers to advertise the resources they make available to search engines for indexing. This choice allows repositories to address both search engine optimization and resource synchronization needs using the same technology. The ResourceSync framework specifies various modular capabilities that a repository can support in order to allow third party systems to remain synchronized with its evolving resources. For example, a Resource List provides an inventory of resources whereas a Change List details resources that were created, deleted or updated during a given temporal interval. Support for capabilities can be combined in order to meet local or community requirements. The framework specifies capabilities that require a third party to recurrently poll for up-to-date information about a repositories’ resources but also publish/subscribe capabilities that keep third parties informed about changes through notifications, thereby significantly reducing synchronization latency.

Transcript of ResourceSync Overview

Page 1: ResourceSync Overview

ResourceSync was funded by the Sloan Foundation & JISC

A Modular Framework for Web-Based Resource Synchronization

Martin KleinLos Alamos National Laboratory

@mart1nkle1n

http://www.openarchives.org/rs #resourcesync

Herbert Van de SompelLos Alamos National Laboratory

@hvdsomp

ResourceSync

Page 2: ResourceSync Overview

Herbert Van de Sompel, Martin Klein - ResourceSyncCNI Spring 2014, St. Louis, MO, March 31 2014

ResourceSync

• Collaboration between NISO and the Open Archives Initiative

• Funded by the Sloan Foundation and JISC

• Goal: Devise a specification for web-based resource synchronization

Page 3: ResourceSync Overview

Herbert Van de Sompel, Martin Klein - ResourceSyncCNI Spring 2014, St. Louis, MO, March 31 2014

This ResourceSync Presentation

• Problem Domain

• Scope

• Framework - Overview

• Framework – Technology

• Demonstration

• Status

Page 4: ResourceSync Overview

Herbert Van de Sompel, Martin Klein - ResourceSyncCNI Spring 2014, St. Louis, MO, March 31 2014

Background - OAI-PMH

• Recurrent metadata exchange from a Data Provider to Service Providers

• XML metadata only

• Repository centric

• Devised 1999-2002, prior to REST, prior to dominance of web search engines

Page 5: ResourceSync Overview

Herbert Van de Sompel, Martin Klein - ResourceSyncCNI Spring 2014, St. Louis, MO, March 31 2014

Revisit the Problem Domain - ResourceSync

• Synchronization of resources from a Source to Destinations

• Web resources, anything with an HTTP URI & representation

• Resource centric

• Devised 2012-2013, leverages key ingredients of web interoperability, existing specifications, existing Search Engine Optimization practice

Page 6: ResourceSync Overview

Herbert Van de Sompel, Martin Klein - ResourceSyncCNI Spring 2014, St. Louis, MO, March 31 2014

Problem Statement

• Consideration:• Source (server) A has resources that change over time: they

get created, modified, deleted• Destination (servers) X, Y, and Z leverage (some)

resources of Source A

• Problem:• Destinations want to keep in step with the resource changes

at Source A

Page 7: ResourceSync Overview

A Source’s Resources

Page 8: ResourceSync Overview

A Source’s Resources Evolve over Time

Page 9: ResourceSync Overview

A Source’s Resources Evolve over Time

Page 10: ResourceSync Overview

A Source’s Resources Evolve over Time

Page 11: ResourceSync Overview

A Source’s Resources Evolve over Time

Page 12: ResourceSync Overview

A Source’s Resources Evolve over Time

Page 13: ResourceSync Overview

A Source’s Resources Evolve over Time

Page 14: ResourceSync Overview

Herbert Van de Sompel, Martin Klein - ResourceSyncCNI Spring 2014, St. Louis, MO, March 31 2014

Problem Statement

• Consideration:• Source (server) A has resources that change over time: they

get created, modified, deleted• Destination (servers) X, Y, and Z leverage (some)

resources of Source A

• Problem:• Destinations want to keep in step with the resource changes

at Source A

• Goal:• Design an approach for resource synchronization aligned

with the Web Architecture that has a fair chance of adoption by different communities

Page 15: ResourceSync Overview

Herbert Van de Sompel, Martin Klein - ResourceSyncCNI Spring 2014, St. Louis, MO, March 31 2014

This ResourceSync Presentation

• Problem Domain

• Scope

• Framework - Overview

• Framework – Technology

• Demonstration

• Status

Page 16: ResourceSync Overview

Herbert Van de Sompel, Martin Klein - ResourceSyncCNI Spring 2014, St. Louis, MO, March 31 2014

Scope – Collection Size

• Size of a Source’s resource collection:

• A few resources - small web sites, repositories

• Millions of resources – large repositories, datasets, linked data collections

Page 17: ResourceSync Overview

Herbert Van de Sompel, Martin Klein - ResourceSyncCNI Spring 2014, St. Louis, MO, March 31 2014

Scope – Change Frequency

• Change frequency of a Source’s resources:

• Low – daily, weekly, monthly

• High – seconds, minutes

Page 18: ResourceSync Overview

Herbert Van de Sompel, Martin Klein - ResourceSyncCNI Spring 2014, St. Louis, MO, March 31 2014

Scope – Synchronization Latency

• Destination’s requirements regarding synchronization latency:

• High latency acceptable

• Low latency essential

Page 19: ResourceSync Overview

Herbert Van de Sompel, Martin Klein - ResourceSyncCNI Spring 2014, St. Louis, MO, March 31 2014

Scope – Collection Coverage

• Destination’s requirements regarding the coverage of a Source’s resources:

• Partial coverage of the Source’s resources acceptable

• Full coverage of the Source’s resources verifiable

Page 20: ResourceSync Overview

Herbert Van de Sompel, Martin Klein - ResourceSyncCNI Spring 2014, St. Louis, MO, March 31 2014

Scope – Bitstream Accuracy

• Destination’s requirements regarding bitstream accuracy:

• Unverifiable bitstream accuracy acceptable

• Verifiable bitstream coverage essential

Page 21: ResourceSync Overview

One to One Synchronization

Page 22: ResourceSync Overview

One to Many – Master Copy

Page 23: ResourceSync Overview

Many to One - Aggregator

Page 24: ResourceSync Overview

Selective Synchronization

Page 25: ResourceSync Overview

Metadata Harvesting

Page 26: ResourceSync Overview

Herbert Van de Sompel, Martin Klein - ResourceSyncCNI Spring 2014, St. Louis, MO, March 31 2014

This ResourceSync Presentation

• Problem Domain

• Scope

• Framework - Overview

• Framework – Technology

• Demonstration

• Status

Page 27: ResourceSync Overview

Herbert Van de Sompel, Martin Klein - ResourceSyncCNI Spring 2014, St. Louis, MO, March 31 2014

A Source’s Resources Evolve over Time

Page 28: ResourceSync Overview

Herbert Van de Sompel, Martin Klein - ResourceSyncCNI Spring 2014, St. Louis, MO, March 31 2014

Solution Perspective - Destination

• Destination needs regarding synchronization:

• Baseline synchronization: Initial catch-up operation to align with the Source’s resources

• Incremental synchronization: Remain synchronized as the Source’s resources evolve

• Audit: Destination determines whether it effectively is in sync with the Source- Bitstream accuracy- Coverage of resources

Page 29: ResourceSync Overview

Herbert Van de Sompel, Martin Klein - ResourceSyncCNI Spring 2014, St. Louis, MO, March 31 2014

Solution Perspective - Source

• Source communicates about the state of its resources:

• Publish inventory: snapshot of the state of resources at a moment in time

• Publish changes: enumeration of resource changes that occurred during a temporal interval

• Notify about changes: send notifications as changes occur

• Communication payload:• Minimal, e.g. HTTP URI of resource• Additional, e.g. content-based hash of resource

Page 30: ResourceSync Overview

Herbert Van de Sompel, Martin Klein - ResourceSyncCNI Spring 2014, St. Louis, MO, March 31 2014

Resource List

• In order to meet a Destination’s need for baseline synchronization, the Source may publish a Resource List

• A Resource List is an inventory, a snapshot of existing resources

• Per resource, it minimally provides the resource’s URI

• Process:- Destination obtains the Resource List- Destination obtains listed resources by their URI- Optimization: Resource Dump, a list pointing to ZIP files

that contain resource representations

Page 31: ResourceSync Overview

Publish Resource List: Inventory at Tx

Resource List @Tx = { A ; B ; C }

Page 32: ResourceSync Overview

Herbert Van de Sompel, Martin Klein - ResourceSyncCNI Spring 2014, St. Louis, MO, March 31 2014

Change List

• In order to meet a Destination’s need for incremental synchronization, the Source may publish a Change List

• A Change List enumerates resource change events that occurred in a temporal interval

• For each event, it minimally lists datetime, URI of the resource, the nature of the change

• Process:- Destination obtains the Change List- Destination obtains created/updated resources, removes

deleted resources- Optimization: Change Dump

Page 33: ResourceSync Overview

Publish Change List: Resource Changes During Interval Ty-Tz

Change List [Ty,Tz] = { A updated @Tc ; B updated @Tc ; C created @Td ; D deleted @Te ; C updated @Tf }

Page 34: ResourceSync Overview

Herbert Van de Sompel, Martin Klein - ResourceSyncCNI Spring 2014, St. Louis, MO, March 31 2014

Change Notification

• In order to meet a Destination’s need for incremental synchronization and low latency, the Source may send Change Notifications

• A Change Notification conveys resource change events as they occur

• For each event, it minimally lists datetime, URI of the resource, the nature of the change

- Process:- Destination receives Change Notification- Destination obtains created/updated resources, removes

deleted resources

Page 35: ResourceSync Overview

Herbert Van de Sompel, Martin Klein - ResourceSyncCNI Spring 2014, St. Louis, MO, March 31 2014

Send Change Notification – Resource Changes at Ta

Change Notification @Ta = { A updated @Ta }

Page 36: ResourceSync Overview

Herbert Van de Sompel, Martin Klein - ResourceSyncCNI Spring 2014, St. Louis, MO, March 31 2014

Send Change Notification – Resource Changes at Tb

Change Notification @Tb = { D updated @Tb }

Page 37: ResourceSync Overview

Herbert Van de Sompel, Martin Klein - ResourceSyncCNI Spring 2014, St. Louis, MO, March 31 2014

Send Change Notification – Resource Changes at Tc

Change Notification @Tc = { A updated @Tc ; B updated @Tc }

Page 38: ResourceSync Overview

Herbert Van de Sompel, Martin Klein - ResourceSyncCNI Spring 2014, St. Louis, MO, March 31 2014

Send Change Notification – Resource Changes at Td

Change Notification @Td = { C created @Td }

Page 39: ResourceSync Overview

Herbert Van de Sompel, Martin Klein - ResourceSyncCNI Spring 2014, St. Louis, MO, March 31 2014

Send Change Notification – Resource Changes at Te

Change Notification @Te = { D deleted @Te }

Page 40: ResourceSync Overview

Herbert Van de Sompel, Martin Klein - ResourceSyncCNI Spring 2014, St. Louis, MO, March 31 2014

Send Change Notification – Resource Changes at Tf

Change Notification @Tf = { C updated @Tf }

Page 41: ResourceSync Overview

Herbert Van de Sompel, Martin Klein - ResourceSyncCNI Spring 2014, St. Louis, MO, March 31 2014

Communication Payload – Metadata & Links

• A Source may provide additional metadata and links pertaining to resources conveyed in Resource Lists, Change Lists, Change Notifications

• Metadata about a resource: content encoding, content length, mime type, content-based hash

• Linking to related resources: mirror copies, alternate representations, resource versions, diff between current and previous version, metadata-to-content link, content-to-metadata link, collection membership, etc.

Page 42: ResourceSync Overview

Herbert Van de Sompel, Martin Klein - ResourceSyncCNI Spring 2014, St. Louis, MO, March 31 2014

Communication Payload – Metadata – Hash

• In order to meet a Destination’s need for audit, the Source may provide a content-based hash pertaining to a resource

• Source computes the content-based hash for a resource

• Source provides the hash as metadata pertaining to the resource in its communication payload

• Destination processes communication payload, obtains the resource

• Destination computes the content-based hash for the obtained resource, compares with the Source’s

Page 43: ResourceSync Overview

Herbert Van de Sompel, Martin Klein - ResourceSyncCNI Spring 2014, St. Louis, MO, March 31 2014

Communication Payload – Link – Interlink Metadata & Content

• In order to allow a Destination to establish the relationship between a Source’s metadata and a Source’s content, the Source may provide appropriate links

• Metadata resources and content resources are just resources identified by HTTP URIs

• Both can independently be subject to synchronization and can be interlinked using appropriately typed links

Page 44: ResourceSync Overview

Herbert Van de Sompel, Martin Klein - ResourceSyncCNI Spring 2014, St. Louis, MO, March 31 2014

Communication Payload – Link – Link to Diff

• In order to minimize content transfer, a Source may link to a diff between the previous and the new version of a resource

• Destination can obtain the diff and patch its (previous) version of the resource

• Connection between the resource and the diff is established by means of appropriately typed link

• Nature of the diff is established by means of MIME type- Few diff MIME types exist. Communities can establish

their own.

Page 45: ResourceSync Overview

Herbert Van de Sompel, Martin Klein - ResourceSyncCNI Spring 2014, St. Louis, MO, March 31 2014

Further Framework Characteristics

• Modular: A Source does not have to implement all capabilities

• Source decides which capabilities to support based on local and community requirements

• Sets of Resources: Division of a Source’s resource collection in logical groupings.

• Supported capabilities can differ per set

• Discovery: Mechanisms for Destinations to determine whether and how a Source supports ResourceSync

• Based on conventions for web discovery and documents that detail the level of support

Page 46: ResourceSync Overview

Herbert Van de Sompel, Martin Klein - ResourceSyncCNI Spring 2014, St. Louis, MO, March 31 2014

This ResourceSync Presentation

• Problem Domain

• Scope

• Framework - Overview

• Framework – Technology

• Demonstration

• Status

Page 47: ResourceSync Overview

Herbert Van de Sompel, Martin Klein - ResourceSyncCNI Spring 2014, St. Louis, MO, March 31 2014

Sitemap Protocol

• ResourceSync builds on the Sitemap protocol used by major search engines

• Similarity between resource synchronization and resource discovery/indexing

• Extends the Sitemap protocol to meet synchronization needs• Cf. Metadata and Links

• Sitemap document format is used throughout the framework to express Resource Lists, Change Lists, etc.• Type of ResourceSync document can be determined

through explicit declaration

Page 48: ResourceSync Overview

Herbert Van de Sompel, Martin Klein - ResourceSyncCNI Spring 2014, St. Louis, MO, March 31 2014

Common Sitemap

<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9”>

<url> <loc>http://example.com/res1</loc> <lastmod>2013-01-02T13:00:00Z</lastmod> </url>

<url> <loc>http://example.com/res2</loc> <lastmod>2013-01-02T14:00:00Z</lastmod> </url> …</urlset>

Page 49: ResourceSync Overview

Herbert Van de Sompel, Martin Klein - ResourceSyncCNI Spring 2014, St. Louis, MO, March 31 2014

Resource List

<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/"> <rs:md capability="resourcelist" at="2013-01-03T09:00:00Z” /> <url> <loc>http://example.com/res1</loc> <lastmod>2013-01-02T13:00:00Z</lastmod> <rs:md hash="md5:1584abdf8ebdc9802ac0c6a7402c03b6" length="8876" type=”application/pdf” /> </url> <url> … </url></urlset>

Page 50: ResourceSync Overview

Herbert Van de Sompel, Martin Klein - ResourceSyncCNI Spring 2014, St. Louis, MO, March 31 2014

Change List

<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/"> <rs:md capability=”changelist" from="2013-01-02T09:00:00Z” until="2013-01-03T09:00:00Z” /> <url> <loc>http://example.com/res2</loc> <lastmod>2013-01-02T13:00:00Z</lastmod> <rs:md change=”updated” /> <rs:ln href=“http://example.com/res2/meta” rel=“describedby” /> </url> <url> … </url></urlset>

Page 51: ResourceSync Overview

Herbert Van de Sompel, Martin Klein - ResourceSyncCNI Spring 2014, St. Louis, MO, March 31 2014

Change Notification

<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/"><url> <loc>http://example.com/res3</loc> <lastmod>2013-01-02T13:00:00Z</lastmod> <rs:md change=”updated” type=”application/json” /> <rs:ln href=“http://example.com/res3/diff” rel=“http://www.openarchives.org/rs/terms/patch” type=“application/json-patch”/> </url> <url> … </url></urlset>

Page 52: ResourceSync Overview

Herbert Van de Sompel, Martin Klein - ResourceSyncCNI Spring 2014, St. Louis, MO, March 31 2014

PubSubHubbub Protocol

• ResourceSync builds on the PuSH protocol used for syndication of Atom/RSS feeds

• Introduces a novel use for pushing change notifications from a Source to subscribing Destinations

• Destinations subscribe to a Source’s change notifications via a Hub

• The Source pushes change notifications out to the Hub

• The Hub relays change notifications from the Source to the subscribing Destinations

Page 53: ResourceSync Overview

Destination Discovers Source’s Notification Channel & Hub

Page 54: ResourceSync Overview

Destination Subscribes to Source’s Notification Channel

Page 55: ResourceSync Overview

Source Sends Change Notification to Hub

Page 56: ResourceSync Overview

Hub Relays Change Notification to Destination(s)

Page 57: ResourceSync Overview

Herbert Van de Sompel, Martin Klein - ResourceSyncCNI Spring 2014, St. Louis, MO, March 31 2014

This ResourceSync Presentation

• Problem Domain

• Scope

• Example Use Cases

• Framework Characteristics

• Demonstration

• Status

Page 58: ResourceSync Overview

Source Sends Change Notifications

Screencam of demo at https://www.youtube.com/watch?v=H2Le9_Bbkdw

Page 59: ResourceSync Overview

Destination Acts Upon Change Notifications

Screencam of demo at https://www.youtube.com/watch?v=H2Le9_Bbkdw

Page 60: ResourceSync Overview

Observe the State of Source and Destination

Screencam of demo at https://www.youtube.com/watch?v=H2Le9_Bbkdw

Page 61: ResourceSync Overview

Herbert Van de Sompel, Martin Klein - ResourceSyncCNI Spring 2014, St. Louis, MO, March 31 2014

This ResourceSync Presentation

• Problem Domain

• Scope

• Example Use Cases

• Framework Characteristics

• Demonstration

• Status

Page 62: ResourceSync Overview

Herbert Van de Sompel, Martin Klein - ResourceSyncCNI Spring 2014, St. Louis, MO, March 31 2014

Status of Specifications

• ResourceSync Core Specification to be ANSI/NISO Z39.99-2014 by Summer 2014

• Spec on OAI web site fully aligned with NISO Standard

• ResourceSync Notification Specification currently in beta• Stable• Testing PubSubHubbub• Hub, Source, Destination Python software to be released • Public Hub for testing will be made available

• ResourceSync Archive Specification currently in beta• Stable

Page 63: ResourceSync Overview

Herbert Van de Sompel, Martin Klein - ResourceSyncCNI Spring 2014, St. Louis, MO, March 31 2014

Pointers

• Specificationhttp://www.openarchives.org/rs/http://www.openarchives.org/rs/resourcesynchttp://www.openarchives.org/rs/notificationhttp://www.openarchives.org/rs/archives

• List for public commenthttps://groups.google.com/d/forum/resourcesync

• Building blockshttp://sitemaps.orghttps://code.google.com/p/pubsubhubbub/

Page 64: ResourceSync Overview

Herbert Van de Sompel, Martin Klein - ResourceSyncCNI Spring 2014, St. Louis, MO, March 31 2014

Papers

• Klein, M., and Van de Sompel, H. (2013) Extending Sitemaps for Resourcesync. http://arxiv.org/abs/1305.4890 ACM/IEEE JCDL 2013

• Haslhofer, B., Warner, S, Lagoze, C., Klein, M., Sanderson, R., Nelson, M.L. and Van de Sompel, H. (2013) ResourceSync: Leveraging Sitemaps for Resource Synchronization. http://arxiv.org/abs/1305.1476 WWW 2013 Developer Track

• Klein, M., Sanderson, R., Van de Sompel, H., Warner, S, Haslhofer, B., Lagoze, C., and Nelson, M.L. (2013) A Technical Framework for Resource Synchronization. http://dx.doi.org/10.1045/january2013-klein D-Lib Magazine.

• Van de Sompel, H., Sanderson, R., Klein, M., Nelson, M.L., Haslhofer, B., Warner, S, and Lagoze, C. (2012) A Perspective on Resource Synchronization. http://dx.doi.org/10.1045/september2012-vandesompel D-Lib Magazine.

Page 65: ResourceSync Overview

Herbert Van de Sompel, Martin Klein - ResourceSyncCNI Spring 2014, St. Louis, MO, March 31 2014

Acknowledgments

• Editors: Martin Klein, Robert Sanderson, Herbert Van de Sompel (Los Alamos National Laboratory), Simeon Warner (Cornell University), Graham Klyne ( University of Oxford), Bernhard Haslhofer (University of Vienna), Michael L. Nelson (Old Dominion University), Carl Lagoze ( University of Michigan)

• Contributors: Peter Murray (Lyrasis), Todd Carpenter, Nettie Lagace (NISO), Richard Jones (Cottage Labs), Stuart Lewis (University of Edinburgh), Paul Walk (University of Edinburgh), Jeff Young (OCLC), Shlomo Sanders (Ex Libris), Kevin Ford (Library of Congress)

• Demonstration: Martin Klein, Harihar Shankar, Herbert Van de Sompel (Los Alamos National Laboratory)

Page 66: ResourceSync Overview

ResourceSync was funded by the Sloan Foundation & JISC

A Modular Framework for Web-Based Resource Synchronization

Martin KleinLos Alamos National Laboratory

@mart1nkle1n

http://www.openarchives.org/rs #resourcesync

Herbert Van de SompelLos Alamos National Laboratory

@hvdsomp

ResourceSync