PDS4 Project Report

PDS MC F2FColumbia, MD

Dan CrichtonApril 2-3, 2013

1

PDS4: The Next Generation PDS

• PDS4 is a PDS-wide project to upgrade from PDS version 3 to version 4 to address many of these challenges

• An explicit information architecture

• All PDS data tied to a common model to improve validation and discovery

• Use of XML, a well-supported international standard, for data product labeling, validation, and searching.

• A hierarchy of data dictionaries built to the ISO 11179 standard, designed to increase flexibility, enable complex searches, and make it easier to share data internationally.

• An explicit software/technical architecture

• Distributed services both within PDS and at international partners

• Consistent protocols for access to the data and services

• Deployment of an open source registry infrastructure to track and manage every product in PDS

• A distributed search infrastructure

2

Information Architecture Concepts

3

Tagged Data Object (Information Object)

Label Schema

Used to Create

Describes

Extracted/Specialized

InformationModel

Data Object

Data Element

Class

has

Planetary ScienceData Dictionary

Expressed As

Product

Validates

<Array_2D_Image>    <local_identifier>MPFL-M-IMP_IMG_GRAYSCALE…    <offset unit="byte">0</offset>    <axes>2</axes>    <axis_index_order>Last Index Fastest…    <Element_Array>        <data_type>UnsignedMSB2</data_type>        <unit>data number</unit>    </Element_Array>               <Axis_Array>        <axis_name>Line</axis_name>        <elements>248</elements>        <unit>not applicable</unit>        <sequence_number>1</sequence_number>    </Axis_Array>    <Axis_Array>        <axis_name>Sample</axis_name>        <elements>256</elements>        <unit>not applicable</unit>        <sequence_number>2</sequence_number>    </Axis_Array></Array_2D_Image>

System Design Approach

• Based on a distributed information services architecture (aka SOA-style)

• Allow for common and node specific network-based (e.g., REST) services.

• Allow for integrating with other systems through IPDA standards.

• System includes services, tools and applications

• Use of online registries across the PDS to track and share information about PDS holdings

• Implement distributed services that bring PDS forward into the online era of running a national data system

• With good data standards, they become critical to ultimately improving the usability of PDS

• Support on-demand transformation to/from PDS

4

Client BClient A

Service

CService Interface

Summary of Progress to Date

• Requirements in place (approved by MC 3/2010)• PDS-wide Architecture defined • Major reviews conducted• Initial PDS products defined using maturing PDS4 specification

• LADEE, MAVEN teams developing labels

• System builds grouped by purpose: build 1(a,b,c,d), 2 (a,b,c), 3(a) • Used to establish a rigorous release process• Includes deployment of software, data standards, documents and system integration testing• As stability has increased, we have moved to 6 month builds

• Operational capabilities deployed• Registry and Harvest infrastructure in place at EN• Central catalog migrated to registry; High level search migrated• Information model released for limited use to select data providers• PDS3 data now being ingested into PDS4 system at EN

• Beta test conducted by Atmospheres to get user input on PDS4 concepts (XML, Bundles, etc)• LADEE PDS4 Peer Reviews conducted• IPDA endorsement and plans to move to PDS4

• International implementation by PSA for Bepi Colombo underway

• International access prototype demonstrated at IPDA 2012 SC

5

PDS4 Software Development Lifecycle

6

NASA Software Management Guidebook – NASA-GB-001

Evaluation by users as early as possible!

Project Lifecycle – mapped to NPR 7120.8 per SDR1

7

Project Lifecycle

Pre-Formulation

Formulation Implementation

Project LifecycleGates &MajorEvents

ProjectReviews

PDSMCConceptReview(Dec 2007)

Study/Concepts

Project Plan PDS4 PrelimArchitecture

PDSMCImplReview(July 2008)

PDSMCPreliminaryDesign(August 2009)

PDSMCArchReview(Nov 2008)

Build 1b Internal Stds Assessment(Dec 2010)

PDS SystemReview(Mar 2010)

Build 1c IPDA Stds Assessment (April 2011)

PDS System Review II(June 2011)

Build 1d External Stds Assessment(Aug 2011)

Operational Readiness Review (LADEE/MAVEN)(November 2011)

PDS4 DesignBeginStudyProject

Build 1: Prototype build

Build 2: Release to LADEE/MAVEN

Build 3: V1.0 of PDS4 Stsandards

1a 1b 1c 1d

KDP: Study KDP: Prelim Design KDP: ProjectPlan & Arch

KDP: ReleaseLADEE/MAVEN

PhoenixBeta Test(Dec 2012)

PDS4: Build Structure

88

Phase Purpose Release Date

Build 1Prototype

• Early formalization of the data standards• Early formalization of software• Integration between software and data standards• Stakeholder input

• PDS4 beta info model, standards reference, data dictionary, schemas baseline•Early releases of Harvest, Registry and Security services•First set of process, documentation and tutorial

October2010

Build 2Early Release to Select Data Providers

•Support LADEE, MAVEN and Early Migration Planning• Begin deployment of PDS4 at Engineering Node and transition• Stakeholder input, particularly from LADEE and MAVEN

• PDS4 beta info model, standards reference, data dictionary, schemas baseline• Production release of Harvest, Registry, Report and Security services• Validation and catalog ingest tools• Updated documentation

March2012(build 2b)

Build 3V1.0 of PDS Data Standards

•Transition entire PDS3 catalog to PDS4 •Baseline PDS4 standards as version 1.0• Support validation of PDS4 bundles• Support ingestion and PDS4 data into the PDS4 registry• Support search and access to PDS3 data sets and PDS4 bundles• Ensure LADEE/MAVEN can move on

• Stable release of data standards• Incremental releases of validation and PD3 catalog ingest tools•Incremental releases of harvest, registry, report and security services•Deployment of the PDS4 search service at EN

October (build 3a)March 2013 (build 3b)

Build 4User Services

• Support PDS4 data distribution services for LADEE, MAVAEN • Support PDS4 data transformation• PDS4 user tools

• Incremental release of PDS4 Standards• Distribution services for initial PDS4 bundles• Transformation services for initial PDS4 bundles

September 2013/March2014

Build 2 ORR

• Begin PDS4 label design for LADEE and MAVEN; Begin planning/testing migration

• Deploy a PDS4 standard that supports the Policy on “Acceptable PDS4 Data Formats”

• Transition the central catalog to the registry infrastructure

• Deploy early PDS4 software tools and services 9

PDS4 Planned Mission Support

10

BepiColumbo (ESA/JAXA)

Osiris-REx (NASA)MAVEN (NASA)

LADEE (NASA)

InSight (NASA)

ISRO, JAXAplanning PDS4missions

PDS3 Implementation

11

PDS3Pipeline

PDS3Ingest

Current Missions

PDS3 Archive @ DNs

PDS3Services

PDS3 CentralCatalog

UsersPDSPortal

Datasets +Products

PDSCatalog Info

PDS3Metadata

Index

Today

12

PDS3Pipeline

PDS3Ingest

Current Missions

PDS3 Archive @ DNs

PDS3Services

PDS3 CentralCatalog

Users

PDS4Registry

Updated PDS

Portal

toPDS4 Transform

Datasets +Products

PDS4 XMLLabel

PDSCatalog Info

PDS4 Metadata

Index

(1) PDS4 infrastructure deployed at EN; Central catalog migrated.

PDSCatalog Info

PDS3Metadata

Index

Support for LADEE/MAVEN

13

PDS3Pipeline

PDS3Ingest

Current Missions

PDS3 Archive @ DNs

PDS3Services

PDS3 CentralCatalog

Users

PDS4Ingest

PDS4Registry

PDS4 Archive @ DNs

PDS4Pipeline

PDS4Services

New Missions (LADEE, MAVEN, O-Rex, Insight)

Updated PDS

Portal

toPDS4 Transform

Datasets +Products

PDS4 XMLLabel

PDSCatalog Info

PDS4 Metadata

Index

NOTE: PDS3 Services phased out overtime

(1) PDS4 infrastructure deployed at EN; Central catalog migrated.(2) Working towards acceptance of new PDS4 mission data

PDSCatalog Info

PDS3Metadata

Index

Build 4

PDS3 Data Migration

Requirements & Domain Knowledge

PDS4 Information Model

Query Models

Information Model

Specification

XML Schema

(pds)

Filter and Translator

Protégé Ontology Modeling

Tool

PDS4 Data Dictionary

(Doc and DB)

PDS4 Data Dictionary

(Doc and DB)

XML Document

(Label Template)

XMI/UML

Registry Configuration Parameters

PDS4 Data Dictionary (ISO/IEC

11179)

The Information Model Driven

Process & Artifacts

This is given to data providers

Capabilities/Operational Deployment

• Design and Generate PDS4 Products• Information Model 0310b frozen (build 3b)• COTS tools tested and in use to support PDS4 Schema development (internal to PDS) (build 2)• DN tools for generating PDS4 products developed at multiple nodes (build 2)

• Validation• Validate tool developed for PDS4 label validation (build 2)

• Harvest/Registration• PDS3 central catalog migrated to registry (build 3a)• CI tool in place to register PDS3 catalog data (done, e.g., MSL) in a PDS4 registry (build 3a)• Harvest in place to register PDS4 bundles and resources in PDS4 registry (build 3a)• Registration of PDS4 web resources (build 3a)• Registration of PDS4 bundles (build 3b)

• Search/Access• Deployment of the PDS4 search service at EN (build 3a)• Generation of a PDS4 search index for PDS3 data (build 3a)• High level search of PDS3 data sets, PDS4 bundles, web resources, IPDA (build 3b)

• Distribution/user tools (build 4)• This the need area for future builds, but is dependent on having good bundles …• We need to take advantage of the flexibility of XML in presenting PDS4 to end users

15

Build 3b Deliverables*

16

• Software System

• Registry Service• Harvest Tool• Validate Tool • Security Service• Report Service• Search Service • CI Tool • Upgraded portal search

and page views to support PDS4

• Data Standards*

• Information Model• XML Schemas• Data Dictionary• Concepts Document• Standards Reference• Data Providers Handbook• PDS4 Example Products

* Posted to http://pds.nasa.gov/pds4

Build 3b Screenshot

17

http://pds-engineering.jpl.nasa.gov/index.cfm?pid=145&cid=187

Build 3b Finalization

• March 2013• Freeze the model• Synchronize the model and software

• 29 March 2013• Capture in CM; Deploy IM, documentation, software, test plans,

etc to PDS4 build 3b site

• April 2013• EN regression testing to verify software against PDS4 examples• Compile test report • Pending no major issue, deploy Standards as V1.0 to support

LADEE, MAVEN SIS finalization and IPDA SC

18

Build 3b Testing Process

• Engineering has developed a test plan (Emily Law will lead)• Test cases• Data product matrix

• Engineering performs regression tests using example bundles and data products• Validate functional requirements L3 – L5

• Engineering will write a test report

19

Beta Testing

• PDS4 development has performed increasing levels of beta testing with users since build 1

• Latest beta test by Atmospheres included review of PDS4 labels, bundles and documentation

20Search Service Deployed for Build 3a Includes links to Archive Resources

PDS4 Archive Support Pages

• Cassini Archive Support page has been received well

• In PDS4, we are developing archive pages (e.g. Phoenix) for our migrated data

• We can add tools/services over time - this gets the data online

• Those pages show up at the top of a search now with the new search service from build 3a

21

IPDA Interactions

• IPDA endorsement of PDS4 (July 2012)

• PSA moving forward with PDS4 • Dave Heather leading a IPDA PDS4

implementation project• Will be at JPL in May 2013 for

coordination meetings for Bepi Columbo

• Search/access to international archives• Demonstrated as a prototype at IPDA SC

and COSPAR in July 2012• Leverages the PDS4 REST-based

architecture• Will be operational with build 3b

deployment

22

7th IPDA Steering Committee (July 2012)(ASI, CNES, ESA, ISRO, JAXA, NASA)

Architecture

July 13, 2012 Cross-Agency Search and Access 23

Registered Objects:•Websites•Data Sets•Investigations, Instruments, etc.•Tools and Services

Search results include mission support pages and other more specific search interfaces.

Search Service supports the PDS and PDAP protocols enabling development of other portals and applications on this infrastructure.

IPDA Search Results

24

Data sets and other results returned include products from NASA and PSA.

PSA data sets currently link directly to a PSA web-based interface.

Summary

• Significant progress has been made and operational capabilities have been deployed since build 2b

• We need stable data standards (v1.0) so we can move forward with LADEE, MAVEN and others, and focus on data distribution services.

• We need to be producing bundles• That will push us forward

• We have international alignment!• We should good progress at the July 2013 meeting

25

Backup

26