Lisa Gaddis (USGS) and Sue LaVoie (JPL)

PDSMC April 3, 2008 1

PDS Imaging Node StatusPDS Imaging Node Status

Mission Interface, Archive and Delivery Services

PDSMC April 3 & 4, 2008Washington, DC

Lisa Gaddis (USGS) and Sue LaVoie (JPL)


Mission Interface• Data Nodes: THEMIS (ASU), HiRISE (UA), LROC (ASU)• Cassini (ISS, VIMS, RADAR)• Mars Exploration Rovers (Cameras – Descent, Haz, MicroImager, Nav, Pan)• Mars Global Surveyor (MOC)• Mars Reconnaissance Orbiter (HiRISE, CTX, MARCI, ONC)• Messenger (MDIS)• Mars Express (HRSC)• Phoenix (MECA-OM, MECA-AFM, RAT, SSI, ‘MARDI’)• Chandrayaan-1 Moon Mineralogy Mapper (M3)• Lunar Reconnaissance Orbiter (LROC, LAMP)• LCROSS• Mars Science Laboratory (MASTCAM, MAHLI, MARDI)

Systems: Online Systems & Support Software• Online access to planetary image data (~9 TB JPL, ~7 TB USGS, ~5 TB ASU, ~18 TB UA = ~39 TB)

• Atlas II, Map-a-Planet, Photojournal (~2 TB/mo of MER, Cassini, MRO)• Deliver improved ancillary data (pointing, position, geometry)

• Unified Planetary Coordinates (UPC) Database

Baseline Services• Provide expert assistance to PDS and users for cartographic and scientific data analysis• Restorations include Lunar Orbiter, Apollo cameras

PDS ‘Community Service’• Serve as PDS representative on NASA ‘cartography’ working groups

Overview: Imaging Node (IMG) ActivitiesOverview: Imaging Node (IMG) Activities (FY08+)


– IMG Web site• Enhanced automation for faster & easier updates• Continued sprucing per advice from Mark Rose (Ames) and users• Mission-specific interface points to online image data and other Node

collections– Map-a-Planet

• Web Map Services (WMS) interface delivers derived PDS products– New: (Moon) Clementine NIR, Lunar Orbiter, Lunar Prospector; (Mars) MGS

TES albedo and thermal inertia; (Jupiter/Io) Galileo SSI color– Atlas II (aka ‘New’ Planetary Image Atlas)

• Includes image data from all past and current planetary missions– Updated monthly with Cassini, THEMIS, HiRISE, MER, etc.

• Single- and multi-mission searches and bulk download• Map-based searches now being implemented (Google Toolkit)

– Will use improved geometry data from UPC– Unified Planetary Coordinates (UPC) Database

• Delivers image metadata with unified coordinates and improved geometry • Geometry ‘backbone’ for Atlas II, ODE (GEO), ISIS, (ILIADS?) etc.

Data Delivery ServicesData Delivery Services


Map-A-PlanetMap-A-Planet• Capabilities and Features:

– Delivers derived PDS map products• Viking Orbiter Mars Digital Image Mosaic (MDIM)

v. 2.1• MGS/MOC WA (256 p/deg), MOLA (128 p/deg)• Clementine UVVIS and NIR mosaics (11 bands)

– Uses WMS to make maps of spatially referenced data from geographic info in labels

– Multiple output file formats• GIS ‘world file’ and common image formats• ISIS detached labels as input and output• Multiple image projections

– Simple cylindrical– Mercator– Sinusoidal– Polar Stereographic (implementation in progress)

• 32-bit output formats (new!)– MGS/TES albedo and thermal inertia data

– Products linked to NASA Nomenclature database and USGS planetary maps

OGC WMS – Open Geospatial Consortium Web Map Service(s)

See http://mapaplanet.org


• Add option for full image download• Add JPEG2000 format for output files

– Uses OGC Geospatial Data Abstraction Library (GDAL) for data translation from PDS

– Smaller file sizes for large image products and mosaics• HiRISE and LROC

• Output ISIS formatted files from PDS input– Create ISIS detached label to augment PDS label for products

• Part of USGS ‘integrated Web services’ plan to simplify access to PDS products and allow ‘on-the-fly’ processing

– Will directly link PDS and ISIS products to a variety of desired output formats– Uses OGC-compliant s/w and data formats to ensure interoperability

• New data added (or soon to be added):– Mars MOLA (128 pixels/deg) global mosaic (highest available resolution)– Mars MOC (256 pix/deg) WA global mosaic– Mars TES global derived data (albedo, thermal inertia @ 8 pix/deg)– Lunar Prospector derived Fe, Ti, Th, H, etc. (5 and/or 0.5 deg)– Clementine NIR 6-band global mosaic– Lunar Orbiter photographic global mosaic (512 pix/deg)– Merged Galileo SSI and Voyager color mosaic for Io

MAP: In DevelopmentMAP: In Development

See http://mapaplanet.org


Atlas IIAtlas II• Atlas II has 4 distinct layers and functions

– Data Access via Web Service (uses OODT)• Performs searches across all data sets in the system• Handles semantics translation (e.g., East or West longitude, SCET vs

FDS) then searches metadata from repository’s OODT Profile Server or SQL database

– User Interface via ‘Forms’• The human-interface part that allows tailored requests

– Processing Capability• Uses JPEG2000 software and server

– Facilitates HiRISE and LROC deliveries– Retrieval (uses OODT)

• Data download via OODT Product server or ‘wget’ to Web Service• Shopping-cart style bulk data ordering• Allows geographic data searches

OODT - Object Oriented Data Technology See: http://pdsimg.jpl.nasa.gov/Missions/index.html


Status:• New missions supported

– Cassini, MER, MGS, Messenger, MRO (HiRISE, CTX, and MARCI) now operational• Support for both mission-specific and multi-mission queries

– The same code base is used for both– Common parameters of each logical grouping (data set) are dynamically found and

displayed and searched across missions

Plans:• System modifications

– Migrate meta-data from Sybase to MySQL to increase performance– Schema optimization– Configuration file to integrate with Atlas II and OODT– Add data processing capability (calibration, mosaicking, geometric processing, etc.)– Map-based searches– Integration with UPC

• Data migration from Atlas I web-interface to Atlas II– FY2008: Voyager, Viking Orbiter, MPF, Clementine, Magellan

Atlas II Status and PlansAtlas II Status and Plans

See: http://pdsimg.jpl.nasa.gov/Missions/index.html


• A geometry database system to unify coordinate systems for planetary image data• Designed in part to resolve a problem with coordinate system disparities in older planetary image data (esp. for Mars)

• Provides significantly improved image positional and geometric information– Calculates geometric parameters incrementally for each image– Uses 3-D surface to project images where possible

Unified Planetary Coordinates (UPC) DatabaseUnified Planetary Coordinates (UPC) Database

INSTRUMENT LATITUDE LONGITUDE DIRECTION

LONGITUDE SYSTEM

PRIME MERIDIAN

DEM?

Viking 1 Ographic Positive West -180 to 180 EME 1950

Viking 2 Ographic Positive West -180 to 180 EME 1950

MGS MOC Ographic Positive West 0 to 360 IAU 1994

MGS TES Ocentric Positive West 0 to 360 IAU 1994

MGS MOLA Ocentric Positive East 0 to 360 IAU 1991

MO THEMIS/VIS Ocentric Positive East 0 to 360 IAU 2000 Yes

MO THEMIS/IR Ocentric Positive East 0 to 360 IAU 2000 Yes

MRO HiRISE Ocentric Positive East 0 to 360 IAU 2000 Yes

Planetographic – Angle between the equatorial planeand a line that is normal to the reference ellipsoid

Planetocentric – Angle between the equatorial planeand a line from the center of the ellipsoid

{


• A ‘geometry database’ for PDS data– Reads position & pointing from PDS image labels

• Designed to be updated and expanded as needed

– Converts to unified coordinates• Planetocentric latitudes• 0 to 360 degree positive east longitudes• Mars/MGS: IAU 2000 Mars Coordinates

– Calculates ‘improved’ pointing and location data• Corner, edge positions every X lines, samples• Images projected onto 3-D surface model where possible

• Why?– Improves data search & retrieval (geographic, feature-

based) for PDS image data– Enhances user ability to identify, combine, correlate

and analyze cross-mission PDS data– Can easily be used by Atlas II, ODE and other search

tools within and outside PDS (OGC-compatible)– Used to identify overlapping image sets to seed

control point and match point files• Will significantly enhance speed and quality of

production of image mosaics in ISIS from data such as HiRISE and LROC

UPC DatabaseUPC Database

CornerFootprint

UPCFootprint

Example: MOC WA Footprint


UPC StatusUPC Status• Database Population (Phase II Development)

– MGS/MOC WA, NA population complete (237,842 images)– Clementine UVVIS and NIR in progress (957,144 images)

• Upgrades to hardware now underway to improve through-put time– Others staged for population

• MRO/CTX (5,020 images)• MRO/HiRISE (157,727 images)• Viking (63,015 images)

– THEMIS IR and VIS now undergoing camera model improvement in ISIS• To be added to UPC when completed (Aug 08?)

– Lunar Orbiter in progress• Camera model (ISIS3) and PDS-compliant label being developed

• Preliminary version of UPC released to JPL (IMG) and Wash U (GEO)– MGS/MOC WA and NA database– Workshop held at USGS on March 27, 2008

• MapServer interface operational– Includes WMS, WFS and Raster support– Allows users to specify desired acquisition parameters– Displays detailed image footprint and basic information– Now being tested with live, direct access to UPC database


UPC Footprints: MGS/MOCUPC Footprints: MGS/MOC

UPC MOC

CumindexMOC

Offset is ~20 km

96 lon E, -36.5 lat

[email protected]


Atlas II and UPC IntegrationAtlas II and UPC Integration


• Restoration Activities– Clementine Near-Infrared (NIR)

• Peer Review complete for Release V. 01 of Clementine near-infrared 6-band lunar mosaic (100 m/pixel)• Complements UVVIS cube, completes full 11-band Clementine multispectral dataset• Final release of V 1.0 to PDS expected May 1, 2008

– Apollo Photo Digitization (Metric, Panoramic, Handheld Cameras)• USGS collaborators working with PI Robinson to evaluate scanned products to ensure quality and usability• 80 TB of data expected!• Working with ASU to create mission, instrument, and data catalog files for PDS archive for Apollo image

data

– Lunar Orbiter Digital Data Archive• Developed tiling scheme and labels for LO global mosaic and projected frames• Working to create mission, instrument, and data catalog files for PDS archive of LO data

– Viking Orbiter (VIS)• Decompressed the full collection and loaded the database with the resultant metadata for AtlasII searches. • AtlasII configuration and release pending

– Support for Lunar Data Providers• Support data providers (as needed) who respond(ed) to NASA PMDAP and/or LASER announcement of

opportunity

Baseline Services: RestorationsBaseline Services: Restorations


• Standards and ‘PDS2010’– Participating in ongoing testing of Label Validation Tools (Garcia,

Isbell)– Participating in Physical Media Working Group ‘Tiger Team’ (Isbell)– Participated in ‘PDS2010’ ‘Data/Information Model’ and ‘User’

Tiger Teams (Isbell, Gaddis, et al.)• PDS Community Service

– Represented PDS at LPSC meeting in March 2008• Presented abstract and poster on PDS lunar data holdings in IMG (Garcia)

– Representing PDS on NASA Working Groups (Gaddis)• Goal is to clarify and/or develop recommendations for standardization of

cartographic products from PDS data– Lunar Mapping Project (ESMD; T. Lavoie, MSFC)– Lunar Geodesy and Cartography Working Group (ESMD; Archinal, Chair)– Geologic Mapping Sub-Group of the PGG Cartography and Geologic Mapping

Working Group (L. Bleamaster, Chair)– ‘Planetary Cartography’ Working Group? (B. Bussey, Chair?)----TBD

Baseline Services: PDS-WideBaseline Services: PDS-Wide


THEMIS DATA NODE (TDN)

• Principal Investigator: Phil Christensen

• Location: Arizona State University (ASU), Tempe, AZ

• Operations– In full operational status since 2001

• Data Holdings– ~5 TB now available to community– Full copy of data online at USGS (updated quarterly), served via Atlas II

• Data Releases– Four per year (#21 to #24 in FY08); last in Jan 2008, next in April 2008 (this week!)

• Data Access– ‘Map Tool’ Image Browser shows coverage for MO/THEMIS, VO, MGS/MOC NA, MRO/CTX & HiRISE– JMARS (Java-based) is mission planning tool for data visualization and integration– THMPROC Web site provides customized, processed THEMIS VIS and IR image products

• ISIS geometric processing s/w used (now undergoing refinement at USGS)– All of these tools will benefit from UPC database

Data NodesData Nodes


HiRISE DATA NODE (HDN)

• Principal Investigator: Alfred McEwen

• Location: University of Arizona, Tuscon, AZ

• Operations– Now in full operational status, Eric Eliason is manager– MRO Extended Mission proposed to NASA (1 Mars Year, FY10 and FY11), ICD now in revision– Transition to PDS of data holdings will be facilitated by MRO/HiRISE data mission holdings in place at USGS

• Data Holdings– 18,500 gigabytes (18 TB) now available to community

• Data Access– JPEG2000 JPIP server allows efficient browse/roam/pan/zoom, used in the HDN and via the AtlasII– HiRISE Web Site (http://hirise.lpl.arizona.edu; tailored for HiRISE data products) and AtlasII (for generic search capability)

provide access to all released products– Data also served by JMARS (ASU) and ODE (GEO/WashU)

• Data Release– MRO project data release #1 in June 2007, #2 in September 2007, #3 in December 2007, #4 in March 2008– Color RDR products delivered October 2008– ~5 Months ahead of schedule!

• Establishes procedures and guidelines that will benefit Data Node for LRO Camera



LRO Camera (LROC) DATA NODE (LDN)

• Principal Investigator: Mark Robinson

• Location: Arizona State University (ASU), Tempe, AZ

• Operations– Plan modeled after HDN Operations, Ernest Bowman-Cisneros is manager– Transition to PDS of data holdings will be facilitated by LRO/LROC data mission holdings in place at USGS

• Data Holdings– LROC EDR Product SIS in review (starts this week!)– LROC RDR Product SIS in development (near final), review will be combined with EDR review– Up to ~60 TB planned for Exploration Phase

• NAC EDR/CDR – 36 TB• WAC EDR/CDR – 24 TB• RDRs – 3 TB

• Data Access– Web access, map-based data searches, parameter search, direct browse capability

• See site for Apollo data, for example: http://apollo.sese.asu.edu/

• Data Release– Quarterly deliveries planned (higher frequency expected after first release)

• LRO to transition from NASA Exploration Systems to Science after Yr 1



Extra Slides Follow


PDS IMG: Sub-Node Services Breakdown (FY08+)PDS IMG: Sub-Node Services Breakdown (FY08+)

Sub-Node Unique Sub-Node Expertise Leverage on Institution FY07 Emphasis and Accomplishments

USGS Planetary Science (Moon, Mars, Icy bodies, Galilean satellites)Planetary remote sensing (spectroscopy, radar, thermal infrared)Planetary CartographyPlanetary Geologic MappingPlanetary Image Analysis and CalibrationNASA mission participation (e.g., MER MI team, New Frontiers Moonrise proposal)

Planetary CartographyPhotogrammetryPhotoclinometryGISPlanetary GeodesyImage Processing and AnalysisISIS software systemParticipation in active NASA missions (Cassini, Mars Odyssey THEMIS, Mars Global Surveyor MOC and TES, MRO/HiRISE, Mars Express HRSC, etc.)

Mission Interface MGS MOC, MO THEMIS, MER MI, Cassini RADAR, MRO HiRISE, LRO LROC, Messenger MDIS, LCROSSBaseline Systems and Tools Online Disk Array/FTP servers/ DB, Hardware System Evolution, MGS/MOC Web Site, Archive Media Research, UPC implementationStandards and User Support Standards WG, IV&T testing, Map-a-Planet Development, Upgrades and MaintenanceRestoration Lunar Orbiter (digitization), Clementine NIR, Apollo cameras

JPL Data system development (active mission and archive systems)Database and Data Management System DevelopmentUser interface developmentImage Processing and AnalysisNASA mission participation (e.g., Mars Global Surveyor MOLA, Mars Odyssey THEMIS, MER, Mars Express HRSC and MARSIS, Cassini ISS and VIMS)Public Outreach

Mission project managementMission GDS/MOSInstrument flight softwareInstrument calibrationIndustry imaging standardsScientific animationVICAR software systemEducation and Public OutreachParticipation in active NASA missions (Cassini, Mars Odyssey, Mars Global Surveyor, MER, MRO, Mars Express, Phoenix, MSL, etc.)

Mission Interface MGS MOC, MO THEMIS, MER ops & eng Prods, Cassini ISS & VIMS, MRO HiRISE, CTX, MARCI, ONC, MEX HRSC, Phx MARDI, MECA-OM,

MECA-AFM, RAC, SSI, LRO LROC & LAMP, Messenger MDIS, M^3, MSL cameras

Baseline Systems & Tools AtlasII development and dataset integration, Hardware system evolution, Web Site usability upgradeStandards & User Support Standards WG, Label Validation Tool, M^3 archive format Photojournal maintenanceRestoration Viking Orbiter, Lunar Orbiter (analog tapes)


What is the UPC?What is the UPC?• Starts from but does not replace the many existing instrument-specific databases

• UPC Ingestion Pipeline uses ‘Conductor’ software developed at UA for MRO/HiRISE– Process scalable to unlimited number of processing systems (now using 8 Linux and Mac

OS X machines)– Uses local and/or remote (retrieved via wget) PDS-formatted source files

• Contains pre-computed geometry information for images, based on– ISIS and NAIF geometry subroutines – Best SPICE data available (including adjusted SPICE from geodetic solutions)– Single set of planet, coordinate, and other relevant definition files

• The UPC can and will be recomputed – To incorporate new images or new SPICE– To change the definition of useful data (ranges of i, , , and resolution)– To include additional search variables such as optical depth

• Users can specify desired ranges for – Latitude, longitude, elevation or Cartesian coordinates (x, y, z) – Resolution– Incidence, emission, and phase angles– Instrument, filter, etc.– Time-of-day– Other variables defined by user


What is the UPC? (2)What is the UPC? (2)• Three areas of functionality

– Availability and management of SPICE data• Four types and/or sources

– Predicts– Nadir– Reconstructed– C-Smith-ed (“refined” SPICE products)

– Generation of database for all instruments• Starts with best available SPICE data• Requires support in ISIS 3

– Primarily a Camera Model• Adaptable to mission diversity and peculiarities

– Numerous ingestion sources» PDS labels (MER, Viking)» Cumulative index tables (MOC)» ISIS 2 & 3?» Others

– Diverse DB query tools• Applicable to different needs

– Data processing, auto-seeding, mosaicking, cartography– PDS data searches

» MGS/MOC dataset is one of the most improved


The UPC DatabaseThe UPC Database• Builds on SPICE database system derived from original mission

kernels– Candidate images initially identified by UPC– Contents and/or paramaters designed to be refined and updated

periodically in ISIS system– Results can be provided back to user community in the form of new

SPICE kernels for included images

• Takes advantage of best available DEMs and SPICE kernels as well as advances in ISIS computing capabilities and techniques

• Full functionality and power of UPC developed and distributed in the ISIS system to science community


Why Use the UPC?Why Use the UPC?

• External users benefit from:– Availability of image data via web interface– Compatibility with web GIS standards– ISIS public availability exports most internal capabilities to

scientific community– Access to a growing library of data products– Access to most recent SPICE/Control Nets– Anticipate a large number of high-quality community-

generated products– Provide system and/or data to other institutions like the PDS


Footprint Example: HiRISEFootprint Example: HiRISE

Corner FootprintUPC Footprint

PSP_002158_2035_RED4_0

UPCOriginal


Footprint Example: MOC-WAFootprint Example: MOC-WA

Corner FootprintUPC Footprint

ab102402


UPC GIS CapabilitiesUPC GIS Capabilities

• UPC provides binary predicates (or Geometry Relationships) using GEOS (JTS) functions and extensions in the database backend (PostGIS)– Binary predicate operations take two geometries (e.g.,

MULTIPOLYGON) and returns a Boolean value indicating if the geometric relationship between them exists

– Examples:• Equals(geometry,geometry)

• Intersects(geometry,geometry)

• Contains(geometry,geometry)


UPC GIS GeometriesUPC GIS Geometries

LineString LinearRing MultiLineString GeometryCollection

Point MultiPoint Polygon MultiPolygon


GIS Analysis MethodsGIS Analysis MethodsIntersection Difference Union

Symmetric DifferenceIntersection of LineString

and Polygon


GIS Predicate ExampleGIS Predicate Example

• This example shows results of LineString and Polygon predicate functions


UPC Population ProcessUPC Population Process

• Initial population of the UPC is compute intensive

• A compute cluster of sorts is utilized to distribute processing tasks

• The process of downloading, computing, collecting and inserting the data into the UPC is called a pipeline

• Each process within the pipeline is called a segment of the pipeline


UPC Population (2)UPC Population (2)• The pipeline is developed using the Conductor software developed

by the Planetary Image Research Laboratory (PIRL) at the University of Arizona

– http://pirlwww.lpl.arizona.edu/software/Conductor.shtml

• Conductor is a Java application for managing queues of source files to be processed by sequences of procedures

• The queues for each segment of the pipeline are actually a set of two (MySQL) database tables

• One table contains the source file, the second contains the procedures required to process each source file completely through the segment

• One or more segments may exist in a single pipeline• The last procedure in a segment passes the output source to the

next segment in the pipeline


UPC Population (3)UPC Population (3)

• Segments of a pipeline are run independently from one another on each compute node in the cluster

• Segments are distributed across one or many compute nodes

• One or more copies of the same pipeline segment may be run on a single node

• This system is scalable within nodes and across many platforms – We are currently run up to 12 different “nodes” around the

clock– These nodes are both Linux and Mac OS X systems



• The UPC pipeline is something of a hybrid• It has 4 segments for each instrument in the

UPC database• Typically each instrument has its own

Download and ISIS Ingestion segment• All instrument data is processed through the

GeomProc (geometric data) and Publish (database insert) segments



DownloadMOCMOCNA

MOCWA

DownloadTHEMISTHEMISIR

THEMISVIS

DownloadCLEM

DownloadCTX

DownloadHiRISE HIRISE

CTX

CLEMENTINE

GeomProc Publish



• Segment design has several criteria– Complexity of instrument

• Some PDS EDRs do not have INSTRUMENT_ID

– Number of images in instrument dataset• MOC has ~238,000 images, 96,618 of them are NA

• Clementine has ~1.8 million

– Ingestion into ISIS differs by instrument• SPICE initialization may also differ

– Geometric processing and database publication (insertion) is consistant or sufficiently generalized

Lisa Gaddis (USGS) and Sue LaVoie (JPL)

Documents

Transcript of Lisa Gaddis (USGS) and Sue LaVoie (JPL)