AusCover Earth Observation Services& Data Cubes

Matt Paget – AusCover data and systems lead coordinator & AGDC team leaderCSIRO Land and Water, Canberra

IntroductionTERN / AusCover

• TERN’s Remote Sensing Data Facility• Coordinated infrastructure for management and access to EO products from

Universities, CSIRO, Geoscience Australia (GA) and Bureau or Meteorology (BoM)

Australian Geoscience Data Cube (AGDC)• Partnership between GA, CSIRO and National Computation Infrastructure (NCI)• Version 2 is presently in development, due June 2016

Committee on Earth Observation Satellites (CEOS)• AGDC forked as a collaboration between CEOS Systems Engineering Office (NASA-

SEO), CSIRO and GA• Implementations for Kenya and Colombia

Copernicus Data Hub• Australia and south Asia mirror of Europe’s Sentinel satellite series and services• Partnership between Federal and State Govt agencies and departments and

AARNET/RDS/NCI

Two PathwaysPresent

Data producers / custodians• Internal infrastructure, downloads etc• Storage and services close to

custodians

Open data access• Low level programmable data services• Interoperability with related and

cross-discipline activities

Aggregation• Promote open data formats and

consistent metadata• Metadata services and data portals

Emerging

Data producers / custodians• Large central data stores• Coordinated data and resource

management between custodians

Open data access• Analytics facility close to data• Optimised storage units with API

access

Aggregation• Mature algorithms and exploratory

data analysis• Community engagement in

validation and methods

AusCoverwww.auscover.org.au

(in transition)

Remote sensing data delivery backbone

• Interoperability formats and services

• Storage and infrastructure

Data production network• Central coordination and regional

nodes• National biophysical data products

Ground validation and characterisation• Data collection and processing

protocols • Field campaigns with airborne

hyperspectral and lidar

AusCover data systemDistributed network of storage and data access services• Extensible nationwide system• Data providers upload to their own

storage

Open standards, systems and services• Non-proprietary formats, OGC

Direct data access• Programmable and Human• No license click-throughs

Metadata and data licensing• Aligned with TERN, ANZLIC and

ANDS/RDA

Central coordination• Website, Metadata, Services• TERN coordination

Emerging EO informaticsEstimated storage and processing needs and opportunities for the next decade• Increasing volume: spatial, temporal and spectral resolution and number of sensors• Increasing velocity: near real time applications and automation• Increasing variety: best practice processing and integration of disparate sources

AGDC and CEOS Data Cubes

Dice… …and Stack

Version 1• GA’s Unlocking Landsat Archive• Tiling, Indexing, Geotiff, HPC workflow• Water Observations from Space

• Collaboration between GA, CSIRO and NCI• Uptake by CEOS and Kenya Cube development

Version 2• Analysis Ready Data• Relax requirement for a common grid• Modular design• Multiple platforms & environments

• Partnership and Governance• Transition Plan (archives, products & users)• Greater open collaboration with the community• Functional requirements due June 2016

www.datacube.org.augithub.com/data-cube/agdc-v2

github.com/ceos-seo

Storage Query and Access Module

Task Execution

N-Dimensional Array Interface

DataIndex Store Data

Multi-Dimensional Storage Unit

Business Processes(Data Management, Access Controls, Job Management, Etc.)

Integrating Tool

Forest Economics App N

Reporting Visualization

Application / GIS Mobile App

Data CubeInfrastructure

Data & Application Platform

UI & Application Layer

Virtual LabPlatform

Analytical Task

Definition

Data Acquisition& Inflow Analysis Ready DataObservations

Data Cube Notional Architecture

Data Collection 1 Data Collection 2

DataIndex Store Data

Multi-Dimensional Storage Unit

Authentication, Authorisation,

Accounting

Discovery Web Services

Ingester (ARD and Result)

SummaryNCRIS and TERN

• Investment has enabled open data access across Australian institutions• Major contribution to, and application of, open source standards and

services

International links• CEOS WGISS Future Data Architectures (NASA, USGS, ESA, JAXA)• Google Earth Engine and Amazon Web Services• Unprecedented Australian Govt and research collaboration through

Copernicus Data Hub

Implications and opportunities• Reducing the need for expert EO know-how• Enabling best available data and analytics from the research sector for

applications and operations• Building innovative and common infrastructures that adapt to massive

data volumes and emerging storage and compute environments

Contact details

AusCover www.auscover.org.au (in transition)Alex Held, CSIRO Canberra alex.held@csiro.auStuart Phinn, UQ & JRSRP stuart.phinn@uq.edu.auMatt Paget, CSIRO Canberra matt.paget@csiro.auPeter Scarth, UQ & DSITI peter.scarth@qld.gov.au

AGDC www.datacube.org.au (in transition)Trevor Dhu, GA trevor.dhu@ga.gov.auRob Woodcock, CSIRO Canberra robert.woodcock@csiro.auLesley Wyborn, ANU & NCI lesley.wyborn@anu.edu.auLeo Lymburner, GA leo.lymburner@ga.gov.au

CEOS Data Cube www.ceos.orgAlex Held, CSIRO Canberra alex.held@csiro.auBrian Killough, NASA-SEO brian.d.killough@nasa.gov

Australia’s Regional Copernicus Data Access/Analysis HubAn integrated ‘Team Australia’ approach toSupport government information requirements.

Support the broader objective of enhancing access to satellite Earth observation data by

research, industry and civil society.

Facilitate collaboration between Australians, Europeans and inhabitants of the South-East

Asia-South Pacific region in exploitation of Earth observation data.

Benefits for: Australia, the region, EC/ESA/EUMETSAT, and the global

satellite EO community.

PartnersConsortium CollaboratorsSupporting

All Sentinel Products For

Partnerships(pay-per-use)

Australia’s Regional Copernicus Data Access/Analysis Hub

Master Repository (NCI)

ESAIntHub

EUMETCASTTerrestrial

Operational ProductsDeep National Archives(Government Systems)

GEANT/AARNET

Original Format Data Files(HSM Storage)

Analysis-Ready Data(Lustre/spinning disk)

Cloud and HPC(Petascale)

Agreed Standards InteroperableServices(e.g. Thredds)

Regional

Portal(a la ‘S

ciHub’)

Industry Exploitation

R&D for:Research/AcademiaCivil SocietyGovernmentIndustryAustralia and Region

Delivery for:AustraliaRegion