GIS in CMSP

• Ecological

Spatial Dimensions of CMSP

• Ecological• Oceanographic

Spatial Dimensions of CMSP

• Ecological• Oceanographic• Economic

Spatial Dimensions of CMSP

• Ecological• Oceanographic• Economic• Political

Spatial Dimensions of CMSP

• Ecological• Oceanographic• Economic• Political• Social

Spatial Dimensions of CMSP

• Uncertainty• Temporal dynamics• Engaging & curating citizen science• Big data integration/synthesis• Spatial connectivity of processes

Key Challenges

Developing analytical tools for the marine ecology

community

Patrick N Halpin

Jason RobertsMarine Geospatial Ecology Lab

Duke University, USA

MGET:Marine Geospatial

Ecology Tools

ESRI Oceans SummitNovember 7, 2012

Linking ocean biogeographic information systems with management applications

Marine Geospatial Ecology LabNicholas School of the Environment

Duke University Marine LabMGEL Lab team:

DurhamProf. Pat HalpinDr. Andre BoustanyEi FujiokaBen DonnellyJesse Cleary

BeaufortDaniel DunnConnie KotCorrie CurticeErin LaBrecqueJerry MoxleyShay Viehman

Remote LocationsJason RobertsBen Best (now @ UCSB)

Marine Ecology

Ecoinformatics Geospatial analysis

Linking biological, satellite and ocean

observing data

Developing innovative data analysis and visualization tools

Field data collection, analysis and modeling

Applied marine science & conservation

http://mgel.env.duke.edu

Marine Geospatial Ecology Lab

We have found ArcGIS software and geoprocessing tools to be not fully capable for many ocean and marine applications.

So we have been augmenting ArcGIS with our own tools:Marine Geospatial Ecology Tools (MGET)

What does the marine community need from ArcGIS?

Better support for scientific data and analysis, particularlydata with time and depth dimensions.

MGET can also be invoked from most programming languagesMGET download:

http://mgel.env.duke.edu/mget

currently operating with ArcGIS 10.x

250+ geoprocessing tools for marine ecologyDesigned for non-programmersFree, open source softwareWritten in Python, R, MATLAB, C#, and C++Minimum requirements: Windows XP, Python 2.4 ArcGIS 9.1 or later currently needed for many toolsArcGIS and Windows are only for-fee requirements

Citation:Roberts, JJ, BD Best, DC Dunn, EA Treml, and PN Halpin (2010) Marine Geospatial Ecology Tools: An integrated framework for ecological geoprocessing with ArcGIS, Python, R, MATLAB, and C++. Environmental Modelling & Software 25: 1197‐1207

Data ConversionNASA MODIS SST

MGET provides tools that detect oceanographic features in remote sensing images

These are some of the most popular tools in MGET

Terra Aqua

Identifying SST fronts

~120 km

AVHRR Daytime SST 03‐Jan‐2005

28.0 °C

25.8 °C

Mexico

Front

Cayula and Cornillion (1992) edge detection algorithm

Freq

uency

Temperature

Optimal break 27.0 °C

Strong cohesion front present

Step 1: Histogram analysis

Step 2: Spatial cohesion test

Weak cohesion no front

Bimodal

Example output

Mexico

ArcGIS model

Analyzing larval connectivityCoral reef ID and % cover maps Ocean currents data

Tool downloads data for the region and dates you specify

Larval density time series rasters

Edge list feature class representing dispersal network

Original research by Eric A. Treml

ChlorophyllSSTBathymetry

Presence/absence observations

Sampled environmental data

Multivariate statistical model

Probability of occurrence predicted from environmental covariates

Binary classification

Model species habitat

86 Countries

~2575 installs since August 2009

Most users with up-to-date ArcGIS installs

What does the ocean / marine GIS community need …?

4D GIS4D data model (x,y,z & t)4D GUI4D Geoprocessing

We need a 4D data modelArcGIS should support 4 dimensions (x, y, z, and time) in its core data model

Full generality (arbitrary number of dimensions, having arbitrary meanings, a la MATLAB) may not be necessary

Arc Marine Data Model addressed some of thisNeed data providers releasing data with Arc Marine schemaNeed tool providers building tools atop Arc Marine

We need better support for 3D and 4D scientific data formatsHDF 4 and 5, HDF-EOS, NetCDF 3 and 4, OPeNDAPNeed solution for the MANY products that do not have CF-compliant metadataNeed solution for MANY products that are sliced into series of files by time and/or depth:

e.g. 3D datasets represented as series of 2D filesSometimes files are actually 3d but 3rd dimension is length 1

Better support for swath data (irregularly spaced grids)Need ArcGIS to understand time averaging

We need a 4D GUINeed to visualize and manipulate 3d and 4d dataBig step forward in Arc 10 with Time Slider

Having a Depth Slider, or similar, would be helpfulSome in user community attracted to other solutions, e.g. Google Earth, EonFusion; easier than ArcSceneSpeed and ability to easily manipulate point of view in space and time are critical, as in a video gameEasy, fast, glitch-free, exportable animation is critical

Other GUI improvementsNeed special symbology for visualizing vector data such as ocean currents and winds

These data often come as a pair of arrays: u array contains horizontal vector components, v array contains verticalArrays may be 2d, 3d (x, y, time) or 4d (x, y, depth, time)Need to display magnitude and direction as rastersNeed to display MATLAB-style quiver plotMGET provides tools for these

Need better support for level 1 and 2 satellite products

We need 4D geoprocessing Some common scenarios:

Combine time series of remote sensing images into monthly, seasonal or annual climatologies

Reduction of 3d array into several or one 2d arrayReduce temporal resolution of time series of remote sensing images to deal with clouds, etc.

Combine 8 daily images into single 8-day imageBatch this across years of data

Interpolate values of a 3d or 4d dataset at points, lines, or polygons

e.g. match up points to SST images, by date, and sample

We need 4D geoprocessing Some common scenarios:

Create Hovmöller plots—2d plots where x and y axes can be any dimension (x, y, z, or time)“Map algebra”-style processing across 3 and 4 dimensions

Useful for algorithms for detecting oceanographic featuresNeed ability to read and write 3d and 4d numpy arrays from Python geoprocessing tools

Create animated maps showing progression through timeE.g. background raster of SST with animal tracks on topCurrent procedure: create bitmaps for each time slice with VB (Arc 9.x) or arcpy (10.0) and make video with ffmpeg

MGET provides many such tools

For example

MGET’s solutionWe built our own 4d data model for MGET’s internal useAbstractions for both tabular (vector) and gridded data

SummaryTo best serve the marine ecology community, ArcGIS needs deep support for time and depth dimensionsEnhancements needed in the core data model, scientific data format support, the GUI, and geoprocessingIn MGET, we worked around this by wrapping ArcGIS in our own object model and leveraging non-ESRI libraries

This required more than a year of development effortThe difficulties discussed here are impeding the use of scientific data with ArcGIS by those who cannot make a substantial programming investment

Discussion?

MGET sponsors

Marine Geospatial Ecology LabNicholas School of the EnvironmentDuke University Marine Lab

MGET download:http://mgel.env.duke.edu/mget

A Distributed Approach to Ocean, Atmosphere & Climate

Model Data InteroperabilityRich Signell USGS, Woods Hole, MA

Curtis Price USGS, Rapid City, SD

ESRI, Redlands, CA Nov 7, 2012

US Integrated Ocean Observing System (IOOS®)

IOOS® Plan defines:• Global Component• Coastal Component

17 Federal Agencies11 Regional Associations

IDVIDV

IOOS Recommended Web Services and Data Encodings

In-situ data (buoys, piers, towed sensors)

Gridded data (model outputs, satellite)

OGC Sensor Observation Service

(SOS)

OpenDAP and/orOGC Web Coverage

Service (WCS)

XML based on OGC Observations and

Measurements (O&M)

NetCDF using Climate and Forecast (CF)

conventions

Images of data OGC Web Map Service (WMS)

GeoTIFF, PNG etc.-possibly with

standardized styles

Data Type Web Service Encoding

Inundation Extra-tropical – Gulf of MaineTropical – Gulf of Mexico

- 4 models: 3 unstructured grid +1 structured grid- Coupled wave-storm surge-inundation (TWL)- Consistent forcing, validation and skill assessment using existing IMEDS tool -Extensive observational data sets for historical storms Ike, Rita and Gustav in standard formats-SURA has provided supercomputer resources

Extratropical Grid

Tropical Grids for Galveston Bay

Structured Curvilinear Orthogonal Grid

Stretched surface and terrain following vertical coordinates

Curvilinear orthogonal horizontal coordinates

NetCDF Climate and Forecast (CF) Conventions provide a solution

Groups using CF:

GO-ESSP: Global Organization for Earth System Science Portal

IOOS: Integrated Ocean Observing System

ESMF: Earth System Modeling Framework

OGC: Open Geospatial Consortium (GALEON: WCS profile)

ROMS

POM

WW3

WRF

ECOM

NcML

NcML

NcML

NcML

NcML

Common Data Model

OPeNDAP+CF

WCS

NetCDF Subset Service

THREDDS Data Server

Standardized (CF) Virtual Datasets

NonstandardOutput Files(distributed)

Web Services

IOOS Model Data Interoperability Design

Matlab

Panoply

IDV

ERDDAP

Clients

NetCDF-Java

Library or Service

NcML, NetCDF-Java and THREDDS Data Server built and supported by Unidata under NSF-support

WMS

ncISO

ArcGISNetCDF4-Python

FVCOM

Python

EDC ERDDAPNetCDF-Java

• Built on top of Unidata’s NetCDF C library

• Allows users to open and access an OPeNDAP URL just like a local NetCDF4 or NetCDF4 file

• Can also write NetCDF3 and NetCDF4 files

NetCDF4-Python: OPeNDAP → ArcGIS

ROMS

POM

WW3

WRF

ECOM

NcML

NcML

NcML

NcML

NcML

Common Data Model

OPeNDAP+CF

WCS

NetCDF Subset Service

THREDDS Data Server

Standardized (CF) Virtual Datasets

NonstandardOutput Files(distributed)

Web Services

IOOS Model Data Interoperability Design

Matlab

Panoply

IDV

ERDDAP

Clients

NetCDF-Java

Library or Service

NcML, NetCDF-Java and THREDDS Data Server built and supported by Unidata under NSF-support

WMS

ncISO

ArcGISNetCDF4-Python

FVCOM

Python

EDC ERDDAPNetCDF-Java

OPeNDAP UGRID → ArcGIS Shapefile

OPeNDAP GRID → ArcGIS Raster

OPeNDAP UGRID → ArcGIS TIN

New 10.1 Python Toolboximport netCDF4import arcpynetCDF4.Dataset(url)arcpy.LandXMLToTin_3d(..arcpy.TinRaster_3d(…

Final Step: TIN → RASTER

• Make sure that NetCDF4-Python and other python packages continue to work for ArcGIS users

• Coordinate with Enthought and Continuum Analytics so that Python distributions like EPD can be compatible. Advance distribution of interface specs?

• Allows ESRI users to leverage many other packages (e.g. Scipy, pyHDF, 100 more in EPD) with minimal effort. That’s why USGS has a site license for EPD.

Wish 1: Coordinate with leading scientific python distributions

• Support for native grid data (Common Data Model data structures) in ArcGIS: We can import raster or TIN, but we still don’t have a good way to move data from curvilinear structured-grid models like ROMS to ArcGIS

• Should just be able to instantiate common data model structures in ArcPy from Numpy Arrays (e.g. Shouldn’t have to write LandXML files to import a TIN)

Wish 2: Support Native Grid Data in ArcGIS

• ArcGIS users (e.g Scientists, coastal decision makers, planners, emergency responders) can “help themselves” to the world of OPeNDAP ocean, meteorological and climate data

• More scientific customers for ESRI from the NetCDF/HDF/OPeNDAP world

Result: The Win-Win!The Community and ESRI Both Benefit!

Opportunities for Improvement:Perspectives from the Monterey Bay Marine GIS User Group

Lisa Wedding1,2, Mary Yoklavich1 Nadine Golden3, Diana Watters1, Shaun Walbridge 4,5,6, Lisa Krigsman1, Rebecca Miller1, David Huff1

U.S. Department of Commerce | National Oceanic and Atmospheric Administration | NOAA Fisheries | Page 45

1 NOAA Fisheries Ecology Division, Santa Cruz Lab2 UCSC Institute of Marine Sciences3 USGS Pacific Coastal and Marine Science Center4 UCSB Ecology , Evolution and Marine Biology Department5 ESRI6 National Center for Ecological Analysis and Synthesis

http://swfsc.noaa.gov/MontereyBayGIS/

•to foster collaboration between agencies, NGOs &marine labs in Monterey Bay

•supports GIS training, capacity building & increasedawareness of GIS data sets

• Invited speakers, MGET workshop

•Members provided inputfor Ocean Summit(~100 members)

Monterey BayMarine GIS User Group

Presentation Outline• MARINE GIS CHALLENGES /OPPORTUNITIES

CASE STUDIES EXAMPLES

• Monterey Bay Marine Marine GIS User Group• Human dimensions• 3D (depth) & 4D (time) geoprocessing• Sharing map products in realtime

• NOAA Fisheries Ecology Division• Data management – expand ArcMarine

• USGS Pacific Coastal & Marine Science Center• Vertical datum• 3D Visualization and metrics• Bathymetric data management

Integrate Human Dimensions

CHALLENGE: Human dimension data sets identified a significant data gap • Priority for spatial management

SeaSketch, UCSB McClintock Lab

OPPORTUNITIES:1) Expand human dimensions in ArcMarine database structure

2) Support GIS tool development to collect and analyze these data sets (e.g. SeaSketch, eCatch, etc.)

http://www.esri.com/news/arcwatch/0812/graphics/feature1-lg.jpg

• Migratory spp. utilize entire water column & move across oceanographic fronts

Expand 3D & 4D capabilities

NOAA & Turtlewatch Program

CHALLENGE: • Limited capability for dynamic 3D & 4D

data sets

OPPORTUNITY:• Support geoprocessing & mapping of

dynamic movement in space & time

Sharing Map Products in real time

CHALLENGE: Real time map sharing for dynamic ocean management

• Ex/ NOAA serves online maps to update boats in Cape Cod of dynamic management areas

OPPORTUNITY:increase ArcGIS capabilities for distributing dynamic maps

• ArcGIS online, ESRI mobile applications, etc.

NOAA NEFSC

Data Management

CHALLENGE: Workflow for integrating, managing and processing large data sets (different software/tools)

NOAA Habitat Ecology Team Visual Surveys

Delta Manned Sub

•Georeferenced data ondistribution & abundance of demersal fishes & macro-invertebrates

Link tools to ArcMarine

OPPORTUNITY:1) Link tools & extensions to ArcMarine

geodatabase structure & feature classes

• Data processing (e.g.,Geospatial ModellingEnvironment)

• Data analysis (e.g.,MGET)

• Provides users further rationale to adopt this standard & streamline workflow (e.g., ArcHydro)

Dawn Wright, Michael Blongewicz, Patrick Halpin & Joe Bremen (2007)

USGS Sea Floor Mapping•collect seafloor data & ground truth information

CHALLENGE: Vertical Datum functionality and management•NGS geodetic tool kit - conversion

OPPORTUNITY: Vertical Datum standard and conversion

USGS, CSUMB Sea Floor Mapping Lab, NOAA and partners

3D VisualizationCHALLENGE: Limited 3-D rendering functionality in ArcGIS• recent raster catalog & mosaic data sets

made improvements

OPPORTUNITIES:1) Further improvements in the functionality & quality of 3D

rendering in ArcMap• interactive hillshading and profiling

2) Enhanced raster data symbolization (2-and 3-D environment)

http://www.usgs.gov/

Bathymetric Data Management

CHALLENGE: Management & processing of large bathymetric data sets in ArcGIS

• Recent 10.1 extension advanced capability

• ArcGIS for Maritime: Bathymetry extension

• management & composite modeling of bathymetric data/metadata

ArcGIS for Maritime:Bathymetry

OPPORTUNITES: 1) Expand from working solely with BAGs (Bathymetric

Attributed Grid) to include ArcGIS GRID format data

2) Support data processing with other extensions and tools (Benthic Terrain Modeler, MGET, etc.)

ESRI Maritime Team Bathymetry group lead: Bertrand Baud

BTMBenthic Terrain Modeler

• BTM recently updated function in ArcGIS 10x with enhanced metrics (e.g., curvature, VRM)

OPPORTUNITES1) Ability to derive metrics at multiple

spatial scales 2) Streamline process with other

extensions/tools

Dawn Wright, Shaun Walbridge

Summary of Opportunities

• Support 3D/4D geoprocessing, visualization & mapping

• increase capabilities for distributing dynamic maps

• Vertical datum standard & conversion

• integrate human dimensions data

The way we view the ocean determines how we manage it

• Advancing marine GIS will help facilitate mapping, ID & placing value to heterogeneity

• See value of spatially discrete areas & provide rationale for spatial management

CONCLUSION

NOAA Biogeography Branch

Thank you and QuestionsContact Information:

Lisa Wedding, PhD Habitat Ecology TeamNOAA Fisheries Ecology Division &UCSC Institute of Marine Sciences110 Shaffer Road, Santa Cruz, CA 95060 phone: 831-420-3993 fax: 831-420-3977 email: lisa.wedding@noaa.gov

http://swfsc.noaa.gov/MontereyBayGIS/U.S. Department of Commerce | National Oceanic and Atmospheric Administration | NOAA Fisheries | Page 60

GIS in CMSPSpatial Dimensions of CMSPSpatial Dimensions of CMSPSpatial Dimensions of CMSPSpatial Dimensions of CMSPSpatial Dimensions of CMSPKey ChallengesSlide Number 8Slide Number 9Slide Number 10Slide Number 11Slide Number 12����Data Conversion�NASA MODIS SSTIdentifying SST frontsSlide Number 15Slide Number 16Slide Number 17What does the ocean / marine GIS community need …?We need a 4D data modelWe need better support for 3D and 4D scientific data formatsWe need a 4D GUIOther GUI improvementsWe need 4D geoprocessing We need 4D geoprocessing MGET provides many such toolsMGET’s solutionSummarySlide Number 28A Distributed Approach to �Ocean, Atmosphere & Climate �Model Data InteroperabilityUS Integrated Ocean Observing System (IOOS®)IOOS Recommended Web Services and Data EncodingsSlide Number 32Structured Curvilinear Orthogonal GridNetCDF Climate and Forecast (CF) �Conventions provide a solutionIOOS Model Data Interoperability DesignNetCDF4-Python: OPeNDAP → ArcGISIOOS Model Data Interoperability DesignOPeNDAP UGRID → ArcGIS ShapefileOPeNDAP GRID → ArcGIS RasterOPeNDAP UGRID → ArcGIS TINFinal Step: TIN → RASTERWish 1: Coordinate with leading �scientific python distributionsWish 2: Support Native Grid Data in ArcGISResult: The Win-Win!�The Community and ESRI Both Benefit!Opportunities for Improvement:� �Perspectives from the Monterey Bay Marine GIS User Group��Monterey Bay�Marine GIS User GroupPresentation OutlineIntegrate Human DimensionsExpand 3D & 4D capabilities Sharing Map Products �in real timeData ManagementLink tools to ArcMarine �USGS Sea Floor Mapping3D VisualizationBathymetric Data ManagementArcGIS for Maritime:�BathymetryBTM�Benthic Terrain ModelerSummary of OpportunitiesCONCLUSION�Thank you and Questions