Analyzing Multidimensional Scientific Data

in ArcGIS

Nawajish Noman

Deng Ding

Outline

• Part I

- ArcGIS Platform

- Scientific Multidimensional Data

- Ingest and Data Management

• Part II

- Data Exploration and Visualization

- Analysis

- Extending Analytical Capabilities using Python

• Part III

- Publishing and Sharing Services

- Consuming Services

Desktop Web Device

Server Online Content

and Services

Portal

Available on the Open Web . . .

. . . and On-Premises

ArcGIS An Integrated Web GIS Platform

Providing Mapping, Analysis, Data

Management, and Collaboration

Identity

Security

Scientific Multidimensional Data

• Stored in netCDF, GRIB, and HDF formats

• Multidimensional

• Ocean data

Sea temperature, salinity, ocean current

• Weather data

Temperature, humidity, wind

• Land

Soil moisture, NDVI, land cover

ArcGIS

direct ingest

data management

visualizationanalysis

share

Scientific Data in ArcGIS - Vision

Challenges

variety of formats

volume & velocity

redundancy

portability

scalability

reproducibility

integration

standards

accessibility

Analyze ShareManage

• Data is represented as

• Raster

• Feature

• Table

• Direct read

• Exports GIS data to netCDF

Representing Scientific Data in ArcGIS

• Directly reads netCDF file using

- Make NetCDF Raster Layer

- Make NetCDF Feature Layer

- Make NetCDF Table View

• Ingest OPeNDAP Service

- Output dynamic multidimensional raster

- Support Sub-setting

• Scientific data formats are supported in mosaic dataset

- netCDF

- HDF

- GRIB

Ingesting Scientific data in ArcGIS

Climate and Forecast (CF) Conventionhttp://cf-pcmdi.llnl.gov/

Initially developed for

• Climate and forecast data

• Atmosphere, surface and ocean model-generated data

• Also for observational datasets

• CF is now the most widely used conventions for geospatial netCDFdata. It has the best coordinate system handling.

• Current version 1.6

• You can use Compliance checker utility to check a netCDF file.

http://cf-pcmdi.llnl.gov/conformance/compliance-checker/

CF Convention

• Geographic Coordinate Systems (GCS)

- X dimension units: degrees_east

- Y dimension units: degrees_north

• Projected Coordinate Systems (PCS)

- X dimension standard_name: projection_x_coordinate

- Y dimension standard_name: projection_y_coordinate

- Variable has a grid_mapping attribute.

- CF 1.6 conventions currently supports thirteen predefined coordinate systems (Appendix F: Grid Mappings)

• Undefined

- If not GCS or PCS

• ArcGIS writes (and recognizes) PE String as a variable attribute.

NetCDF and Coordinate Systems

Demo: Ingest

What about Aggregation?

• Create a seamless multi-dimensional cube from

- files representing different regions

- files representing different time steps/slices

• Mosaic dataset supports multiple files and variables, normalize time and depth

Multidimensional Mosaic Dataset

mosaic

table

raster pixels

Representing multivariate collection of multidimensional rasters in ArcGIS

mosaic dataset

spatially-indexed catalog

multi-resolution, multivariate, multidimensional

reduce storage redundancy & pixel resampling

defines information products

on-demand processing

HDF

GRIB

netCDF

d-aware rasters

Tabular View

Tabular view of items in a multivariate multidimensional mosaic dataset

Raster Shape Variable StdTime StdZ

… … Temperature 3/22/2016 -10

… … Temperature 3/23/2016 -10

… … Temperature 3/24/2016 -10

… … Salinity 3/22/2016 -10

… … Salinity 3/23/2016 -10

… … Salinity 3/24/2016 -10

… … Temperature 3/22/2016 -20

… … Temperature 3/23/2016 -20

… … … … …

• Supports netCDF, HDF and GRIB- Spatial Aggregation

- Temporal Aggregation

- On-the-fly analysis

• Serve as Multidimensional- Image Service

- Map Service

- WMS

• Supports direct ingest

• Eliminates data conversion

• Eliminates data processing

• Improves workflow performance

• Integrates with service oriented architecture

Scientific data support in Mosaic Dataset

Demo: Management

Behaves the same as any layer or table

• Display

- Same display tools for raster and feature layers will work on multi-

dimensional raster and feature layers.

• Graphing

- Driven by the table just like any other chart.

• Animation

- Multi-dimensional data can be animated through time dimension.

• Analysis Tools

- Will work just like any other raster layer, feature layer, or table. (e.g.

create buffers around points, reproject rasters, query tables, etc.)

Using Scientific Data in ArcGIS

• Slicing

• Temporal animation using Time Slider

• Dimensional animation using Range Slider

• Predefined renderer

Visualization of Scientific Data

Time = 1

141 241 341

131 231 331

121 221 321

111 211 311

441

431

421

411

142 242 342

132 232 332

122 222 322

112 212 312

442

432

422

412

143 243 343

133 233 333

123 223 323

113 213 313

443

433

423

413

Y

X

Time

Changing Time Slice

Animating through Time using Time Slider

Animating through Depth using Range Slider

• New Vector Field renderer for raster

- Supports U-V and Magnitude-direction

- Dynamic thinning

- On-the-fly vector calculation

• Eliminates raster to feature conversion

• Eliminates data processing

• Improves workflow performance

Visualization of Raster as Vectors

Demo: Visualization

• Hundreds of analytical tools available for raster, features, and table

• Temporal Modeling

- Looping and iteration in ModelBuilder and Python

Spatial and Temporal Analysis

• Several analytical functions are available out of the box

• Functions are chained together to create complex model

• Used to perform on-the-fly analysis

• Extend analytical capability using Python Raster Function

On-the-Fly Processing using Raster Functions

Modeling with Raster Function Template (RFT)

• A scientific model = a raster function template

• Windchill = 35.74 + 0.6215T - 35.75(V^0.16) + 0.4275T(V^0.16)

• RFT

- Created using

- Contains Query, GroupName, Variables

- Set Mosaic dataset properties

- Used during adding raster data

• netCDF4-python is included in 10.3/Pro

• Read and write netCDF file

• Conversion time values to date

• Multi-file aggregasion

• Compression

• SciPy

• Python Raster Function

https://www.unidata.ucar.edu/software/netcdf/workshops/2012/netcdf_python/netcdf4python.pdf

Python Package: netCDF4-Python, SciPy

Create Space-Time Cube & Emerging Hot Spot Analysis

Analyzing Multidimensional Scientific Data in ArcGIS

Supplemental tools

• OPeNDAP to NetCDF

• Make NetCDF Regular Point Layer

• Make NetCDF Station Point Layer

• Make NetCDF Trajectory Point Layer

• Describe Multidimensional Dataset

• Get Variable Statistics

• Get Variable Statistics Over Dimension

• Multidimensional Zonal Statistics

• Multidimensional Zonal Statistics As Table

http://blogs.esri.com/esri/arcgis/2013/05/24/introducing-the-multidimension-supplemental-tools-2/

Python – Extending Analytical Capabilities

• Marine Geospatial Ecology Tools (MGET)

• Developed at Duke Univ.

• Over 180 tools for import management,

and analysis of marine data

http://mgel.env.duke.edu/mget

• Australian Navy tools

(not publicly available)

Community Developed Tools

Demo: Analysis

Desktop Web Device

Online

Content and

Services

Server

multivariate multidimensional

mosaic dataset

Disseminating

professional

geospatial analysts

• Map Service (supports WMS)

- Makes maps available to the web.

• Image Service (supports WMS)

- Provides access to raster data through a web service.

• Geoprocessing Service

- Exposes the analytic capability of ArcGIS to the web.

Sharing / WMS Support (for multi-dimensions)

Sharing Your Map as Map Services

• File ≻ Share As ≻ Service

• Features overlaid on rasters

• Static vs Dynamic map service

• Multidimensional WMS

Sharing Data & Information Products as Image Service

• Mosaic Dataset ≻ Share As Image Service

• Pixels & item metadata

• On-demand server-side processing

• Raster models: predefined or client-specified

• Multidimensional info, filtering, vector fields

Enable access to a dynamic representation of your information product as an image service

Publishing a WMS on ArcGIS Server

• Enable WMS capabilities on Service Editor or Manager

Multi-dimensional data support in WMS

• getCapabilities

- Supports time, elevation and other dimensions (e.g. depth)

• getMap

- Returns map for any dimension value

&DIM_<dimensionName>=<value>&

- Supports CURRENT for time dimension

&TIME=CURRENT&

• getFeatureInfo

- Returns information about feature for any dimension value

Multi-dimensional WMS in ArcMap

• Supports WMS layer like any other layer

• Animates a time enabled WMS layer using time-slider

• Slices for any dimension value are accessible with ArcObjectsPublic Sub UpdateWMSServiceLayerDimensionValue()

'UID for wms service layer type

Dim pUid As New uid

pUid = "{27ABB9EC-7A26-4cf8-8BD4-70EC1D274E17}"

Dim pWMSMapLayer2 As IWMSMapLayer2

'calling a function to find the layer from active dataframe

Set pWMSMapLayer2 = GetLayer(pUid, "myWMSLayer")

'setting values to dimensions

Dim pDimNameValues As IPropertySet

Set pDimNameValues = New PropertySet

pDimNameValues.SetProperty "Depth", "500" 'dimension#1

pDimNameValues.SetProperty "T1", "500" 'dimension#2

Set pWMSMapLayer2.DimensionValues = pDimNameValues

'calling a function to redraw the layer

RefreshActiveDataFrame

End Sub

ArcGIS Online: The Living Atlas

www.arcgis.com

The Living Atlas

• Imagery

• Basemaps

• Demographics & Lifestyle

• Boundaries & Places

• Landscape

• Community Maps

• Transportation

• Urban Systems

• Earth Observations

• Historic Maps

Services of Scientific Data

• MODIS data

- MODIS land cover 2000-2011

- MODIS Vegetation Analysis

- MODIS Greenland Sea Ice

• Live NOAA wind service

• NASA Global Land Data Assimilation (GLDS)

- Soil moisture

- Evapotranspiration

- Snow pack

• More

Online Imagery content that can be directly used:

SUBHEAD INFORMATION

Demo: Publishing

Consuming Scientific Data Services

• ArcGIS Desktop/Pro/Server

• Web Map Viewer

• Web Applications

• Story maps

• Operational Dashboard

Corrected Reflectance – Consumed in ArcGIS Desktop

WMS in Dapple Earth Explorer

Tell the story of your scientific data – Create Story Maps

http://dtc-sci01.esri.com/DeadZoneStoryMap/

Demo: Consumption