for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason...
Transcript of for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason...
![Page 1: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/1.jpg)
Overview for the 2007 ESRI User ConferenceOverview for the 2007 ESRI User Conference2121‐‐JunJun‐‐2007 2007
Jason Roberts, Ben Best, Daniel Dunn, and Pat HalpinJason Roberts, Ben Best, Daniel Dunn, and Pat HalpinDuke University Marine Geospatial Ecology LabDuke University Marine Geospatial Ecology Lab
![Page 2: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/2.jpg)
Marine Geospatial Ecology ToolsGeoprocessing toolbox for marine ecology
Oceanographic data management and analysisSophisticated sampling and statistical modelingEmphasis on batch processing and interoperability
Open source, implemented mostly in PythonTools are platform independent, when possibleSome tools do not even require ArcGIS
Minimum requirements: Python 2 4 ArcGIS
![Page 3: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/3.jpg)
Talk OutlineHistory of MGETWalkthrough typical user scenarioHighlight interesting tools and features
Invitation to collaborateAdvanced topics: (time permitting)Ben’s Connectivity ModelerHow to build an MGET tool
![Page 4: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/4.jpg)
History of MGETWe have produced many geoprocessing tools but have done a poor job sharing themStaff developed tools independentlyTools shared ad hoc with collaboratorsLittle effort to package and document tools for easy re‐use by anonymous users
It is time to unify our efforts!
![Page 5: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/5.jpg)
ArcRStats by Ben Best
Toolbox for sampling raster layers and running
statistical analyses to predict habitats
RandomPoints pts_rand
rstr_aspect
rstr_dem
rstr_landcov
rstr_tci
StatisticalPlots dir_plots
Sample toTable tbl_env
pts_obs
MultivariateRegression,
GLM
rstr_glm
rstr_glmroc
<> Lakes
rstr_viable
![Page 6: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/6.jpg)
Marine Ecology Tools by Jason Roberts
Unreleased toolbox for batch processing of oceanography
Aviso Geo stro phicCurre nt u ComponentNe tCDFs to Ra ste rs
Raste rFo lde r (2)
Ne tCDFs
Raste rFo lde r
Aviso Geo stro phicCurre nt v ComponentNe tCDFs to Ra ste rs
Raste rFo lde r (3)
Ve cto r Sha pe filesfrom XY Component
Ra ste rs
Shap e fileFo ld e r
![Page 7: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/7.jpg)
Benthic Complexity Modeler by Daniel Dunn
Predicts hardbottom
from coarse‐grain (90 m) bathymetryGEODAS
bathymetry
Substrate Type ~ depth difference + aspect variety + slope + error
Statistical model: GLM with logit link
Probability of hardbottom
![Page 8: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/8.jpg)
Unifying Our Tools
ArcRStats, HabMod, ConnMod
Marine Ecology ToolsBenthic Complexity ModelerE. Treml’s
Connectivity Tools
And others
MGET
![Page 9: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/9.jpg)
Typical User ScenarioResearcher has spatially‐explicit observations of a species and wants to investigate why it is there
Are there spatial and temporal patterns?Correlations with environmental conditions?Correlations with occurrence of other species?Can we predict its occurrence and thereby improve our management of it?
![Page 10: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/10.jpg)
Typical Observation Data
IATTCOlive Ridley Encounters1990-2005
Fishery bycatch data
Surveys
Argos satellite tracks
Scott Eckert
![Page 11: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/11.jpg)
Typical Workflow
Import species observations into GIS
Download oceanographic datasets
Prepare oceanographic data for use
Create derived oceanographic datasetsSample
oceanographic data
Explore maps of oceano. and observations
MGET includes (or will include) tools
that assist with all steps
Model species habitat or behavior
![Page 12: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/12.jpg)
Typical Workflow
Import species observations into GIS
Download oceanographic datasets
Prepare oceanographic data for use
Create derived oceanographic datasetsSample
oceanographic data
Explore maps of oceano. and observationsModel species
habitat or behavior
![Page 13: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/13.jpg)
Import Argos Results Into Tables
37120 Date : 12.12.03 09:58:10 LC : 2 IQ : 50 Lat1 : 34.282N Lon1 : 77.010W Lat2 : 32.640N Lon2 : 69.121W Nb mes : 012 Nb mes>-120dB : 000 Best level : -121 dB Pass duration : 706s NOPC : 2 Calcul freq : 401 647695.6 Hz Altitude : 0 m
49 54 00 0000 00 218 17 96 96 29 159
255 94 98
DIAG record
11373 37120 9 31 J 2003-12-03 09:49:11 1 00 00 00 00
00 106 40 116 02 230 97 145
211 214 72 44 158 139 118 119 193 80 00 00 00 00 00 00 00 00 00
DS record
![Page 14: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/14.jpg)
Import Argos Results Into Tables
DS Folder
DIAG Folder
ResultTable
Message Table
![Page 15: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/15.jpg)
Table Points Lines
Input Table Point Features
Line Features
XY Table to Point Feature Class Connect Points
![Page 16: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/16.jpg)
Typical Workflow
Import species observations into GIS
Download oceanographic datasets
Prepare oceanographic data for use
Create derived oceanographic datasetsSample
oceanographic data
Explore maps of oceano. and observationsModel species
habitat or behavior
![Page 17: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/17.jpg)
Calculate URLs and Download
NODC AVHRR v5URL fo r Da te sLis te d in T a b le
Ob se rva tio ns.shp
Ob se rva tio ns.shp (2)
Do wnlo a d File sLis te d in T a b le
Ob se rva tio ns.shp (3)
Do wnlo a dFo ld e r (2)
Do wnlo a dFo ld e r
1.
Given points’
dates, calculates URL to NOAA SST HDFs
2.
Given URLs, download the files to a given directory
![Page 18: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/18.jpg)
Calculate URLs and DownloadNODC AVHRR v5 URLs for Dates Download Files
![Page 19: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/19.jpg)
Typical Workflow
Import species observations into GIS
Download oceanographic datasets
Prepare oceanographic data for use
Create derived oceanographic datasetsSample
oceanographic data
Explore maps of oceano. and observationsModel species
habitat or behavior
![Page 20: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/20.jpg)
Preparing Oceanography For UseMost oceanographic datasets are not immediately usableCommon preprocessing steps include:Converting to a supported formatProjecting to a desired projectionClipping to region of interestPerforming arbitrary map algebraBuilding pyramids
![Page 21: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/21.jpg)
MGET Tools for OceanographyImplemented in three layers:
1. Single‐input, single output for general format
HDF to ArcGIS Raster
2. Batch processing versions for general format
HDFs Listed in Table to ArcGIS RastersFind HDFs and Convert to ArcGIS Rasters
3. Specialized versions for particular products
NODC AVHRR v5 HDF to ArcGIS RasterNODC AVHRR v5 HDFs Listed in Table to RastersFind NODC AVHRR v5 HDFs and Convert to Rasters
![Page 22: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/22.jpg)
Example: HDF to ArcGIS Raster
![Page 23: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/23.jpg)
Batch Processing Design Pattern 1“Process inputs listed in table”
pattern:
Table fields contain the paths to the inputs to process and the outputs to produceUser can populate these columns using any technique (e.g. Download Files tool)The batch tool accepts a SQL where clause to select the rows to process, and an order by clause to specify the
i d
![Page 24: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/24.jpg)
Example: SDSes
in HDFs
Listed in Table to ArcGIS Rasters
Same as single- file tool
![Page 25: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/25.jpg)
Batch Processing Design Pattern 2“Find and process inputs”
pattern:
User specifies:Input and output locations (e.g. workspaces)Optional search parameters (e.g. wildcard)Python expression for naming outputs (a sensible default is always provided)
The batch tool searches the input location, processes all inputs that are found and stores them in the output
![Page 26: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/26.jpg)
Example: SDSes
in HDFs
Listed in Table to ArcGIS Rasters
Same as single- file tool
![Page 27: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/27.jpg)
Example: SDSes
in HDFs
Listed in Table to ArcGIS Rasters
![Page 28: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/28.jpg)
Example Product‐Specific ToolNOAA NODC 4km AVHRR Pathfinder v5 SST
![Page 29: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/29.jpg)
Other SST ProductsPO.DAAC GOES 10/12 NOAA CoastWatch
AVHRR
Also: PO.DAAC MODIS Aqua and Terra, GOES 9
![Page 30: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/30.jpg)
Sea Surface ChlorophyllNASA OceanColor
Group SeaWiFS
Also: MODIS Aqua and combined MODIS/SeaWiFS
![Page 31: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/31.jpg)
AVISO SSH and Geostrophic Currents
![Page 32: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/32.jpg)
PO.DAAC QuickSCAT
Winds
Katrina28-Aug-2005 Also: BYU
QuickSCAT Sigma‐0
(approximate s sea surface rougness)
![Page 33: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/33.jpg)
Global Bathymetries
ETOPO2GEBCOS2004
Map shows S2004 clipped
to ETP
![Page 34: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/34.jpg)
Typical Workflow
Import species observations into GIS
Download oceanographic datasets
Prepare oceanographic data for use
Create derived oceanographic datasetsSample
oceanographic data
Explore maps of oceano. and observationsModel species
habitat or behavior
![Page 35: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/35.jpg)
~120 km
AVHRR Daytime SST 03-Jan-2005
28.0 °C
25.8 °C
Cayula-Cornillion Edge Detection Algorithm (1992)
Identifying Surface Temperature Fronts
Fre
qu
en
cy
Temperature
Optimal break 27.0 °C
Strong cohesion front present
Step 1: Histogram analysis Step 2: Spatial cohesion test
Weak cohesion no front
Bimodal
Example Output
Processed all daily AVHRR 4 km images from 1985‐2005
Over 15,000 images, requiring 2 months of computer time
Also processed GOES 10 and 12 images
Mexico
Mexico
Front
![Page 36: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/36.jpg)
Typical Workflow
Import species observations into GIS
Download oceanographic datasets
Prepare oceanographic data for use
Create derived oceanographic datasetsSample
oceanographic data
Model species habitat or behavior
Explore maps of oceano. and observations
![Page 37: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/37.jpg)
Batch Sampling Tool
Sample rasters in 1 or more fieldsStores values directly in fields!Can apply Python expression to sampled values
![Page 38: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/38.jpg)
Typical Workflow
Import species observations into GIS
Download oceanographic datasets
Prepare oceanographic data for use
Create derived oceanographic datasetsSample
oceanographic data
Model species habitat or behavior
Explore maps of oceano. and observations
![Page 39: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/39.jpg)
Invoking R from ArcGIS
![Page 40: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/40.jpg)
Invoking R from ArcGIS
R messages logged to ArcGIS
Value of last statement returned
![Page 41: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/41.jpg)
MGET Project StatusVersion 0.2 just released“Framework” nearly complete, but only simple building‐block tools are implementedHDF to raster converter might interest you
Oceanographic processing tools will be released in JulyBen and Jason will integrate ArcRStats
![Page 42: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/42.jpg)
Installing MGETUser executes installation program
Registers ArcGIS toolbox
Installs Python package Configures Start menu
Registers COM objects
![Page 43: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/43.jpg)
ArcGIS documentationIn HTML files
In Arc toolbox
![Page 44: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/44.jpg)
Python documentationIn HTML,
formatted like Python library documentatio
n
![Page 45: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/45.jpg)
MGET includes extensive validation and logging; log levels are configurable
![Page 46: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/46.jpg)
Invitation to CollaborateDo you need a specific tool developed for your project?We would consider developing it for MGET, especially if it would be widely applicable
Do you develop tools yourself?Become a contributor/coauthor! We could help you integrate your tools into MGET.
![Page 47: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/47.jpg)
Thanks! Any questions?
Download site:http://code.env.duke.edu/projects/mget
Contact: Jason Roberts [email protected]
![Page 48: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/48.jpg)
Advanced Modeling: Ben’s Connectivity Modeler
CreateNetwork
tin
pt_nodes
ln_edges
poly_patches_
pt_centroids
txt_network
poly_patches
rstr_cost
NetworkLeast Cost
Path
txt_networkl
ln_edgeslc
NetworkCentrality
Metrics
poly_patchsm
Best et al 2007 (In Review)
![Page 49: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/49.jpg)
The Connectivity ProblemSay you have a set of patches and a cost surface that describes migration costHow to efficiently compute how “connected” patches are to each other?
![Page 50: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/50.jpg)
Step 1: create a network from the cost surface
CreateNetwork
tin
pt_nodes
ln_edges
poly_patches_
pt_centroids
txt_network
poly_patches
rstr_cost
NetworkLeast Cost
Path
txt_networkl
ln_edgeslc
NetworkCentrality
Metrics
poly_patchsm
![Page 51: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/51.jpg)
Cost surface is converted to a TIN to create the network
![Page 52: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/52.jpg)
Step 2: calculate the least cost paths for the network
CreateNetwork
tin
pt_nodes
ln_edges
poly_patches_
pt_centroids
txt_network
poly_patches
rstr_cost
NetworkLeast Cost
Path
txt_networkl
ln_edgeslc
NetworkCentrality
Metrics
poly_patchsm
![Page 53: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/53.jpg)
Network least cost pathsDjikstraalgorithm highly efficient over ArcGIS CostPathfunctionFuture: create corridors with CostDistancefrom paths
![Page 54: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/54.jpg)
Step 3: compute network centrality metrics as indices of connectivity
CreateNetwork
tin
pt_nodes
ln_edges
poly_patches_
pt_centroids
txt_network
poly_patches
rstr_cost
NetworkLeast Cost
Path
txt_networkl
ln_edgeslc
NetworkCentrality
Metrics
poly_patchsm
![Page 55: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/55.jpg)
Network centrality metrics
Brandes, 2000. “Faster Evaluation of Shortest‐Path Based Centrality Indices.”
CiteSeer.
Degree Closeness Betweenness
![Page 56: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/56.jpg)
Developer walk‐through:
How do you develop an MGET tool?
![Page 57: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/57.jpg)
Goals for a common development framework for Duke’s tools
Let developers select the best technologies for the jobRequire tools to formally declare their dependenciesEncourage devs to choose set of standard technologies
Automate tedious stuff, such as:Tool dependency and input parameter validationInteroperability plumbing (Arc toolboxes, COM objects)Generation of installation packagesGeneration of documentation
Provide a library of common utility functions:Invoking Arc/R/MATLAB, manipulating files and data,
h dli l i l li i
![Page 58: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/58.jpg)
Marine Geospatial Ecology ToolsCore framework implemented in Python and C++
Python was selected due to wide appeal to Duke developers and perceived strategic importance to ESRI
Core framework is platform independentIndividual tools determine their own dependencies
Tools may be implemented in any languageBut language interoperability is expensive to developCurrently planning for Python, R and MATLAB tools
Provides all features from previous slide, and more!
![Page 59: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/59.jpg)
Creating a “Hello, World”
tool in Python
You implement a Python‐based “tool”
in MGET by creating a Python instance method or
classmethod:1.
Create the module and class that will receive the new method (or locate an existing module and
class)2.
Define the method’s name and input parameters
3.
Fill in the method’s body (i.e., write the code)4.
Specify some metadata about the method
5.
Run a script that rebuilds the MGET installation package
![Page 60: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/60.jpg)
from
GeoEco.DynamicDocString
import
DynamicDocStringfrom
GeoEco.Internationalization
import
_from
GeoEco.Logging
import Logger
class
Example(object):__doc__ = DynamicDocString()
Import needed modules provided by the core MGET framework
GeoEco is the name of the MGET Python package; I chose this name after Ben expressed a desire that the package name not imply that the tools are not only for marine problems
Class definitionHack to allow metadata to be added to class
![Page 61: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/61.jpg)
from
GeoEco.DynamicDocString
import
DynamicDocStringfrom
GeoEco.Internationalization
import
_from
GeoEco.Logging
import Logger
class
Example(object):__doc__ = DynamicDocString()
@classmethoddef
GreetPerson(cls, personName):pass
Method definition
Method body, to be filled in as next step
Input parameter
![Page 62: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/62.jpg)
from
GeoEco.DynamicDocString
import
DynamicDocStringfrom
GeoEco.Internationalization
import
_from
GeoEco.Logging
import Logger
class
Example(object):__doc__ = DynamicDocString()
@classmethoddef
GreetPerson(cls, personName):cls.__doc__.obj.ValidateMethodInvocation()message = _(u’Hello, %s!’) % personNameLogger.Info(message)return
message
Call to function that validates the tool’s dependencies and input parameter values
The _ function and use of Unicode strings enable localization of MGET tools
Call to logging system
Output parameter
![Page 63: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/63.jpg)
from
GeoEco.Metadata
import
*from
GeoEco.Types
import
*
AddMethodMetadata(Example.GreetPerson,shortDescription=_(u’Greets
a specified person’),isExposedToPythonCallers=True,isExposedByCOM=True,isExposedAsArcGISTool=True,arcGISDisplayName=_(u’Greet
Person’),arcGISToolCategory=_(u’Example
Tools’))
AddArgumentMetadata(Example.GreetPerson, u’personName’,typeMetadata=UnicodeStringTypeMetadata(canBeNone=False),description=_(u’The
person to greet with a friendly message.’),arcGISDisplayName=_(u’Person
to greet’))
Configure interoperability
Specify appearance of ArcGIS toolbox
Specify strong parameter type and validation options
Write documentation in reStructuredText
![Page 64: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/64.jpg)
![Page 65: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/65.jpg)
The build scriptUsing the tool metadata as input, the script generates:ArcGIS toolbox (Marine Geospatial Ecology Tools.tbx)Python wrapper scripts for invoking tools from toolboxMicrosoft COM type library and registration scripts, so tools can be invoked as COM objectsPython reference documentation (HTML)ArcGIS geoprocessing documentation (HTML)COM documentation (HTML, not implemented yet)Installation package (GeoEco-1.0.win32-py25.exe)
![Page 66: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/66.jpg)
Invoking your tool from Python
Initialize logging (optional)
Import the module, invoke methodLog message (format is configurable)Print returned value
Python callers can import your module and invoke your method directly:
The core framework is platform-independent; it is up to you to determine what platforms your method supports.
![Page 67: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/67.jpg)
Invoking your tool from ArcGIS
Documentation appears in geoprocessing UI
Fancy formatting is supported (e.g. bullets, hyperlinks, indentation, code, images)
Log messages appear in progress window
![Page 68: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/68.jpg)
Invoking your tool through COM Automation
Set logger = WScript.CreateObject("GeoEco.Logger")logger.InitializeSet example = WScript.CreateObject("GeoEco.Example")greeting = example.GreetPerson("Joe")WScript.StdOut.WriteLine(greeting)
Set logger = WScript.CreateObject("GeoEco.Logger")logger.InitializeSet example = WScript.CreateObject("GeoEco.Example")greeting = example.GreetPerson("Joe")WScript.StdOut.WriteLine(greeting)
VBScript example (many other languages supported)
![Page 69: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/69.jpg)
Invoking your tool from .Net
Add a reference to the GeoEco Type Library
Invocation occurs through early-bound (“vtable”) COM, not COM Automation
C# Example
IntelliSenseis fully functional!
![Page 70: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/70.jpg)
Logging with Debug messages disabled
Core framework only reports one message (the other is the “Hello, Joe!” greeting).
![Page 71: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/71.jpg)
Logging with Debug messages enabled
![Page 72: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/72.jpg)
Example: Declaring dependencies and calling ArcGIS geoprocessor
functions
@classmethoddef
CreateZeroRaster(cls, outputRaster):cls.__doc__.Obj.ValidateMethodInvocation()gp
= GeoprocessorManager.GetWrappedGeoprocessor()gp.CreateConstantRaster_sa(outputRaster, 0.0, 'FLOAT', 1.0, '0.0 0.0 1.0 1.0')
AddMethodMetadata(Example.CreateZeroRaster,shortDescription=_(u'Creates
a raster with all cells set to zero.'),isExposedToPythonCallers=True,isExposedByCOM=True,isExposedAsArcGISTool=True,arcGISDisplayName=_(u'Create
Zero Raster'),arcGISToolCategory=_(u'Examples'),dependencies=[ArcGISDependency(9, 1),
ArcGISExtensionDependency(u'spatial')])ArcGIS dependencies
Wrapped geoprocessor object logs all calls
Dependencies checked and geoprocessor initialized here
![Page 73: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/73.jpg)
Resulting output (debug logging enabled)
Otherdependency checks
LoggedArcGISinteractions
Initialize geoprocessor
![Page 74: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/74.jpg)
Dependencies implemented so farWindowsDependency – minimum Windows versionPythonDependency – minimum Python versionPythonModuleDependency –module is installedArcGISDependency – minimum ArcGIS versionArcGISProductDependency – minimum product levelArcGISExtensionDependency – extension is availableRDependency – minimum R versionRPackageDependency – package is installed, minimum version optional
![Page 75: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/75.jpg)
Example of dependency failureThe Scenario:The user invokes the Evaluate R
Statements
tool but the required
rpy Python module is not installed
rpy is not installed and the framework raises a SoftwareNotInstalled error
![Page 76: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/76.jpg)
ArcGIS documentationHTML pages generated from XML metadata using XSL transforms
Documentation also added to Arc toolbox by build script
![Page 77: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/77.jpg)
Python documentation
HTML pages generated from XML metadata using XSL transforms
![Page 78: for the ESRI User Conference · Overview for the 2007 ESRI User Conference 21‐Jun‐2007 Jason Roberts, Ben Best, Daniel Dunn, and Pat Halpin Duke University Marine Geospatial Ecology](https://reader030.fdocuments.us/reader030/viewer/2022011819/5e9421b401cd90491f691f4f/html5/thumbnails/78.jpg)
Using metdata
to generate batch‐ processing versions of your method
from
GeoEco.BatchProcessing
import
BatchProcessingfrom
GeoEco.DataManagement.Fields
import
Field
BatchProcessing.GenerateForMethod(HDF.ExtractHeader,inputParamNames=[u'inputFile'],inputParamFieldArcGISDisplayNames=[u'Input
HDF file field'],inputParamDescriptions=[u'%s
paths of the input HDF files.'],outputParamNames=[u'outputFile'],outputParamFieldArcGISDisplayNames=[u'Output
text file field'],outputParamExpressionArcGISDisplayNames=[u'Output
file Python expression'],outputParamDescriptions=[u'%s
paths of the text files to write.'],…processListMethodName=u‘ExtractHeaderList',processTableMethodName=u‘ExtractHeaderTable',…
)
Batch processing methods to generate