Data Integration for ArcGIS Users - Amazon S3 Integration for ArcGIS Users ... –Join attributes to...

Charmel Menzel, ESRICharmel Menzel, ESRIDon Murray, Safe SoftwareDon Murray, Safe Software

Data Integration for ArcGIS UsersData Integration for ArcGIS UsersData InteroperabilityData Interoperability

Product overviewProduct overview

•• Extension to ArcGIS (optional)Extension to ArcGIS (optional)

•• Jointly developed with Safe SoftwareJointly developed with Safe Software–– Based on Feature Manipulation EngineBased on Feature Manipulation Engine

(FME) technology(FME) technology–– Includes FME WorkbenchIncludes FME Workbench

AgendaAgenda

•• Desktop Product OverviewDesktop Product Overview

–– Format SupportFormat Support

–– Data TranslationData Translation

–– Spatial ETL (Extract, Transform and Load)Spatial ETL (Extract, Transform and Load)•• WorkbenchWorkbench

•• ArcGIS Engine and ArcGIS Server OverviewArcGIS Engine and ArcGIS Server Overview

•• QuestionsQuestions

Format SupportFormat Support

Added format support Added format support

•• Directly read vector data formatsDirectly read vector data formats––Support for 100+ data formatsSupport for 100+ data formats

•• GIS, CAD, databaseGIS, CAD, database––ReadRead--onlyonly

Added format support Added format support

•• Directly read vector data formatsDirectly read vector data formats––Support for 100+ data formatsSupport for 100+ data formats

•• GIS, CAD, databaseGIS, CAD, database––ReadRead--onlyonly

•• Use in ArcGIS Desktop applicationsUse in ArcGIS Desktop applications(ArcMap, ArcCatalog, ArcScene, etc.)(ArcMap, ArcCatalog, ArcScene, etc.)

Visualization

Mapping

Analysis

ArcCatalog format supportArcCatalog format support

•• Manage your dataManage your data––Browse treeBrowse tree


•• Manage your dataManage your data––Browse treeBrowse tree–– Interoperability ConnectionsInteroperability Connections


•• Manage your dataManage your data––Browse treeBrowse tree–– Interoperability ConnectionsInteroperability Connections

––View your dataView your dataGeographyGeographyFeature attributesFeature attributesMetadataMetadata

ArcMap format supportArcMap format support

•• Display data in native formatDisplay data in native format––No data conversionNo data conversion

•• Support for all standard map functionsSupport for all standard map functions

Selections

Labels

Identify

Symbolization

Format Support Format Support -- New at 9.3New at 9.3

•• Upgrade to FME 2008Upgrade to FME 2008

•• New formatsNew formats–– Industry Foundation Class STEP Files (IFC)Industry Foundation Class STEP Files (IFC)–– CityGMLCityGML–– LandXMLLandXML–– GeoRSSGeoRSS–– Trible JobXMLTrible JobXML–– Autodesk 2007 (DWG/DXF)Autodesk 2007 (DWG/DXF)–– Aeronautical Information Exchange Model (AIXM)Aeronautical Information Exchange Model (AIXM)

Updated FormatsUpdated Formats•• KML 2.1KML 2.1•• Autodesk AutoCAD Reader/Writer now supports Release 2007 Autodesk AutoCAD Reader/Writer now supports Release 2007

filesfiles. .

NEW

Data TranslationData Translation

Data translation toolsData translation tools

•• Data Interoperability ToolsData Interoperability Tools––Geoprocessing ToolboxGeoprocessing Toolbox––Quickly convert data to new formatQuickly convert data to new format

GMLXMLWFS

FormatSupport

OraclePostGreSQL

DB2

CAD

ShapefileGeodatabase

MapInfoGeoMedia



•• Quick Import Quick Import ––New file or personal geodatabaseNew file or personal geodatabase

GMLXMLWFS

FormatSupport

OraclePostGreSQL

DB2

CAD


MapInfoGeoMedia



•• Quick Import Quick Import ––New file or personal geodatabaseNew file or personal geodatabase

•• Quick Export (75+ formats)Quick Export (75+ formats)––Publish data to many formatsPublish data to many formats

GMLXMLWFS

FormatSupport

OraclePostGreSQL

DB2

CAD


MapInfoGeoMedia

Geoprocessing integrationGeoprocessing integration

•• Use data (formats) as input to core geoprocessing toolsUse data (formats) as input to core geoprocessing tools––Buffer, Clip, Union, Frequency, etc.Buffer, Clip, Union, Frequency, etc.––Outputs to ESRI formatOutputs to ESRI format

Build interoperability into modelsBuild interoperability into models

•• Combine tools in geoprocessing modelsCombine tools in geoprocessing models––Use interoperability data sources as input to the modelUse interoperability data sources as input to the model––Export and distribute results in desired formatExport and distribute results in desired format

An ESRI shapefile is generated from the buffer operation, which is then output to three different formats:GML, file geodatabase, and GeoMedia

This model uses a MapInfo MIF fileas input to a buffer operation.

New at 9.3New at 9.3Help / DocumentationHelp / Documentation

•• Help Topics updatedHelp Topics updated–– ReorganizedReorganized–– New contentNew content

NEW

DemonstrationDemonstration

Format SupportFormat SupportData TranslationData TranslationGeoprocessingGeoprocessing

Spatial ETLSpatial ETL(extract, transform, load)(extract, transform, load)

Spatial ETLSpatial ETL

•• ETL: Extract, transform, loadETL: Extract, transform, load––Extract data from a source systemExtract data from a source system––Transform the data to match the schema of the destination Transform the data to match the schema of the destination

systemsystem––Load the data into the destination systemLoad the data into the destination system

Spatial ETL Spatial ETL



•• Spatial ETLSpatial ETL––Extract, transform, and load Extract, transform, and load spatialspatial datadata––Data transformation, semantic data translationData transformation, semantic data translation

ETL

Spatial ETLSpatial ETL



•• Spatial ETLSpatial ETL––Extract, transform, and load Extract, transform, and load spatialspatial datadata––Data transformation, semantic data translationData transformation, semantic data translation

•• Data transformations may include changes toData transformations may include changes to––FormatFormat––Coordinate systemCoordinate system––Feature typesFeature types––Attribute schemaAttribute schema

Data Migration ExampleData Migration ExampleMigrate data to a common data modelMigrate data to a common data model

Spatial ETL

Nation GIS

Regional / CountyRegional / County

StateState

FederalFederal

Workbench applicationWorkbench application

•• Graphically create spatial ETL data flow from source to Graphically create spatial ETL data flow from source to destination destination

Transform or adjust the way your data flows from its source to the destination.

Extract your source dataset onto the workspace.

Load your data into the destination dataset.

Workbench transformersWorkbench transformers

•• Workbench transformers Workbench transformers ––225 + available transformers225 + available transformers––Use transformers to manipulate source dataUse transformers to manipulate source data

•• AttributesAttributes•• GeometryGeometry

Source data Transformer(s) Destination data


•• Attribute operationsAttribute operations––Join attributes to features Join attributes to features ––Map new attribute values Map new attribute values ––Create new attribute values Create new attribute values

•• Geometric operationsGeometric operations––Alter/build geometry Alter/build geometry ––Filter by geometry Filter by geometry ––Spatial relationships Spatial relationships

•• Select a subset of data to loadSelect a subset of data to load––RoadType = RoadType = ““PavedPaved””

––GeometryType = PolygonGeometryType = Polygon

AttributeFiltertransformer

GeometryFiltertransformer

Create, manipulate, and convert geometry and attributesCreate, manipulate, and convert geometry and attributes


•• SelectSelect a subset of data to loada subset of data to load––RoadType = RoadType = ““PavedPaved””


•• TranslateTranslate coded values coded values 00 NonNon--ResidentialResidential

11 ResidentialResidential

NN NonNon--ResidentialResidential

RR ResidentialResidentialValueMappertransformer

Workbench transformersWorkbench transformersCreate, manipulate, and convert geometry and attributesCreate, manipulate, and convert geometry and attributes

•• SelectSelect a subset of data to loada subset of data to load––RoadType = RoadType = ““PavedPaved””


•• TranslateTranslate coded values coded values

•• DeriveDerive new attribute values or construct geometrynew attribute values or construct geometry––Points geometries to linear featuresPoints geometries to linear features

PointConnectortransformer


•• JoinJoin together data from multiple sourcestogether data from multiple sources

11 SalesSales

22 ProductsProducts

JeffJeff 11

PhilPhil 22 JeffJeff 11 SalesSales

PhilPhil 22 ProductsProductsJoiner

transformer



•• Summarize/concatenateSummarize/concatenate multiple rows of data multiple rows of data 11 ABAB

22 ACAC

1AB1AB

2AC2ACConcatenator

transformer



•• Summarize/concatenateSummarize/concatenate multiple rows of data multiple rows of data

•• SplitSplit a single attribute into multiple attributesa single attribute into multiple attributes

ESRI, 380 New York St, Redlands, CAESRI, 380 New York St, Redlands, CA

ESRIESRI 380 New York St.380 New York St. RedlandsRedlands CACA

AttributeSplittertransformer


ArcGIS integration of spatial ETL ArcGIS integration of spatial ETL •• Spatial ETL tool Spatial ETL tool

––Data is convertedData is converted––Custom geoprocessing toolCustom geoprocessing tool––Define your output schema Define your output schema

once and use many timesonce and use many times

ArcGIS integration of spatial ETL ArcGIS integration of spatial ETL •• Spatial ETL tool Spatial ETL tool

––Data is convertedData is converted––Custom geoprocessing toolCustom geoprocessing tool––Define your output schema Define your output schema

once and use many timesonce and use many times

•• Custom formatsCustom formats––Dynamic data transformationDynamic data transformation

•• No data conversionNo data conversion•• Custom view of the data Custom view of the data

––Define once and use many timesDefine once and use many times

Workbench and ModelBuilderWorkbench and ModelBuilder––Graphical authoring environmentsGraphical authoring environments–– WorkbenchWorkbench processes data at the feature levelprocesses data at the feature level–– ModelBuilder ModelBuilder processes data at the dataset or layer levelprocesses data at the dataset or layer level

WorkbenchWorkbench ModelBuilderModelBuilder

ModelBuilder and WorkbenchModelBuilder and Workbench––Graphical authoring environmentsGraphical authoring environments–– Workbench processes data at the feature levelWorkbench processes data at the feature level–– ModelBuilder processes data at the dataset or layer levelModelBuilder processes data at the dataset or layer level

•• Complementary technologiesComplementary technologies–– Workbench creates processes that can run in ModelBuilderWorkbench creates processes that can run in ModelBuilder

•• Use Spatial ETL tools in models, scripting, and command line Use Spatial ETL tools in models, scripting, and command line

Spatial ETL tasksSpatial ETL tasks

•• Data migrationData migration––Process of moving data between systemsProcess of moving data between systems

•• Data cleanup/data manipulationData cleanup/data manipulation––Clean up errors in the data and streamline data preparationClean up errors in the data and streamline data preparation

•• Data distributionData distribution––Distribute data to different systemsDistribute data to different systems

•• Change detectionChange detection––Determine changes between two different filesDetermine changes between two different files

•• Data validationData validation––Verify and validate spatial dataVerify and validate spatial data

New at 9.3New at 9.3TransformersTransformers

•• ArcMeasureExtractorArcMeasureExtractor•• ArcMeasureSetterArcMeasureSetter•• ArcPropertyExtractorArcPropertyExtractor•• ArcPropertySetterArcPropertySetter•• ArcSDEGridSnapperArcSDEGridSnapper•• AttributeExploderAttributeExploder•• CoordinateSystemCoordinateSystem

DescriptionConverterDescriptionConverter•• CreatorCreator•• EllipsePropertyEllipseProperty

ExtractorExtractor•• EllipsePropertySetterEllipsePropertySetter•• FeatureHolderFeatureHolder•• FMEFunctionCallerFMEFunctionCaller

•• PathSplitterPathSplitter•• PointMeasureExtractorPointMeasureExtractor•• PointMeasureSetterPointMeasureSetter•• PythonCreatorPythonCreator•• SummaryReporterSummaryReporter•• TextAdderTextAdder•• TextLocationExtractorTextLocationExtractor•• TextPropertyExtractorTextPropertyExtractor•• TextPropertySetterTextPropertySetter•• URLFetcherURLFetcher•• VariableSetterVariableSetter•• VariableRetrieverVariableRetriever

•• GeometryRefinerGeometryRefiner•• GeometryReplacerGeometryReplacer•• GeometryValidatorGeometryValidator•• GeoRSSFeatureExtractorGeoRSSFeatureExtractor•• GeoRSSFeatureReplacerGeoRSSFeatureReplacer•• LineMeasureExtractorLineMeasureExtractor•• LineMeasureSetterLineMeasureSetter•• MinimumSpanning MinimumSpanning

CircleReplacerCircleReplacer•• NeighborColorSetterNeighborColorSetter•• OrientationExtractorOrientationExtractor•• ParameterFetcherParameterFetcher•• PartCounterPartCounter

•• 35+ additional transformers35+ additional transformers

NEW

New at 9.3New at 9.3Workbench UpdatesWorkbench Updates

•• Overview WindowOverview Window

•• Floating WindowsFloating Windows

•• Customizable Transformer GalleryCustomizable Transformer Gallery–– (i.e. My Favorites GP toolbox)(i.e. My Favorites GP toolbox)

NEW

New at 9.3 New at 9.3 Feature Types and GeometryFeature Types and Geometry

•• Select Feature Types Dialog Select Feature Types Dialog –– choose layers to processchoose layers to process

•• Rich Geometry Model Rich Geometry Model –– complex geometrycomplex geometry––Advanced Geometries Advanced Geometries

•• paths (measures)paths (measures)•• bulged polylinesbulged polylines

NEW

New at 9.3New at 9.3Transformer and Stream InteractionTransformer and Stream Interaction

•• Incomplete TransformersIncomplete Transformers

•• Disconnecting Streams / Isolating Transformers Disconnecting Streams / Isolating Transformers –– testingtesting

NEW

New at 9.3New at 9.3Update Feature TypesUpdate Feature Types

•• Feature Type ManipulationFeature Type Manipulation–– Updating Feature Types for Source and Destination datasets Updating Feature Types for Source and Destination datasets

when schema has changed.when schema has changed.

NEW

New at 9.3New at 9.3Schema Mapping TransformersSchema Mapping Transformers

•• Replace manual field maps with AttributeCopierReplace manual field maps with AttributeCopier–– Persist throughout workspacePersist throughout workspace

NEW

DemonstrationDemonstration

WorkbenchWorkbenchSpatial ETLSpatial ETL

ArcGIS Engine and ArcGIS Server ArcGIS Engine and ArcGIS Server OverviewOverview

ArcGIS Engine and ArcGIS Server SupportArcGIS Engine and ArcGIS Server Support

ExtensionsExtensions

•• Data Interoperability ArcGIS Engine extensionData Interoperability ArcGIS Engine extension–– DataData–– Geoprocessing toolsGeoprocessing tools

•• Data Interoperability ArcGIS Server extensionData Interoperability ArcGIS Server extension–– Map ServicesMap Services–– Geoprocessing ServicesGeoprocessing Services

Enables extensionEnables extension’’s data handling and support s data handling and support capabilities to be embedded in custom desktop or capabilities to be embedded in custom desktop or serverserver--based applications.based applications.

ArcGIS Engine extensionArcGIS Engine extensionProvides custom desktop applicationsProvides custom desktop applications……

•• Access to Data Interoperability data sourcesAccess to Data Interoperability data sources

•• Ability to run Data InteroperabilityAbility to run Data Interoperability’’s geoprocessing s geoprocessing tools in custom applicationstools in custom applications

–– Quick Import, Quick ExportQuick Import, Quick Export–– Spatial ETLSpatial ETL

ArcGIS Server Workflow BasicsArcGIS Server Workflow Basics

Share geographic content:Share geographic content:

1. 1. AuthorAuthor the GIS resource the GIS resource using ArcGIS Desktopusing ArcGIS Desktop

2. 2. PublishPublish the resource as a the resource as a service using ArcGIS Serverservice using ArcGIS Server

3. 3. UseUse the service from a client the service from a client applicationapplication

–– (Desktop, ArcGIS Explorer, Custom Web App)(Desktop, ArcGIS Explorer, Custom Web App)

UseUse

PublishPublish

AuthorAuthor

Web MapWeb MapViewerViewer

DesktopDesktop ExplorerExplorer

GIS ServerGIS Server

MobileMobile OpenOpenAPIsAPIs

Enterprise Service Bus

Data Interoperability ArcGIS ServerData Interoperability ArcGIS ServerThe power of Desktop Data Interoperability on the ServerThe power of Desktop Data Interoperability on the Server

•• Map ServicesMap Services–– Publish maps that contain nonPublish maps that contain non--ESRI formats to ArcGIS Server ESRI formats to ArcGIS Server

•• Geoprocessing ServicesGeoprocessing Services–– Publish models that contain Data InteroperabilityPublish models that contain Data Interoperability’’s Data Integration s Data Integration

tools to ArcGIS Servertools to ArcGIS Server

** ** All resources authored using ArcGIS DesktopAll resources authored using ArcGIS Desktop

MapServices

ArcGISServer

GeoprocessingServices

ArcGISServer

DemonstrationDemonstrationPublishing Data Interoperability resources to ArcGIS ServerPublishing Data Interoperability resources to ArcGIS Server

Quick Export Geoprocessing ModelQuick Export Geoprocessing Model

Future Development PlansFuture Development Plans•• Upgrade released software to latest FME platform (FME2009)Upgrade released software to latest FME platform (FME2009)•• New Vector Formats:New Vector Formats:

–– ADAC XML ReaderADAC XML Reader–– Adobe PDF 2D WriterAdobe PDF 2D Writer–– AutoDesk 3DS WriterAutoDesk 3DS Writer–– CityGML WriterCityGML Writer–– IBM Informix ReaderIBM Informix Reader & Writer& Writer–– IBM Informix Spatial Reader & WriterIBM Informix Spatial Reader & Writer–– MS SQL Server Spatial Reader & WriterMS SQL Server Spatial Reader & Writer–– OpenStreetMap ReaderOpenStreetMap Reader–– Wavefront OBJ Reader & WriterWavefront OBJ Reader & Writer

•• Vector Format Updates:Vector Format Updates:–– Improved Geodatabase annotation support.Improved Geodatabase annotation support.

ResourcesResources

•• ArcGIS Data InteroperabilityArcGIS Data Interoperability–– Including web demosIncluding web demoshttp://www.esri.com/datainteroperabilityhttp://www.esri.com/datainteroperability

•• GIS Standards and Interoperability GIS Standards and Interoperability http://www.esri.com/interoperabilityhttp://www.esri.com/interoperability

•• FME Workbench Introductory TrainingFME Workbench Introductory Trainingwww.safe.com/esriwww.safe.com/esri

•• ArcGIS Data Interoperability TutorialArcGIS Data Interoperability Tutorialhttp://webhelp.esri.com/arcgisdesktop/9.3/http://webhelp.esri.com/arcgisdesktop/9.3/

Data Interoperability SummaryData Interoperability Summary

1.1.Incorporates multiple data formats from open or Incorporates multiple data formats from open or proprietary sources directly into your GIS proprietary sources directly into your GIS

2.2.Maintains data quality (spatial and attribute Maintains data quality (spatial and attribute information) as you convert between different data information) as you convert between different data models.models.

3.3.Migrates data from other systems/data models or Migrates data from other systems/data models or maintain multiple systems.maintain multiple systems.

4.4.Integrates into the ArcGIS geoprocessing frameworkIntegrates into the ArcGIS geoprocessing framework

Session Evaluations ReminderSession Evaluations Reminder

Please turn in your session evaluations.Please turn in your session evaluations.

Questions??Questions??

. . . Thank you. . . Thank you

Data Integration for ArcGIS Users - Amazon S3 Integration for ArcGIS Users ... –Join attributes to...

Documents

Transcript of Data Integration for ArcGIS Users - Amazon S3 Integration for ArcGIS Users ... –Join attributes to...