Data Integration for ArcGIS Users...ArcGIS Server Workflow Basics Share geographic content: 1....
Transcript of Data Integration for ArcGIS Users...ArcGIS Server Workflow Basics Share geographic content: 1....
Charmel Menzel, ESRICharmel Menzel, ESRIDon Murray, Safe SoftwareDon Murray, Safe Software
Data Integration for ArcGIS UsersData Integration for ArcGIS UsersData InteroperabilityData Interoperability
Product overviewProduct overview
•• Extension to ArcGIS (optional)Extension to ArcGIS (optional)
•• Jointly developed with Safe SoftwareJointly developed with Safe Software–– Based on Feature Manipulation EngineBased on Feature Manipulation Engine
(FME) technology(FME) technology–– Includes FME WorkbenchIncludes FME Workbench
AgendaAgenda
•• Desktop Product OverviewDesktop Product Overview
–– Format SupportFormat Support
–– Data TranslationData Translation
–– Spatial ETL (Extract, Transform and Load)Spatial ETL (Extract, Transform and Load)•• WorkbenchWorkbench
•• ArcGIS Engine and ArcGIS Server OverviewArcGIS Engine and ArcGIS Server Overview
•• QuestionsQuestions
Format SupportFormat Support
Added format support Added format support
•• Directly read vector data formatsDirectly read vector data formats––Support for 100+ data formatsSupport for 100+ data formats
•• GIS, CAD, databaseGIS, CAD, database––ReadRead--onlyonly
Added format support Added format support
•• Directly read vector data formatsDirectly read vector data formats––Support for 100+ data formatsSupport for 100+ data formats
•• GIS, CAD, databaseGIS, CAD, database––ReadRead--onlyonly
•• Use in ArcGIS Desktop applicationsUse in ArcGIS Desktop applications(ArcMap, ArcCatalog, ArcScene, etc.)(ArcMap, ArcCatalog, ArcScene, etc.)
Visualization
Mapping
Analysis
ArcCatalog format supportArcCatalog format support
•• Manage your dataManage your data––Browse treeBrowse tree
ArcCatalog format supportArcCatalog format support
•• Manage your dataManage your data––Browse treeBrowse tree–– Interoperability ConnectionsInteroperability Connections
ArcCatalog format supportArcCatalog format support
•• Manage your dataManage your data––Browse treeBrowse tree–– Interoperability ConnectionsInteroperability Connections
––View your dataView your dataGeographyGeographyFeature attributesFeature attributesMetadataMetadata
ArcMap format supportArcMap format support
•• Display data in native formatDisplay data in native format––No data conversionNo data conversion
•• Support for all standard map functionsSupport for all standard map functions
Selections
Labels
Identify
Symbolization
Format Support Format Support -- New at 9.3New at 9.3
•• Upgrade to FME 2008Upgrade to FME 2008
•• New formatsNew formats–– Industry Foundation Class STEP Files (IFC)Industry Foundation Class STEP Files (IFC)–– CityGMLCityGML–– LandXMLLandXML–– GeoRSSGeoRSS–– Trible JobXMLTrible JobXML–– Autodesk 2007 (DWG/DXF)Autodesk 2007 (DWG/DXF)–– Aeronautical Information Exchange Model (AIXM)Aeronautical Information Exchange Model (AIXM)
Updated FormatsUpdated Formats•• KML 2.1KML 2.1•• Autodesk AutoCAD Reader/Writer now supports Release 2007 Autodesk AutoCAD Reader/Writer now supports Release 2007
filesfiles. .
NEW
Data TranslationData Translation
Data translation toolsData translation tools
•• Data Interoperability ToolsData Interoperability Tools––Geoprocessing ToolboxGeoprocessing Toolbox––Quickly convert data to new formatQuickly convert data to new format
GMLXMLWFS
FormatSupport
OraclePostGreSQL
DB2
CAD
ShapefileGeodatabase
MapInfoGeoMedia
Data translation toolsData translation tools
•• Data Interoperability ToolsData Interoperability Tools––Geoprocessing ToolboxGeoprocessing Toolbox––Quickly convert data to new formatQuickly convert data to new format
•• Quick Import Quick Import ––New file or personal geodatabaseNew file or personal geodatabase
GMLXMLWFS
FormatSupport
OraclePostGreSQL
DB2
CAD
ShapefileGeodatabase
MapInfoGeoMedia
Data translation toolsData translation tools
•• Data Interoperability ToolsData Interoperability Tools––Geoprocessing ToolboxGeoprocessing Toolbox––Quickly convert data to new formatQuickly convert data to new format
•• Quick Import Quick Import ––New file or personal geodatabaseNew file or personal geodatabase
•• Quick Export (75+ formats)Quick Export (75+ formats)––Publish data to many formatsPublish data to many formats
GMLXMLWFS
FormatSupport
OraclePostGreSQL
DB2
CAD
ShapefileGeodatabase
MapInfoGeoMedia
Geoprocessing integrationGeoprocessing integration
•• Use data (formats) as input to core geoprocessing toolsUse data (formats) as input to core geoprocessing tools––Buffer, Clip, Union, Frequency, etc.Buffer, Clip, Union, Frequency, etc.––Outputs to ESRI formatOutputs to ESRI format
Build interoperability into modelsBuild interoperability into models
•• Combine tools in geoprocessing modelsCombine tools in geoprocessing models––Use interoperability data sources as input to the modelUse interoperability data sources as input to the model––Export and distribute results in desired formatExport and distribute results in desired format
An ESRI shapefile is generated from the buffer operation, which is then output to three different formats:GML, file geodatabase, and GeoMedia
This model uses a MapInfo MIF fileas input to a buffer operation.
New at 9.3New at 9.3Help / DocumentationHelp / Documentation
•• Help Topics updatedHelp Topics updated–– ReorganizedReorganized–– New contentNew content
NEW
DemonstrationDemonstration
Format SupportFormat SupportData TranslationData TranslationGeoprocessingGeoprocessing
Spatial ETLSpatial ETL(extract, transform, load)(extract, transform, load)
Spatial ETLSpatial ETL
•• ETL: Extract, transform, loadETL: Extract, transform, load––Extract data from a source systemExtract data from a source system––Transform the data to match the schema of the destination Transform the data to match the schema of the destination
systemsystem––Load the data into the destination systemLoad the data into the destination system
Spatial ETL Spatial ETL
•• ETL: Extract, transform, loadETL: Extract, transform, load––Extract data from a source systemExtract data from a source system––Transform the data to match the schema of the destination Transform the data to match the schema of the destination
systemsystem––Load the data into the destination systemLoad the data into the destination system
•• Spatial ETLSpatial ETL––Extract, transform, and load Extract, transform, and load spatialspatial datadata––Data transformation, semantic data translationData transformation, semantic data translation
ETL
Spatial ETLSpatial ETL
•• ETL: Extract, transform, loadETL: Extract, transform, load––Extract data from a source systemExtract data from a source system––Transform the data to match the schema of the destination Transform the data to match the schema of the destination
systemsystem––Load the data into the destination systemLoad the data into the destination system
•• Spatial ETLSpatial ETL––Extract, transform, and load Extract, transform, and load spatialspatial datadata––Data transformation, semantic data translationData transformation, semantic data translation
•• Data transformations may include changes toData transformations may include changes to––FormatFormat––Coordinate systemCoordinate system––Feature typesFeature types––Attribute schemaAttribute schema
Data Migration ExampleData Migration ExampleMigrate data to a common data modelMigrate data to a common data model
Spatial ETL
Nation GIS
Regional / CountyRegional / County
StateState
FederalFederal
Workbench applicationWorkbench application
•• Graphically create spatial ETL data flow from source to Graphically create spatial ETL data flow from source to destination destination
Transform or adjust the way your data flows from its source to the destination.
Extract your source dataset onto the workspace.
Load your data into the destination dataset.
Workbench transformersWorkbench transformers
•• Workbench transformers Workbench transformers ––225 + available transformers225 + available transformers––Use transformers to manipulate source dataUse transformers to manipulate source data
•• AttributesAttributes•• GeometryGeometry
Source data Transformer(s) Destination data
Workbench transformersWorkbench transformers
•• Attribute operationsAttribute operations––Join attributes to features Join attributes to features ––Map new attribute values Map new attribute values ––Create new attribute values Create new attribute values
•• Geometric operationsGeometric operations––Alter/build geometry Alter/build geometry ––Filter by geometry Filter by geometry ––Spatial relationships Spatial relationships
•• Select a subset of data to loadSelect a subset of data to load––RoadType = RoadType = ““PavedPaved””
––GeometryType = PolygonGeometryType = Polygon
AttributeFiltertransformer
GeometryFiltertransformer
Create, manipulate, and convert geometry and attributesCreate, manipulate, and convert geometry and attributes
Workbench transformersWorkbench transformers
•• SelectSelect a subset of data to loada subset of data to load––RoadType = RoadType = ““PavedPaved””
––GeometryType = PolygonGeometryType = Polygon
•• TranslateTranslate coded values coded values 00 NonNon--ResidentialResidential
11 ResidentialResidential
NN NonNon--ResidentialResidential
RR ResidentialResidentialValueMappertransformer
Workbench transformersWorkbench transformersCreate, manipulate, and convert geometry and attributesCreate, manipulate, and convert geometry and attributes
•• SelectSelect a subset of data to loada subset of data to load––RoadType = RoadType = ““PavedPaved””
––GeometryType = PolygonGeometryType = Polygon
•• TranslateTranslate coded values coded values
•• DeriveDerive new attribute values or construct geometrynew attribute values or construct geometry––Points geometries to linear featuresPoints geometries to linear features
PointConnectortransformer
Workbench transformersWorkbench transformersCreate, manipulate, and convert geometry and attributesCreate, manipulate, and convert geometry and attributes
•• JoinJoin together data from multiple sourcestogether data from multiple sources
11 SalesSales
22 ProductsProducts
JeffJeff 11
PhilPhil 22 JeffJeff 11 SalesSales
PhilPhil 22 ProductsProductsJoiner
transformer
Workbench transformersWorkbench transformersCreate, manipulate, and convert geometry and attributesCreate, manipulate, and convert geometry and attributes
•• JoinJoin together data from multiple sourcestogether data from multiple sources
•• Summarize/concatenateSummarize/concatenate multiple rows of data multiple rows of data 11 ABAB
22 ACAC
1AB1AB
2AC2ACConcatenator
transformer
Workbench transformersWorkbench transformersCreate, manipulate, and convert geometry and attributesCreate, manipulate, and convert geometry and attributes
•• JoinJoin together data from multiple sourcestogether data from multiple sources
•• Summarize/concatenateSummarize/concatenate multiple rows of data multiple rows of data
•• SplitSplit a single attribute into multiple attributesa single attribute into multiple attributes
ESRI, 380 New York St, Redlands, CAESRI, 380 New York St, Redlands, CA
ESRIESRI 380 New York St.380 New York St. RedlandsRedlands CACA
AttributeSplittertransformer
Workbench transformersWorkbench transformersCreate, manipulate, and convert geometry and attributesCreate, manipulate, and convert geometry and attributes
ArcGIS integration of spatial ETL ArcGIS integration of spatial ETL •• Spatial ETL tool Spatial ETL tool
––Data is convertedData is converted––Custom geoprocessing toolCustom geoprocessing tool––Define your output schema Define your output schema
once and use many timesonce and use many times
ArcGIS integration of spatial ETL ArcGIS integration of spatial ETL •• Spatial ETL tool Spatial ETL tool
––Data is convertedData is converted––Custom geoprocessing toolCustom geoprocessing tool––Define your output schema Define your output schema
once and use many timesonce and use many times
•• Custom formatsCustom formats––Dynamic data transformationDynamic data transformation
•• No data conversionNo data conversion•• Custom view of the data Custom view of the data
––Define once and use many timesDefine once and use many times
Workbench and ModelBuilderWorkbench and ModelBuilder––Graphical authoring environmentsGraphical authoring environments–– WorkbenchWorkbench processes data at the feature levelprocesses data at the feature level–– ModelBuilder ModelBuilder processes data at the dataset or layer levelprocesses data at the dataset or layer level
WorkbenchWorkbench ModelBuilderModelBuilder
ModelBuilder and WorkbenchModelBuilder and Workbench––Graphical authoring environmentsGraphical authoring environments–– Workbench processes data at the feature levelWorkbench processes data at the feature level–– ModelBuilder processes data at the dataset or layer levelModelBuilder processes data at the dataset or layer level
•• Complementary technologiesComplementary technologies–– Workbench creates processes that can run in ModelBuilderWorkbench creates processes that can run in ModelBuilder
•• Use Spatial ETL tools in models, scripting, and command line Use Spatial ETL tools in models, scripting, and command line
Spatial ETL tasksSpatial ETL tasks
•• Data migrationData migration––Process of moving data between systemsProcess of moving data between systems
•• Data cleanup/data manipulationData cleanup/data manipulation––Clean up errors in the data and streamline data preparationClean up errors in the data and streamline data preparation
•• Data distributionData distribution––Distribute data to different systemsDistribute data to different systems
•• Change detectionChange detection––Determine changes between two different filesDetermine changes between two different files
•• Data validationData validation––Verify and validate spatial dataVerify and validate spatial data
New at 9.3New at 9.3TransformersTransformers
•• ArcMeasureExtractorArcMeasureExtractor•• ArcMeasureSetterArcMeasureSetter•• ArcPropertyExtractorArcPropertyExtractor•• ArcPropertySetterArcPropertySetter•• ArcSDEGridSnapperArcSDEGridSnapper•• AttributeExploderAttributeExploder•• CoordinateSystemCoordinateSystem
DescriptionConverterDescriptionConverter•• CreatorCreator•• EllipsePropertyEllipseProperty
ExtractorExtractor•• EllipsePropertySetterEllipsePropertySetter•• FeatureHolderFeatureHolder•• FMEFunctionCallerFMEFunctionCaller
•• PathSplitterPathSplitter•• PointMeasureExtractorPointMeasureExtractor•• PointMeasureSetterPointMeasureSetter•• PythonCreatorPythonCreator•• SummaryReporterSummaryReporter•• TextAdderTextAdder•• TextLocationExtractorTextLocationExtractor•• TextPropertyExtractorTextPropertyExtractor•• TextPropertySetterTextPropertySetter•• URLFetcherURLFetcher•• VariableSetterVariableSetter•• VariableRetrieverVariableRetriever
•• GeometryRefinerGeometryRefiner•• GeometryReplacerGeometryReplacer•• GeometryValidatorGeometryValidator•• GeoRSSFeatureExtractorGeoRSSFeatureExtractor•• GeoRSSFeatureReplacerGeoRSSFeatureReplacer•• LineMeasureExtractorLineMeasureExtractor•• LineMeasureSetterLineMeasureSetter•• MinimumSpanning MinimumSpanning
CircleReplacerCircleReplacer•• NeighborColorSetterNeighborColorSetter•• OrientationExtractorOrientationExtractor•• ParameterFetcherParameterFetcher•• PartCounterPartCounter
•• 35+ additional transformers35+ additional transformers
NEW
New at 9.3New at 9.3Workbench UpdatesWorkbench Updates
•• Overview WindowOverview Window
•• Floating WindowsFloating Windows
•• Customizable Transformer GalleryCustomizable Transformer Gallery–– (i.e. My Favorites GP toolbox)(i.e. My Favorites GP toolbox)
NEW
New at 9.3 New at 9.3 Feature Types and GeometryFeature Types and Geometry
•• Select Feature Types Dialog Select Feature Types Dialog –– choose layers to processchoose layers to process
•• Rich Geometry Model Rich Geometry Model –– complex geometrycomplex geometry––Advanced Geometries Advanced Geometries
•• paths (measures)paths (measures)•• bulged polylinesbulged polylines
NEW
New at 9.3New at 9.3Transformer and Stream InteractionTransformer and Stream Interaction
•• Incomplete TransformersIncomplete Transformers
•• Disconnecting Streams / Isolating Transformers Disconnecting Streams / Isolating Transformers –– testingtesting
NEW
New at 9.3New at 9.3Update Feature TypesUpdate Feature Types
•• Feature Type ManipulationFeature Type Manipulation–– Updating Feature Types for Source and Destination datasets Updating Feature Types for Source and Destination datasets
when schema has changed.when schema has changed.
NEW
New at 9.3New at 9.3Schema Mapping TransformersSchema Mapping Transformers
•• Replace manual field maps with AttributeCopierReplace manual field maps with AttributeCopier–– Persist throughout workspacePersist throughout workspace
NEW
DemonstrationDemonstration
WorkbenchWorkbenchSpatial ETLSpatial ETL
ArcGIS Engine and ArcGIS Server ArcGIS Engine and ArcGIS Server OverviewOverview
ArcGIS Engine and ArcGIS Server SupportArcGIS Engine and ArcGIS Server Support
ExtensionsExtensions
•• Data Interoperability ArcGIS Engine extensionData Interoperability ArcGIS Engine extension–– DataData–– Geoprocessing toolsGeoprocessing tools
•• Data Interoperability ArcGIS Server extensionData Interoperability ArcGIS Server extension–– Map ServicesMap Services–– Geoprocessing ServicesGeoprocessing Services
Enables extensionEnables extension’’s data handling and support s data handling and support capabilities to be embedded in custom desktop or capabilities to be embedded in custom desktop or serverserver--based applications.based applications.
ArcGIS Engine extensionArcGIS Engine extensionProvides custom desktop applicationsProvides custom desktop applications……
•• Access to Data Interoperability data sourcesAccess to Data Interoperability data sources
•• Ability to run Data InteroperabilityAbility to run Data Interoperability’’s geoprocessing s geoprocessing tools in custom applicationstools in custom applications
–– Quick Import, Quick ExportQuick Import, Quick Export–– Spatial ETLSpatial ETL
ArcGIS Server Workflow BasicsArcGIS Server Workflow Basics
Share geographic content:Share geographic content:
1. 1. AuthorAuthor the GIS resource the GIS resource using ArcGIS Desktopusing ArcGIS Desktop
2. 2. PublishPublish the resource as a the resource as a service using ArcGIS Serverservice using ArcGIS Server
3. 3. UseUse the service from a client the service from a client applicationapplication
–– (Desktop, ArcGIS Explorer, Custom Web App)(Desktop, ArcGIS Explorer, Custom Web App)
UseUse
PublishPublish
AuthorAuthor
Web MapWeb MapViewerViewer
DesktopDesktop ExplorerExplorer
GIS ServerGIS Server
MobileMobile OpenOpenAPIsAPIs
Enterprise Service Bus
Data Interoperability ArcGIS ServerData Interoperability ArcGIS ServerThe power of Desktop Data Interoperability on the ServerThe power of Desktop Data Interoperability on the Server
•• Map ServicesMap Services–– Publish maps that contain nonPublish maps that contain non--ESRI formats to ArcGIS Server ESRI formats to ArcGIS Server
•• Geoprocessing ServicesGeoprocessing Services–– Publish models that contain Data InteroperabilityPublish models that contain Data Interoperability’’s Data Integration s Data Integration
tools to ArcGIS Servertools to ArcGIS Server
** ** All resources authored using ArcGIS DesktopAll resources authored using ArcGIS Desktop
MapServices
ArcGISServer
GeoprocessingServices
ArcGISServer
DemonstrationDemonstrationPublishing Data Interoperability resources to ArcGIS ServerPublishing Data Interoperability resources to ArcGIS Server
Quick Export Geoprocessing ModelQuick Export Geoprocessing Model
Future Development PlansFuture Development Plans•• Upgrade released software to latest FME platform (FME2009)Upgrade released software to latest FME platform (FME2009)•• New Vector Formats:New Vector Formats:
–– ADAC XML ReaderADAC XML Reader–– Adobe PDF 2D WriterAdobe PDF 2D Writer–– AutoDesk 3DS WriterAutoDesk 3DS Writer–– CityGML WriterCityGML Writer–– IBM Informix ReaderIBM Informix Reader & Writer& Writer–– IBM Informix Spatial Reader & WriterIBM Informix Spatial Reader & Writer–– MS SQL Server Spatial Reader & WriterMS SQL Server Spatial Reader & Writer–– OpenStreetMap ReaderOpenStreetMap Reader–– Wavefront OBJ Reader & WriterWavefront OBJ Reader & Writer
•• Vector Format Updates:Vector Format Updates:–– Improved Geodatabase annotation support.Improved Geodatabase annotation support.
ResourcesResources
•• ArcGIS Data InteroperabilityArcGIS Data Interoperability–– Including web demosIncluding web demoshttp://www.esri.com/datainteroperabilityhttp://www.esri.com/datainteroperability
•• GIS Standards and Interoperability GIS Standards and Interoperability http://www.esri.com/interoperabilityhttp://www.esri.com/interoperability
•• FME Workbench Introductory TrainingFME Workbench Introductory Trainingwww.safe.com/esriwww.safe.com/esri
•• ArcGIS Data Interoperability TutorialArcGIS Data Interoperability Tutorialhttp://webhelp.esri.com/arcgisdesktop/9.3/http://webhelp.esri.com/arcgisdesktop/9.3/
Data Interoperability SummaryData Interoperability Summary
1.1.Incorporates multiple data formats from open or Incorporates multiple data formats from open or proprietary sources directly into your GIS proprietary sources directly into your GIS
2.2.Maintains data quality (spatial and attribute Maintains data quality (spatial and attribute information) as you convert between different data information) as you convert between different data models.models.
3.3.Migrates data from other systems/data models or Migrates data from other systems/data models or maintain multiple systems.maintain multiple systems.
4.4.Integrates into the ArcGIS geoprocessing frameworkIntegrates into the ArcGIS geoprocessing framework
Session Evaluations ReminderSession Evaluations Reminder
Please turn in your session evaluations.Please turn in your session evaluations.
Questions??Questions??
. . . Thank you. . . Thank you