ESRI Quality Solution · •Cartographic Representations avoid data copies •Versioning to protect...

ESRI Quality Solution

Paul HardyBusiness Development Consultant (NMCAs)

ESRI Europe

[email protected]

mailto:[email protected]�

ESRI Quality Solution• Geodatabase Integrity

–Schema constraints (Attribute Domains etc)–Rich data types (cartographic representations, networks, etc)–Geoprocessing tools (Check Feature etc)–SDE Load Checks

• Topology–Topology Rules–Topology error handling

• PLTS – Production Line Tool Set–Real-time attribute validation (domains, CNT, VVTs)–GIS Data ReViewer

• Metadata (& standards conformance)– ISO 19115 (19113, 19114, 19138, …)

• ArcObjects API–For tailored and bespoke solutions

ESRI - EuroGeographics Quality Workshop, EG Q-KEN, Hamar, 2009-05

We share your vision: Collect once, use many times


Maps/Charts• Hardcopy• PDF

Data Products• Imagery• Grids/Terrains• Vector Data

Web Portal Services

Geodatabase

Database driven mapping

Organising, Managing, Securing …….

Networks

Surveys

Addresses

Annotation

Vectors

3D Objects

Topology

Dimensions

Images

Terrain

Attributes

Cartography

Cadastral

Schematics

ArcGIS

Geodatabase

Comprehensive Information Model(Any Type of Data)

• Any DBMS• Files

Scalable Storage Environments


Collect Manage

ProduceShare

Geodatabase

GIS Mapping Data FlowsField Survey & GPS

Map Scan & VectorisePhotogrammetry

CAD dataEdit & update

Quality & IntegrityStructure & Topology

Versions & TransactionsHistory & Archiving

Multiple representations

Analysis & ReportsVisualization & Modelling

GeneralizationCartographic design

Map authoringRepresentation overrides

Paper mapsDownloadable maps (PDF)

Web mapsRaster data productsVector data products

Metadata, Portals, SDIs

Standards

Quality

Requirements Workflow


6

DLMMedium

DeriveFeatures

DLMCoarse

DeriveFeatures

1:200K Atlas

1:250K SheetDerive RepsAddRepresentations

to featuresDerive Reps

1:25K Topo

1:50K TopoDerive RepsAddRepresentations

to featuresDerive Reps

Derive RepsAddRepresentations

to Features Derive Reps

DLM Fine(Master)

Landscape Model

Multi-scale, Multi-product Database …Cartographic

Generalization

1:15K Street

1:10K Topo

VisualProduct

Cartographic Model

1 to 1 relationship

Model Generalization

n to

m re

latio

nshi

p


GEODATABASE INTEGRITY


Database Schema & Quality

• ArcCatalog is main tool• Tools to help create and then load UML models• Feature Datasets, Feature Classes, Subtypes• Attribute domains ensure clean attribution• Cartographic Representations avoid data copies• Versioning to protect integrity of long transactions• Database archiving to record change through time• Database replication for reliable distributed DB


Data Loading and Checking

• ArcGIS reading of shapefiles etc is tolerant of errors–Null geometry–Self intersections– Invalid holes

• So two geoprocessing tools to detect and repair–Check_Features–Repair_Features

• When data stored in SDE geodatabase – less tolerant–SDE will clean geometry to canonical forms

• Geoprocessing framework and model builder–400+ other geoprocessing tools

• Data Interoperability Extension–70+ formats read–Safe Software FME and workbench


TOPOLOGY


Topology in maps

• Describes exact spatial relationships of map features– Adjacency– Coincidence– Connectivity– Containment– Others …

• GIS uses– Analysis– Data integrity


Topology in the geodatabase

• Helps ensure spatial integrity of data– Finds errors in data – Fix with edit and topology tools

• Relationships not stored– Discovered on-the-fly by ArcMap

• Coincidence-based topology– Snaps feature vertices– Between one or two feature classes

• Projection independent relationships

Monuments

Lot lines

Parcels

Zoning


Coincident geometry

• Each feature stored as a complete feature

5 features7 coordinate pairs

X: 100.50Y: 200.50

122.50200.50

144.50200.50

1

23 4 5

Polygons2 features

8 coordinate pairs

X: 100.50Y: 200.50

100.50220.00

1

2

Lines

Points


Real-world examples

• Single feature class examples– Postal code boundaries must not overlap– Sewer lines must not have dangles– Vegetation polygons must not have gaps

• Two feature class examples– Parcel boundary must be covered by parcel lines– Woodlands must not overlap with water– City points must be properly inside of countries

Rule: Soil polygons must not overlap

Find error

Fix error


Geodatabase topology

• Created in feature dataset• User specifies properties• Topology properties

– Cluster tolerance – Participating feature classes– Ranks – Topology rules

• Discovers and enforces topological relationships– Snaps feature vertices during validation

• Edited in ArcMap


Ranks

• Control how vertices move during validation• Lower-ranked vertices move to higher-ranked vertices

– Ranks: 1 (highest) to 50 (lowest)– Set ranks between feature classes (not subtypes or features)

Endpoints move to average location

Equal ranks

H = Highest rank L = Lowest rank

L line moves to H line

Unequal ranks

Cluster tolerance

H

LL H H

L

Which data should be ranked

higher?

= feature class 1= feature class 2old location

A ESRI - EuroGeographics Quality Workshop, EG Q-KEN, Hamar, 2009-05

Topology rules

• Define valid spatial relationships– Between feature classes or subtypes

• 25 rules available– Many rules for each type of feature (point, line, polygon)

• Set when creating topology; tested during validation

“Parcels … boundary must be covered by … Parcel Lines”Topology rule

FC 1 FC 2

“Roads … must not overlap with … Railroads”

FC 1 FC 2Topology rule


Topology Rules


Dirty areas

• Areas in the dataset that have not been validated– Entire extent is “dirty” when topology is first created– May contain existing or undiscovered errors– Edits create dirty areas

Insert vertex

Before Edit session

Dirty area created


Validating a topology

• Checks dirty areas for errors• Snaps coordinates using cluster tolerance and ranks• ArcCatalog validates entire topology

– Cannot undo

• ArcMap has additional options– Can undo

• No new features created

Validating in ArcMap

Validating in ArcCatalog


Topology errors

• Found during validation– Rule violations: Dangles, overlaps, etc.– If there are no rules, there can be no errors

• Stored in topology as specialized geometries– Use to locate errors in ArcMap

• Error properties– Which rule was violated?– Which features caused

the error?– Where the error is located?


Managing errors

• Errors cannot be deleted but may be:– Resolved by editing features that break rules– Marked as an exception and stored in the topology– Ignored (data can still be used)

• Integrate error management into your workflow– Who is responsible for finding and fixing errors?– When are they fixed?– Who performs final quality assurance tests?


Topology workflow

• User may stop at any point in the process

Create topology

Dirtyareas

Fixerrors

Errors

Exceptions

Makeexception

Validatetopology

Editfeatures

ArcCatalog ArcMap

Cluster toleranceRanksRules

Validate

Data clean


TOPOLOGY DEMO


PLTS


What is PLTS for ArcGIS? – and Why?

• ArcGIS extension• Database-centric production environment

– For high-volume database and map production

• Promotes efficient end-to-end production workflow– Avoid ‘stovepipes’, or repeat work

• Higher quality DB and maps at lower production costs• Create maps that meet set specifications• Provides a Knowledge Base of Map Specifications

– Database Models– Validation Rules– Configuration Rules– Visualization & Symbolization rules


PLTS for ArcGIS

PLTS Foundation

Defense Solution Mapping Agency

Solution

Nautical Solution

Custom Solutions

Enhanced EditingTools

GIS Data ReViewer(also sold stand-alone) Knowledge base

MPS(also sold stand-alone)

JTX(also sold stand-alone)

ComplexDomains

PMC

AeronauticalSolution

DataLoader

IntelligenceSolution


PLTS Foundation adds to core ArcMap

• Enhanced use of the ArcMap TOC– PLTS Selection tab– PLTS Target tab – easy attribution

• Metadata editing• On the fly error verification

– Run-time– Knowledge Base Tables

• Condition tables (SQL)• Valid value tables



PLTS Editing Suite

• TOC Manager– Quickly and efficiently control TOC

• Editing Environment– Controls Edit Session and

Selection Environment

• Editing Tools– Single-click access to edit tasks– Context tools

Map Production System-Atlas

• High-quality/high-volume production–Sheet series–Atlas and map book creation–Batch printing and export

• Visual Specifications and Product Views–New in 9.3–Uses cartographic Representations

• Dynamic layouts–Graphic table elements in layouts–Automated element placement rules –Grid Manager to create standard grids and graticules


JTX Job Tracking Extension

• Workflow Management–Standardize consistent workflows

• Job/Task Management–Track progress and status

• Geodata Management–Simplify administration

• User Management – Improve productivity

• Centralized Information• Decision Support

…solution for managing enterprise geospatial workflows


GIS DATA REVIEWER



• System to process and assess spatial data quality– Find

• Automated checks• Visual checks

– Correct • PLTS Edit tools

– Verify• Identify corrections

GIS Data Reviewer

GIS Data ReViewer

Components of the GIS Data ReViewer

• Main toolbar–Manage ReViewer sessions–Perform data review

• Database environment to manage the review–ReViewer Table


The ReViewer Table

• Heart of the review process• Tracks all anomalies and errors• Records correction and verification status• Contains tools to help manage the records


Automated Data Review

Running Data Checks

• Select a batch check• Run the desired test• Browse results or write to table


A


Committing to the ReViewer Table

• Two options–Browse features–Write directly to the error table

• Browse feature–Visit each record one at a time–Choose to write all or write

individual


ReViewer Checks

Download

• ReViewer Checks Poster (pdf file)

• www.esri.com/plts–Literature link


http://www.esri.com/plts�

DATA REVIEWER DEMO


Visual Review tools

• Manual data review• From a selection set

–Browse through features using dialog box

• Commit features to ReViewer Table

• Single features–Select feature –Commit to ReViewer Table button

• Missing Features–Flag Missing Feature–Notepad Sketch Tools

• Gridding for sampling


• Batch Jobs:– Saved groups of checks– Allow execution of multiple

checks – Allow for consistent validation

from known source– Designed once and executed

many times– Easy to disseminate to

production team

• Design Batch Jobs in Batch Job Manager

Batch Job Manager


More on the ReViewer Table…


Toggle Table Edit Mode

Symbolize Reviewer Records

Grid Selection

ReViewer Table Properties

Statistics

Duplicate Records

Line Notepad

Point Notepad

Polygon Notepad

Grouping Columns

Browse Through ReViewer table

RecordsA

METADATA


ArcCatalog - Metadata


ISO 19115 Metadata

• Supported in current versions–As alternative to FGDC or ESRI styles–But limited panels

• Full support planned from ArcGIS 9.4


TAILORED QA


ESRI Switzerland Data Quality Assurance (QA) Framework and Tools

• ArcObjects-based data QA system–Fast

• Optimized for large data volumes

–Flexible • Configurable in custom Oracle data dictionary (not in versioned GDB schema)

– Easier to adjust/update on production system• Does not require geodatabase topologies or geometric networks…

– …but can use if they exist

–Test types• Geometric• Attribute• Topological• Network connectivity• Graphic overlap


QA Toolbar

Choose/Customize Specification

Verify Box, Selection, …

Verification ResultsNavigate Errors

Error Selection Tool

Navigate Allowed Errors


Test Conditions

• Example „Major Street Intersects Polygon Water Body“

• A generic test („Intersects Other“) configured to specific data („Streets“ and „Water Bodies“ feature classes)


Quality Specification

• List of Test Conditions• Example „Integration“• Condition properties

–Hard or Soft–Stop on error or Continue

• Admin tool (not shown)–GUI to build a specification–XML import/export


Verification Results

• Saved to Oracle data dictionary

–Summary– Individual errors


Navigator Window

• Geo-navigate through errors

• Shortcut tools to help fix errors

• Example–„Trail intersects lake“


Summary

• ESRI has a wide range of tools and facilities for Quality• Prime tool is GIS Data ReViewer• Don’t forget others:

–PLTS Foundation–Topology–Metadata–Data load checks and SDE

• And can build bespoke tools using ArcObjects toolkit


Questions and Comments?

What Quality tools do you want?

[email protected]

ESRI Quality Solution · •Cartographic Representations avoid data copies •Versioning to protect...

Documents

Transcript of ESRI Quality Solution · •Cartographic Representations avoid data copies •Versioning to protect...