Post on 26-Dec-2015
Enterprise ETL & the Home Again Enterprise ETL & the Home Again Web Mapping Application: Columbus, OhioWeb Mapping Application: Columbus, Ohio
2009 Ohio GIS Conference
September 16-18, 2009
Crowne Plaza North Hotel
Columbus, Ohio
2009 Ohio GIS Conference
September 16-18, 2009
Crowne Plaza North Hotel
Columbus, Ohio
Erick LobaoGIS Project ManagerStantec
Robert ParsonsSoftware EngineerCity of Columbus,Dept. of Technology
Sonia KrammesGIS AnalystCity of Columbus,Dept. of Technology
Nick SoltesGIS AnalystStantec
Home Again• Home Again is a Public – Private partnership put forth
by the Mayor to revitalize areas of Columbus by purchasing, renovating, selling properties, and demolishing vacant homes.
• The Department of Development maintains data using several applications and creates reports from these data sources to manage the Home Again initiative.
ColumbusColumbus
Project Goals• Combine data stored in different file formats with data
models / table structures.
• Recreate and automate reports that were manually produced by a former employee.
• Display Home Again point information on a web map and provide a basic level of spatial analysis.
Workflow Diagram
Home AgainArcSDE
GeodatabaseDoT
Columbus Housing
Partnership(Excel
Spreadsheet)
Extract Transform & Load Process
(Store Data in One Table)
Affordable Housing Trust
(Excel Spreadsheet)
City Attorney’s Office
(Unknown)
Vacant Housing Database
(DoT – Oracle)
Join Process Attach Table Information to
Repository Address Points
ArcGIS Server Web Service
(Data Viewable in Web-based Application)
Land Bank Database
(Redevelopment – Approach)
Building Permits(Development -
Accela)
Data Sources
Description Owner Format Frequency
Vacant Housing Application (V.H.A.) Development / Code Enforcement Oracle Table Daily
Land Bank Land Bank Approach Table As Needed
Columbus Housing Partnership CHP Excel Worksheet Monthly
Affordable Housing Trust AHT Excel Worksheet Monthly
Building Permits, Private Demolition Permits
Department of Development Acella (Oracle Table) Daily
Public Demolitions Microsoft Access As Needed
RESPEC Department of Development Microsoft Access Daily
City Attorney's Foreclosure Information
City Attorney's Office Unknown As Needed
This is a Job for Model Builder!• Model Builder allows you to string together ArcGIS
processes to automate multi-step operations.
• Models can be exported to a script and scheduled to run as a scheduled task.
Model Builder scripts can be extended with Java, Python or VBA code if needed.
GIS Workflow
Home AgainArcSDE
GeodatabaseDoT
Columbus Housing
Partnership(Excel
Spreadsheet)
Extract Transform & Load Process
(Store Data in One Table)
Affordable Housing Trust
(Excel Spreadsheet)
City Attorney’s Office
(Unknown)
Vacant Housing Database
(DoT – Oracle)
Join Process Attach Table Information to
Repository Address Points
ArcGIS Server Web Service
(Data Viewable in Web-based Application)
Land Bank Database
(Redevelopment – Approach)
Building Permits(Development -
Accela)
Model goes here
And here
Why Model Builder?• Easy to use.
• Basic understanding of available geoprocessing tools.
• Programming language skills not required.• Future updates, revisions, changes can be made by GIS staff.
• Models are re-usable / speeds up repetitive tasks.
• Execute tasks in a defined order.
• Open a single file to perform a total of 31 geoprocessing operations.
Under The Hood• Models developed with one or two processes at a time.
• Allows you to debug / tune model properly
• Export, format, combine data from multiple sources.
• Join tables based on Parcel ID• Assign spatial information to non-spatial data
• Export to python script for scheduling processes using MS Windows Task Scheduler
Adding Location to Table• Issue: Data stored in a table without spatial reference.
• Address information and “Home Again” attributes stored in a table.
• Solution 1: Create an Oracle table view to join staging table to City’s Master Address Point Layer.• Provides a “Live-View” of final point layer.
• Significant amount of lag time when viewing this layer.
• Solution 2: Use model builder to extract Home Again Points utilizing the coordinates of the Address Point Layer.• Produces a “static” file each time model is executed.
Adding Location to Table
• Join staging table to address points based on Parcel ID.
• Update staging table with (X,Y) from address points.
• Create event feature from staging table.
Good… But Not Perfect• Adds Complexity.
• Model requires ArcMap to call ArcGIS functions.• Limits use to machines with ArcMap installed.
• ArcGIS file “locking issues” (Not an issue in Python)
• Some operations in Model Builder were buggy.
• Model Builder can do a lot, but at what point do you make the jump to something else?
ROI: Process• Who’s available to maintain and update ETL
procedures and what’s their skill level?• Leverage existing staff, software, and experience.
• Leverage existing software licenses (ESRI, Crystal Reports).
• Modular approach spreads responsibility.
• Time = $• Repeatable and Automated ETL
ROI: Home Again• Web Application provides GIS capabilities to non GIS
savvy users.• Allows GIS staff to work on higher level tasks.
• Provides spatial analysis capabilities to City leadership.
• Spatially enabling data.• Getting more use / functionality from existing data.
• Development staff can examine relationships between multiple data sources.
• Informs City leadership and Home Again program staff.
A More Permanent Solution• Streamline ETL procedures by improving data
capture and storage.
• Ex. use common field names & data types
• Helps with report generation and data aggregation.
• Web site with forms to enforce data standards.
• Provide flexible reporting.
Questions & Comments
Erick LobaoGIS Project ManagerStantecErick.Lobao@stantec.com(614) 844-4011
Robert ParsonsSoftware EngineerCity of Columbus,Dept. of TechnologyRGParsons@columbus.gov(614) 645-5711
Sonia KrammesGIS AnalystCity of Columbus,Dept. of TechnologySRKrammes@columbus.gov(614) 645-5392
Nick SoltesGIS AnalystStantecNSoltes@stantec.com(614) 486-4383