Using FME to Convert TIGER Spatial Data From Oracle Spatial to ESRI Shapefiles

Mei-Leng Freeman, Tim Puzycki, Jay E. Spurlin Spatial Products Software Branch Geography Division, Bureau of the Census

2

Introduction

•  Public products – TIGER/Line Shapefiles – Partnership Program Shapefiles

•  Internal products – Data Access and Dissemination Staff

(DADS) shapefiles for use in the American Fact Finder websites (internal and external)

– Live Individual Shapefile Request and Delivery System (LISRDS)

– Quality Control (QC) shapefile for various projects

3

Shapefile Creation: Legacy Approach

•  Legacy TIGER – Utilized for 1990 and 2000 Decennial Censuses –  In-house database format – Binary files – Partitioned by county

•  Legacy Shapefile Generation (shpGen) –  Java shapefile classes – Perl driver scripts – Directly read from a TIGER partition – Legacy TIGER C API – Directly wrote to a shapefile – ASCII Job configuration files (user defined tags)

4

Modernized Approach

•  MAF/TIGER – The Legacy Master Address File (MAF) and

Topologically Integrated Geographic Encoding and Reference (TIGER) datasets were combined.

– Oracle Spatial and Oracle Topology Data Model v10.2

– Seamless, national data set •  Current shpGen

– Perl driver scripts – Pre-generated FME mapping files (.FME) – FME Engine handles reading from Oracle Spatial

and writing to ESRI shapefiles

5

Initial Process

•  Considerations for automation – Automate everything possible. – Use COTS software, wherever possible. – Ensure consistency of data after conversion. – Provide ease of maintenance. – Make source code reusable. – Fairly stable source data structure. – Fairly stable target shapefile formats.

6

FME Mapping File Usage

•  Manually create FME mapping files (.FME) – Previous FME mapping files used as

templates. – A standardized set of macros is used.

•  The FME mapping files are released as part of the software installation.

•  Automated production control system drives the process and provides any dynamic inputs.

7

Oracle Wallet Support

•  Stores database credentials: – Username – Password – Database

•  Avoids providing Username and Password to command line interface

8

Example FME Mapping File

9

Example FME Mapping File

Manually Created FME Mapping Files

•  TIGER/Line Shapefiles – Produced from “Benchmarks” (snapshots) of

the MAF/TIGER Database. – Public TIGER/Line shapefiles are posted on

the world wide web. •  Partnership Shapefiles

– MAF/TIGER Partnership Software (MTPS)

•  LISRDS Shapefiles – On-demand access to MAF/TIGER data – Data source may be “live” or a snapshot

10

Current process

•  A wider set of shapefile requirements – Mapping performance issues

•  Chained line representation of area boundaries • Un-duplication of lines in partitioned layers

– Visual representation issues •  Partial features •  ‘Balance of’ features

– Connect with Oracle Wallet

11

FME Mapping Files from FME Workbench

•  DADS/AFF Shapefiles – Both .FME and .FMW files are archived with

the source code. – The .FME files are released to production,

and used by FME Engine. – A similar procedure (though not yet as

automated, and not making use of the geoprocessing tools) is used by another branch for the generalized and projected shapefiles.

– Samples on the following slides:

12

13

14

15

16

17

18

19

20

Example: DADS/AFF Line Shapefiles

21

State 09

State 05

State 02

State 13

Example: DADS/AFF Line Shapefiles

22

State 09

State 05

State 02

State 13

23

24

Minnesota (State 27)

Iowa (State 19)

25

Workbench File Example: County Subdivisions

26

Example: Connect with Oracle Wallet

Thank You!

•  Questions? •  For more information:

–  Jay E. Spurlin •  jay.e.spurlin@census.gov

–  Bureau of the Census •  4600 Silver Hill Road, Stop 7400 •  Washington, DC 20233

•  Products made using FME Engine –  TIGER/Line Shapefiles

•  http://www.census.gov/geo/www/tiger/index.html –  American FactFinder (map base data)

•  http://www.factfinder.census.gov/home/saff/main.html?_lang=en