FME Desktop Tutorial 2011: Introduction - FME Data Transformation
Creating a Provincial Wide Consolidated Cutblocks Layer using FME
-
Upload
safe-software -
Category
Technology
-
view
164 -
download
1
Transcript of Creating a Provincial Wide Consolidated Cutblocks Layer using FME
Re-creating the Provincial Consolidated Cutblocks Dataset
Iaian McDougall
About Me Spatial Data Specialist at the Forest Analysis and
Inventory Branch (FAIB) – FLNRO
We manage the Vegetation Resources Inventory (VRI) dataset
VRI Provides answers to 2 questions: Where vegetation resource are located How much vegetation resources are in each inventory unit
The VRI is too hot to handle…
Project Objective Re-create a provincial wide dataset of Cutblocks that:
Can be easily be replicated every year
Consolidates data from multiple sources
Is an accurate estimate (We are your best guess)
Contains Harvest Date and Timber Supply Area (i.e.forest admin boundary)
All in one FME workbench
Potential Challenges Provincial wide datasets
Large datasets (over 4 million VRI polygons with over 100 attributes alone)
Potential to run out of Memory (overlays will be difficult)
Several datasets stored in a Relational Database
With no write access With complex data structures
Definition
Cutblock - “A specific area, with defined
boundaries, authorized for harvest.” 1
1. Timber Supply Review Program. 2007. Standard glossary of terms. Revised. Forest Analysis and Inventory Branch, B.C. Ministry of Forests and Range, Victoria, B.C.
Aren’t Cutblocks already in the VRI?
Cutblock Data Sources Forest Tenures Database – stored in Oracle Relational Database
FTEN_CUT_BLOCK_POLY
RESULTS – stored in Oracle Relational Database Forest Cover Poly RSTLS OPENING POLY RSLT_ACTIVITY_TREATMENT
VRI VEG_COMP_LYR_R1_POLY
4.5 million polygons , flat file
Change Detection Landsat analysis - GDB
Cutblock Data Sources: VRI Dataset Aren’t Cutblocks already in the VRI?
YES….mostly HARVEST YEAR and OPENING_ID
Attributes indicates cutblock PROBLEM: They are not all there
Some OLDER openingIDs were lost during VRI re-inventories
Some cutblocks have yet to be cut into the VRI
Cutblock Data Sources : RESULTS RESULTS -Reporting Silviculture Updates and Land
status Tracking System. Designed to track
Openings (e.g. cutblocks, disturbances) Disturbances Silviculture activities
Licensees must submit all harvest activities To RESULTS
Started in 2005 , less accurate for older cutblocks Some RESULTS Submissions have been rejected
(incomplete info)
Cutblock Data Sources– Forest Tenures Database Licensees must submit request for Permits
Cutblocks assigned a unique Opening_ID
Status of Opening ID is updated throughout process
Generalized cutblock line work only
Cutblock Data Sources– Change detection
Uses change detection/Land classification procedures
Measures major differences in Landsat 7 band values year after year
Needed for missed or un updated cutblocks in Results, FTEN and VRI
RESULTS
VRIMS
Annual Attribute Projectio
n
Annual BCGW Release
Phase 1
LVI
LiDAR
Wildfire
Other
FTEN
Chg. Det
Forest Tenures RESULTS VRI
Other VRI Inputs
Setup: Assigning Layer Priority Give each data source a priority
1. Veg_Comp_R1_Layer 2. RESULTS Forest_Cover_POLY 3. RESULTS OPENING_POLY 4. FTEN FOREST_TENURES_POLY 5. Change Detection
Remove any cutblock that exists in higher priority layer
Layers 1- 4 all have cutblocks with openings ids Some of Layer 1 has a Harvest_Date but no opening_ID
Dataset
Oracle Table Where Clause
VRI VEG_COMP_LYR_R1_POLY Harvest_Date > 1955
RESULTS
1. FOREST_COVER_POLY (spatial)2. RSLT_ACTIVITY_TREATMENT_UNIT
(non_spatial)3. RSLT_OPENING_VW (non-spatial)
- Internal Joins on Opening_ID- WHERE (( "SILV_BASE_CODE" = 'DN') and
("DISTURBANCE_CODE" in ('L','R', 'S') ) or (("SILV_BASE_CODE" = 'DN') and ("SILV_SYSTEM_CODE" is not null) and (“DISTUBANCE_CODE”) <> “E”))
- OR (("OPEN_CATEGORY_CODE" not in ( 'NREQ', 'NDWL', 'NDAML' , 'NDCF' , 'NDFS', 'NDVML', 'SPEX', 'SMPCF', 'SMPEX', 'SMPFS', 'SMPML', 'SMPWL')) and ((“SILV_BASE_CODE” = ‘DN’) and “Forest_File_ID” is not Null)
RESULTS
1. RSLT_OPENING_VW (spatial)2. RSLT_ACTIVITY_TREATMENT_UNIT
(non_spatial)3. FOREST_COVER_POLY (non -
spatial)
Same as above
FTEN FTEN_CUT_BLOCK_OPEN_ADMIN
BLOCK_STATUS_CODE in ('LC','S','HB') and DISTURBANCE_END_DATE is not Null and DISTURBANCE_START_DATE is not Null and DISTURBANCE_GROSS_AREA > 0.1
Setup: Extracting cutblocks from Data Sources
Initial attempt
• Worked well on initial small sample size• But……
Group-based transformers Feature Merger, InlineQuerier, Matcher are all group
based transformers
Hold all features in memory at a single time
Multiple group-based transformers = storing multiple copies of the data at any one time
No wonder I was using up to much memory (feature merger 8 times in 4 different streams + AreaOnAreaOverlayer)
• Worked well on initial small sample size• But……
Reader VS SQL Creator
Reader
• Read each input table individually
• Use InlineQuerier (or Feature Merger), to Join tables
• Matcher or FeatureMerger to remove OPENING_IDS from higher priority layers
SQL Creator/SQLExecutor• Do all the joining and
filtering using SQL Statements
• Use NOT IN statements to remove OPENING_IDS from layer with higher priority
SQL Creator Have Oracle do the work using
SQLCreator SQL - Joins, Where/Not IN statements
Updated Workbench
• Easier to Follow
• Runs in less than 1.5 hrs
• Final Result yields ~ 688 000 cutblocks polygons
Adding TSA Information
Adding TSA Information Clipper works
BUT takes 2 days to Process…..
Leaves slivers
Solution 1 – Adding TSA Information Use add clipper first setting
Solution 1 – Adding TSA Information How to get the clipper to arrive first?
Make sure it is on top of navigator
Solution 1 – Adding TSA Information What if using SQLCreator in the
workspace, NOT all Readers? SQLCreator SQLExecutor Use a Creator Transformer to Initiate
Solution 1 – Adding TSA Information Still one day to run….
Solution 2 – Adding TSA Information Let other programs do heavy lifting ArcGIS – Identity Tool to Add TSA info
(completes in under 1 hr)
ArcGIS Eliminate tool to eliminate slivers (completes in 1 hr)
Solution 1 – Adding TSA Information Incorporate a workbench into ESRI Model
Builder (requires FME Extension for Arcgis)
Create FME ETL Tool for intergration
How to incorporate ArcGIS toolsOption 2
Shutdown Python Script
No extensions needed
Calls python script at end of workbench run
Final Results
Conclusion FME has enabled us to produce a script that is
All in one workspace Fast Handles large datasets
Group-Based transformers can have trouble with large data
SQL Creator and SQL Executor allow RDBMS to do heavy lifting
ArcGIS tools/python can be easily incorporated into workbench to further customize
Useful links The VRI dataset can be downloaded from
:http://catalogue.data.gov.bc.ca/dataset/vri-forest-vegetation-composite-polygons-and-rank-1-layer
Thank you!Contact info:
[email protected]@gov.bc.ca