Re-creating the Provincial Consolidated Cutblocks Dataset

Iaian McDougall

About Me Spatial Data Specialist at the Forest Analysis and

Inventory Branch (FAIB) – FLNRO

We manage the Vegetation Resources Inventory (VRI) dataset

VRI Provides answers to 2 questions: Where vegetation resource are located How much vegetation resources are in each inventory unit

The VRI is too hot to handle…

Project Objective Re-create a provincial wide dataset of Cutblocks that:

Can be easily be replicated every year

Consolidates data from multiple sources

Is an accurate estimate (We are your best guess)

Contains Harvest Date and Timber Supply Area (i.e.forest admin boundary)

All in one FME workbench

Potential Challenges Provincial wide datasets

Large datasets (over 4 million VRI polygons with over 100 attributes alone)

Potential to run out of Memory (overlays will be difficult)

Several datasets stored in a Relational Database

With no write access With complex data structures

Definition

Cutblock - “A specific area, with defined

boundaries, authorized for harvest.” 1

1. Timber Supply Review Program. 2007. Standard glossary of terms. Revised. Forest Analysis and Inventory Branch, B.C. Ministry of Forests and Range, Victoria, B.C.

Aren’t Cutblocks already in the VRI?

Cutblock Data Sources Forest Tenures Database – stored in Oracle Relational Database

FTEN_CUT_BLOCK_POLY

RESULTS – stored in Oracle Relational Database Forest Cover Poly RSTLS OPENING POLY RSLT_ACTIVITY_TREATMENT

VRI VEG_COMP_LYR_R1_POLY

4.5 million polygons , flat file

Change Detection Landsat analysis - GDB

Cutblock Data Sources: VRI Dataset Aren’t Cutblocks already in the VRI?

YES….mostly HARVEST YEAR and OPENING_ID

Attributes indicates cutblock PROBLEM: They are not all there

Some OLDER openingIDs were lost during VRI re-inventories

Some cutblocks have yet to be cut into the VRI

Cutblock Data Sources : RESULTS RESULTS -Reporting Silviculture Updates and Land

status Tracking System. Designed to track

Openings (e.g. cutblocks, disturbances) Disturbances Silviculture activities

Licensees must submit all harvest activities To RESULTS

Started in 2005 , less accurate for older cutblocks Some RESULTS Submissions have been rejected

(incomplete info)

Cutblock Data Sources– Forest Tenures Database Licensees must submit request for Permits

Cutblocks assigned a unique Opening_ID

Status of Opening ID is updated throughout process

Generalized cutblock line work only

Cutblock Data Sources– Change detection

Uses change detection/Land classification procedures

Measures major differences in Landsat 7 band values year after year

Needed for missed or un updated cutblocks in Results, FTEN and VRI

RESULTS

VRIMS

Annual Attribute Projectio

n

Annual BCGW Release

Phase 1

LVI

LiDAR

Wildfire

Other

FTEN

Chg. Det

Forest Tenures RESULTS VRI

Other VRI Inputs

Setup: Assigning Layer Priority Give each data source a priority

1. Veg_Comp_R1_Layer 2. RESULTS Forest_Cover_POLY 3. RESULTS OPENING_POLY 4. FTEN FOREST_TENURES_POLY 5. Change Detection

Remove any cutblock that exists in higher priority layer

Layers 1- 4 all have cutblocks with openings ids Some of Layer 1 has a Harvest_Date but no opening_ID

Dataset

Oracle Table Where Clause

VRI VEG_COMP_LYR_R1_POLY Harvest_Date > 1955

RESULTS

1. FOREST_COVER_POLY (spatial)2. RSLT_ACTIVITY_TREATMENT_UNIT

(non_spatial)3. RSLT_OPENING_VW (non-spatial)

- Internal Joins on Opening_ID- WHERE (( "SILV_BASE_CODE" = 'DN') and

("DISTURBANCE_CODE" in ('L','R', 'S') ) or (("SILV_BASE_CODE" = 'DN') and ("SILV_SYSTEM_CODE" is not null) and (“DISTUBANCE_CODE”) <> “E”))

- OR (("OPEN_CATEGORY_CODE" not in ( 'NREQ', 'NDWL', 'NDAML' , 'NDCF' , 'NDFS', 'NDVML', 'SPEX', 'SMPCF', 'SMPEX', 'SMPFS', 'SMPML', 'SMPWL')) and ((“SILV_BASE_CODE” = ‘DN’) and “Forest_File_ID” is not Null)

RESULTS

1. RSLT_OPENING_VW (spatial)2. RSLT_ACTIVITY_TREATMENT_UNIT

(non_spatial)3. FOREST_COVER_POLY (non -

spatial)

Same as above

FTEN FTEN_CUT_BLOCK_OPEN_ADMIN

BLOCK_STATUS_CODE in ('LC','S','HB') and DISTURBANCE_END_DATE is not Null and DISTURBANCE_START_DATE is not Null and DISTURBANCE_GROSS_AREA > 0.1

Setup: Extracting cutblocks from Data Sources

Initial attempt

• Worked well on initial small sample size• But……

Group-based transformers Feature Merger, InlineQuerier, Matcher are all group

based transformers

Hold all features in memory at a single time

Multiple group-based transformers = storing multiple copies of the data at any one time

No wonder I was using up to much memory (feature merger 8 times in 4 different streams + AreaOnAreaOverlayer)

• Worked well on initial small sample size• But……

Reader VS SQL Creator

Reader

• Read each input table individually

• Use InlineQuerier (or Feature Merger), to Join tables

• Matcher or FeatureMerger to remove OPENING_IDS from higher priority layers

SQL Creator/SQLExecutor• Do all the joining and

filtering using SQL Statements

• Use NOT IN statements to remove OPENING_IDS from layer with higher priority

SQL Creator Have Oracle do the work using

SQLCreator SQL - Joins, Where/Not IN statements

Updated Workbench

• Easier to Follow

• Runs in less than 1.5 hrs

• Final Result yields ~ 688 000 cutblocks polygons

Adding TSA Information

Adding TSA Information Clipper works

BUT takes 2 days to Process…..

Leaves slivers

Solution 1 – Adding TSA Information Use add clipper first setting

Solution 1 – Adding TSA Information How to get the clipper to arrive first?

Make sure it is on top of navigator

Solution 1 – Adding TSA Information What if using SQLCreator in the

workspace, NOT all Readers? SQLCreator SQLExecutor Use a Creator Transformer to Initiate

Solution 1 – Adding TSA Information Still one day to run….

Solution 2 – Adding TSA Information Let other programs do heavy lifting ArcGIS – Identity Tool to Add TSA info

(completes in under 1 hr)

ArcGIS Eliminate tool to eliminate slivers (completes in 1 hr)

Solution 1 – Adding TSA Information Incorporate a workbench into ESRI Model

Builder (requires FME Extension for Arcgis)

Create FME ETL Tool for intergration

How to incorporate ArcGIS toolsOption 2

Shutdown Python Script

No extensions needed

Calls python script at end of workbench run

Final Results

Conclusion FME has enabled us to produce a script that is

All in one workspace Fast Handles large datasets

Group-Based transformers can have trouble with large data

SQL Creator and SQL Executor allow RDBMS to do heavy lifting

ArcGIS tools/python can be easily incorporated into workbench to further customize

Useful links The VRI dataset can be downloaded from

:http://catalogue.data.gov.bc.ca/dataset/vri-forest-vegetation-composite-polygons-and-rank-1-layer

Thank you!Contact info:

iaian.mcdougall@gov.bc.caTim.salkeld@gov.bc.ca