portland2003_animated

Portland, Oregon

October 27-28, 2003

A

Sustainable Resource Development

Lessons Learned

From Handling More Data

Than

Grains of Sand on the Earth

Presenter: Tammy Kobliuk

Resource Analysis Section

Forest Management Branch

Public Lands and Forest Division

Alberta SRD

Portland, Oregon

October 27-28, 2003

A


Brief Overview • A little bit about Alberta

• A brief history of GIS in SRD

• Our technical setup

• Data, data, data…

• Analysis challenges

• Technical challenges

• Lessons learned

• What the future holds

Portland, Oregon

October 27-28, 2003

A


A Bit About Alberta

Portland, Oregon

October 27-28, 2003

A


An Alberta Primer

• We’re big.

• We have oil & gas, cows, trees, and grain.

• We are considered a “Prairie” province.

• We are where the prairies meet the mountains

(those little things we call the Rockies).

Portland, Oregon

October 27-28, 2003

A


Where are we?

ÊÚ

ÊÚ

BritishColumbia

Alberta Saskatchewan

Manitoba

Ontario

Washington

OregonIdaho

Montana

Wyoming

North Dakota

South Dakota

Nebraska

Minnesota

Iowa

Wisconsin

Michigan

California Nevada UtahColorado Kansas Missouri

Illinois

Alaska

Edmonton

Portland

Portland, Oregon

October 27-28, 2003

A


How big are we?

• 66,258,096 Hectares

• 662,581 Square Km

• 255,824 Square Miles

• 163,726,700 Acres

• 2 UTM zones

Portland, Oregon

October 27-28, 2003

A


A Size Comparison

Washington +

Oregon +

Nevada =

ALBERTA

Portland, Oregon

October 27-28, 2003

A


Tiling the province

• 50 1:250 000 NTS mapsheets

• 764 1:50 000 NTS mapsheets

• 7204 Townships

• > 1 million quartersections

• 4400+ Phase 3 mapsheets

Portland, Oregon

October 27-28, 2003

A


Topography

Portland, Oregon

October 27-28, 2003

A


Ecological

Regions

Prairie

Parkland

Forest

Portland, Oregon

October 27-28, 2003

A


Green / White

Areas

Portland, Oregon

October 27-28, 2003

A


Federal

Lands

Portland, Oregon

October 27-28, 2003

A


Land

Ownership

Portland, Oregon

October 27-28, 2003

A


Protected

Areas

Portland, Oregon

October 27-28, 2003

A


Forest

Management

Units

Portland, Oregon

October 27-28, 2003

A


Forest

Management

Agreements

Portland, Oregon

October 27-28, 2003

A


Long Term

Allocations

Portland, Oregon

October 27-28, 2003

A


Who Are the Resource Analysis Section?

Forest Protection

Resource Analysis Section

Forest Health

Forest Planning

Harvest and Renewal

Forest Management Branch

Forest Operations

Forest Business

Public Lands and Forest Fish & Wildlife PLFD Regions


Portland, Oregon

October 27-28, 2003

A


What do we do?

• Review timber supply and technical analysis of company management plans.

• Timber supply analysis and technical analysis for Crown management plans.

• Provincial-scale reporting to federal govt.

• Development of modeling tools and analysis procedures.

• Development/maintenance of specialized databases.

Portland, Oregon

October 27-28, 2003

A


What else do we do?

• Technical support for departmental

initiatives.

• On-demand analysis for the

Minister’s office.

• Answer Freedom-of-Information

(FOIP) requests.

• Any other priorities-of-the-day.

Portland, Oregon

October 27-28, 2003

A


A Brief History of GIS in the

Forest Management Branch • In the beginning…there were dot grids and

colored pencil crayons.

• Terrasoft was purchased in the late 1980’s to early 1990’s.

• Terrasoft was retired in 1995 when ArcInfo and ArcView were first purchased.

• Widespread ArcView training commences in 2001 with the implementation of the Citrix ArcView project. Forest Management is the primary driver.

Portland, Oregon

October 27-28, 2003

A


…

• Base data has always been managed by the

Resource Data Branch.

• The original capture of digital elevation data was

in the 1980’s for the purpose of contour creation.

This data is still in use today.

• The original capture of the base data was by CAD

systems. This has been improved and updated to

attributed ArcInfo-ready files.

Portland, Oregon

October 27-28, 2003

A


…

• Resource Data Branch was de-centralized in the mid-to-late 1990’s as “GIS” staff were sent out to colonize the regions. These staff were primarily CAD users and had little GIS training.

• Forest Management GIS staff are located only in Edmonton.

• Edmonton no longer has control over regional GIS staff.

• Control over computer equipment purchases has been returned to Edmonton.

Portland, Oregon

October 27-28, 2003

A


Technical

Setup

Portland, Oregon

October 27-28, 2003

A


General Setup

PC SunBlade workstation

Data drive

Data drive

Data drive

Data drive

PC PC

PC PC

Citrix Application Server

Data Server

Citrix

Server

Farm

Server Room

FMB

Portland, Oregon

October 27-28, 2003

A


Basic Hardware

• Sunblade 1000 dual CPU workstation with

500 GB storage space.

• Compaq workstations: dual CPU, dual

harddrive, CD writer, DVD, 21” monitors

• HP2500 Plotter (soon to be replaced)

Portland, Oregon

October 27-28, 2003

A


GIS Software

• ArcInfo 7.2.1 and 8.2 Workstation (Solaris)

– 2 Node locks

– 2 Floating

• ArcView 3.2 (Windows 2000)

• ArcView 8.2

• ArcIMS and ArcExplorer

• SDE (Resource Data Branch only)

Portland, Oregon

October 27-28, 2003

A


Remote Sensing Software

• ERMapper 6.3

– Image processing

– Image compression (.ECW)

• Erdas Imagine Professional 8.5

– Primarily legacy modelling

Portland, Oregon

October 27-28, 2003

A


Enabling Software

• Hummingbird Exceed for XTerming into unix workstation.

• Samba for serving out unix data drives.

• Citrix Metaframe for serving out thin client ArcView and regional data.

• Oracle for enterprise database applications. *Note: Not currently available to ArcInfo users.

Portland, Oregon

October 27-28, 2003

A


Little Added Extras

• Adobe Acrobat 5

• CorelDraw 9 and Corel

PhotoPaint 9

• Microsoft Office 2000

• FoxPro

Portland, Oregon

October 27-28, 2003

A


Software Maintenance

• When you’re big, costs are high.

• $225,000 to ESRI annually for SRD.

• ArcView 3.2 to ArcView 8.x

conversion comes with a steep annual

pricetag.

• ~$68,000 annual FMB costs.

• $ 3 million just to migrate operating

systems.

Portland, Oregon

October 27-28, 2003

A


Data, data everywhere…

Portland, Oregon

October 27-28, 2003

A


Sure there’s data, but…

• Who has it?

• Can you get it?

• What scale is it?

• Where do you store it?

• What does it cover?

• How good is it?

• How old is it?

Portland, Oregon

October 27-28, 2003

A


Where’s the Data?

Data

Resource Data Branch

Spatial Data Warehouse

FMA-Holders

Federal Government

Research

Organizations

Internet

Other Provincial

Agencies

Portland, Oregon

October 27-28, 2003

A


Data Acquisition in Canada

• Very little available for free.

• Cost recovery model for governments.

• Most data is licensed.

• Can be very difficult to find out who has it.

• When you can find it, costs are often prohibitive.

Portland, Oregon

October 27-28, 2003

A


Base Data in Alberta

• ESRI Digital Chart of the World

• Provincial 1:1 million

• Federal 1:250,000 and 1:50,000

• Provincial 1:20,000 (and

1:50,000)

• Local (municipalities, cities and

utilities)

Portland, Oregon

October 27-28, 2003

A


Data Storage

• Resource Data Branch is the primary data

repository and distribution arm for SRD.

• Regional/Area offices load and maintain

detailed base information for their areas.

• The Citrix data server is the only central

accessible data repository for users.

• FMB stores all data acquired for projects

for internal use only.

Portland, Oregon

October 27-28, 2003

A


Vegetation Data

• Canada/Alberta Land Inventories

• Ecological and biophysical inventories

• Satellite classification

• Strategic level Forest/Vegetation Inventories

• Wetland Inventory

• Prairie Vegetation Inventory

Portland, Oregon

October 27-28, 2003

A


More on Strategic Level Inventories

• Phase 1

• Phase 2

• Phase 3

• AVI 1.0

• AVI 2.1 * The current standard

• AVI 2.2

Portland, Oregon

October 27-28, 2003

A


Data Coverage

It’s hard to know what coverage we have for

what data. There are few accessible maps

detailing the age and coverage of base,

vegetation, and other data.

Portland, Oregon

October 27-28, 2003

A


Coverage Example:

Vegetation Inventories

• Phase 1, Phase 2, Phase 3

• Alberta Vegetation Inventory: CVI, AVI

1.0, AVI 2.1, AVI 2.2.

Portland, Oregon

October 27-28, 2003

A


Present

Digital

Vegetation

Inventory

Status

Portland, Oregon

October 27-28, 2003

A


Data Quality?

Reports on data quality and

acquisition specifications

are not readily available to

internal staff, let alone the

general public.

Portland, Oregon

October 27-28, 2003

A


Analysis

Challenges

Portland, Oregon

October 27-28, 2003

A


Projections, anybody?

• Alberta covers 2 UTM

zones

• A single projection

requires a custom

projection (10TM)

• Local: 3TM

• Agriculture: Albers

• Canada: Lambert

11 12

Portland, Oregon

October 27-28, 2003

A


Size of study areas…

• Extremely large areas of contiguous

industrial forest land.

• Massive size of individual tenures and

management units.

• Undergoing amalgamation of units.

• Multi-jurisdiction study areas.

Portland, Oregon

October 27-28, 2003

A


The Multi-Use Landbase

• There is a phenomenal amount of activity

modifying the landscape in some areas of

the province.

• Oil & gas, grazing, forestry and recreation

are all on some of the same areas.

• Datasets are out-of-date the moment they

are completed.

Portland, Oregon

October 27-28, 2003

A


Seamless Data?

• Only recently has it been possible to order seamless base data in coverage format.

• 110+ vegetation inventory datasets to cover forested areas.

– Overlaps

– Different dates

– Differing quality

– Differing database formats

Portland, Oregon

October 27-28, 2003

A


Provincial Analysis

• No seamless spatial vegetation dataset.

• 3.85 million inventory polygons over 110+ datasets.

• How to create a provincial dataset that can incorporate data of differing dates and specifications?

• Do you need stand-level data?

• Can you use proxies?

Portland, Oregon

October 27-28, 2003

A


Change Over Time

• Keeping data for a project “up-to-date” is impossible.

• Choose an effective date for analysis. Document this.

• Make and document assumptions surrounding dated data.

• Choose a projection and datum and stick with it. Document this.

Portland, Oregon

October 27-28, 2003

A


Confidentiality

• Company DFMP submissions are

confidential.

• Who should have access to what?

• What type of access should they have?

• How do you implement those access

restrictions?

Portland, Oregon

October 27-28, 2003

A


Getting Information From Your Data

• Does your data require massaging to extract

useful information?

• Can the data be enhanced by attribute

rollup?

• Can the data be enhanced by predictive

modelling?

Portland, Oregon

October 27-28, 2003

A


Case Study:

Interior Forest Analysis

• Study Area: single FMA 3.5 million Ha

• Source data: ArcInfo Library (~2.25

million polygons in 406 tiles)

• Attributes in separate .dbf file.

• Problem undefined.

• Solution undefined.

Portland, Oregon

October 27-28, 2003

A


Timeline

• 2 weeks to nail down the exact problem and formulate the solution(s). Solution(s) were programmed for automation.

• 2 weeks to complete data prep and spatial analysis.

• 1 day to compile final datasets.

• 5 minutes to run Patch Analyst for final landscape metrics.

Portland, Oregon

October 27-28, 2003

A


Obstacles

• Had to create input datasets tile-by-tile since the entire spatial net landbase could not be extracted.

• Could not grid off the entire FMA at 5 m.

• Difficult to partition the landscape into smaller pieces.

• Vector processes took an inordinate amount of time.

• We seemed to hit every software and hardware limit possible.

Portland, Oregon

October 27-28, 2003

A


Raster vs Vector

• Raster data sets were very large.

• Raster processes were comparable to vector

processes on simpler landscapes.

• Raster processes were faster than vector

processes on complex landscapes.

• Raster outputs were easier to check than

vector.

Portland, Oregon

October 27-28, 2003

A


From Plan A to Plan D

• Plan A: Grid off the entire FMA and run

analysis.

• Plan B: Run entire FMA as vector

• Plan C: Partition the FMA and run pieces

as raster.

• Plan D: Run remaining partition as vector.

Portland, Oregon

October 27-28, 2003

A


Technical

Challenges

Portland, Oregon

October 27-28, 2003

A


Lots of Data Means…

• Disk space issues:

– How many should you have?

– Where do you put them?

– How big should they be?

• Backup issues.

• Archiving issues.

• File management issues.

Portland, Oregon

October 27-28, 2003

A


Large Datasets Mean…

• Streaming live data requires lots of

network bandwidth.

• Hardware, operating systems, and software

need to be optimally tuned.

• Unix systems may still out-perform PC’s.

• Data display needs to be managed.

• Data format needs to be carefully chosen.

Portland, Oregon

October 27-28, 2003

A


Size is Relative

• What’s large for you?

• What’s large for the

software developer?

• Are software limits

documented? Can you

find limits prior to running

a process?

Portland, Oregon

October 27-28, 2003

A


So Your Command Ran…

So What?

• ArcInfo commands may run to completion

on large datasets, but:

– Did it really complete?

– Did it run correctly?

– How do you check it?

– Did it clean up after itself?

Portland, Oregon

October 27-28, 2003

A


Plotting Large Files

• Print directly from ArcView or ArcGIS.

OR

• Export to .eps and use Image Alchemy

to convert to on-the-fly RTL.

• Export to .eps and use Adobe Acrobat

Distiller to create .PDF files.

Portland, Oregon

October 27-28, 2003

A


Data Transfer

How to transfer very large files?

• Attempt to avoid ftp timeout.

• Compress and burn to CD.

• Get into burning DVD’s.

• Dig out the old tape drive.

Portland, Oregon

October 27-28, 2003

A


Moving to Geodatabase

• The 2GB Personal Geodatabase limit is too

small for many datasets.

• We don’t currently have Oracle access.

• We don’t have an Oracle DBA.

• We don’t have an SDE license. Needed?

• It’s not feasible to port all of our data.

• Is Oracle Personal an option?

Portland, Oregon

October 27-28, 2003

A


SDE?

• SDE requires a lot of up-front Oracle database tuning.

• Requires a knowledgeable administrator.

• $$$ Purchase cost and $$ annual maintenance.

• Fine for map display and browsing/querying.

• Not good for spatial analysis.

Portland, Oregon

October 27-28, 2003

A


Lessons

Learned

Portland, Oregon

October 27-28, 2003

A


The Basics

• Keep it simple!

• Newer doesn’t necessarily

mean better.

• Older doesn’t always mean

bad. (ie. AML)

• Don’t put all your eggs in one

basket (ie. Web deployment).

Portland, Oregon

October 27-28, 2003

A


Knowledge and Creativity

• Know your data and what it can and should

be used for.

• Don’t reinvent the wheel. Look at what

others have done.

• There’s always more than one way to skin a

cat. Always have a plan B and C for when

plan A fails.

Portland, Oregon

October 27-28, 2003

A


Technology

• Know your software. Use it.

• Use the best tool for the job. No one toolset will

do everything you need.

• Be cautious. Let other people find the bugs.

• Poorly planned software upgrades can become

“downgrades”.

• Monitor software and hardware developments.

• Do performance tuning on hardware/software.

Portland, Oregon

October 27-28, 2003

A


Practice Good File Management

• Get rid of unnecessary and duplicate files.

• Archive to CD, DVD, and/or tape.

• Take responsibility for managing files

and data.

• Attempt to keep documentation.

• Standardize wherever possible.

Portland, Oregon

October 27-28, 2003

A


Cost Management

• Consolidate software client numbers to take advantage of secondary license pricing.

• Use your organization size or partnership arrangement(s) to negotiate a deal.

• Consider floating licenses or software servers to cut down on total license numbers.

• Keep an up-to-date inventory of licenses.

Portland, Oregon

October 27-28, 2003

A


Project Risk Management

• Don’t put all your eggs in the ArcView project

or ArcGIS map document baskets.

• Sever the reliance on data availability and

location.

• Save out legend and layer files.

• Save important maps as image files and/or .pdf

files. Burn them to CD. Have hard copies.

• Document, document, document…

Portland, Oregon

October 27-28, 2003

A


Check Your Outputs

• Spatially view your data.

• Query out results.

• Do spot checks.

• Do topology checks.

Portland, Oregon

October 27-28, 2003

A


To Contract or Not To

Contract? • DO Use consultants wisely.

• DO contract out simple one-time projects that regular staff don’t have time for.

• DO ensure transfer of skills to internal staff.

• DON’T use them as a replacement for internal staff.

• DON’T get too closely tied to any one company

Portland, Oregon

October 27-28, 2003

A


Failing the Bus Test?

• Inventory your data.

• Inventory your folders.

• Standardize your directory structure.

• Keep up on metadata for data.

• Keep up on metadata for projects.

Portland, Oregon

October 27-28, 2003

A


Data Access Management

• Limit data exposure.

• Use different Samba shares to share out

specific directories.

• Use different Samba shares to enforce

access control.

• Use different Unix account groups to allow

for setting Unix permissions.

Portland, Oregon

October 27-28, 2003

A


Change Management

• Don’t hardcode things that will change over

time. (ie. Administrative boundaries)

• Make use of relational databases to enable

partial updates.

• Plan for scalability.

Portland, Oregon

October 27-28, 2003

A


Data Models

• Develop them from the start.

• Document them and make them available.

• Don’t let non-data people control Data

Model creation.

• Ensure proper representation of end-users in

the Data Model creation process.

• Ensure that the data model meets needs.

Portland, Oregon

October 27-28, 2003

A


Learn • Admit what you don’t know. Then try

to learn.

• The pace of change in both software and hardware is extraordinary. Keep tabs on it.

• You can’t be an expert on everything. Know a little about a lot and a lot about a little. Specialize.

Portland, Oregon

October 27-28, 2003

A


Communicate

• Sell the technology from both ends. (ie. To management while also recruiting end-users).

• Communicate your accomplishments.

• Share what you learn through technology transfer. Generosity pays dividends.

• Network with your peers.

Portland, Oregon

October 27-28, 2003

A


What the Future Holds

Portland, Oregon

October 27-28, 2003

A


• ArcGIS 9 Suite

• XML-based metadata

• A possible move to geodatabase

• Still AML, but possibly Python

• SDE and Oracle?

• More imagery

• Possible server consolidation

Software/Data

Portland, Oregon

October 27-28, 2003

A


Expectations

• More demand for services.

• More complex analyses requested.

• More internal expectation of basic

ArcView skills.

• Internal training switch from

ArcView 3.2 to ArcView 8.x.

• More staff?

Portland, Oregon

October 27-28, 2003

A


And the Last Word…

Keep your sense of humour.

portland2003_animated

Documents

Transcript of portland2003_animated