Module 1: Why Use Statistics? An Introduction to the ERP Case Study Steve DeGabriele (Massachusetts...

Module 1: Why Use Statistics?

An Introduction to the ERP Case Study

Steve DeGabriele(Massachusetts DEP)

For the Workshop: Using Essential Statistics for Effective Program Management

Presented at: 2008 Symposium onInnovating for Sustainable Results

Chapel Hill, North CarolinaJanuary 7, 2008

Overview: Using Essential Statistics forEffective Program Management

• Module 1• Why use statistics?

• Why use the Environmental Results Program (ERP) as a case study?

• Introduction to ERP and its measurement approach

• Module 2• Key concepts for planning and analyzing a single random sample

• Orientation to ERP statistical spreadsheet tools

• Group exercises and discussion

• Module 3• Key concepts for planning and analyzing comparisons of two random

samples

• Orientation to ERP statistical spreadsheet tools

• Group exercises and discussion

Why Use Statistics?

• Statistics are valuable whenever it’s too costly or inefficient to look at everything of interest– whether widgets or dry cleaners

• Random samples of facilities provide a picture of everyone’s performance, with measurable uncertainty

• Sample must be “representative”• E.g., a sample of volunteer facilities cannot be extrapolated to the

population as a whole

Uncomfortable with Statistics?

• That’s normal, but consider the alternatives• Census of all facilities

• Accurate, but expensive• Visiting samples of facilities non-randomly

• Cheaper, but little sense of how representative the data are

How to Use Statistics As Measurement Tool

• Primary uses for statistics:• Estimate how well a group of facilities is performing at a

particular point in time• Assess whether performance for a group of facilities has

changed over time• Assess whether there are differences in performance between

two different groups of facilities

What Gets Measured?

• You can apply statistical analysis to just about any measurable characteristic of a group of facilities• Are they in compliance with an important requirement? With

all requirements?• Are they implementing a key pollution prevention practice?• How much hazardous waste are they generating?

How Might This Help You?

• Statistics can help you assess the need for (and efficacy of)... • traditional compliance approaches, • innovative environmental performance initiatives, • pollution prevention efforts, and • voluntary programs.

• Why are you here? Do you think you might be able to use statistics in your work?

Why Use ERP As a Case Study?

• ERP’s measurement approach can be used to evaluate group performance...• at a single point in time, • over time, and • across states

• ERP itself involves elements of... • traditional compliance approaches, • innovative initiatives, • voluntary programs, and • pollution prevention

What is ERP?

• Pioneered by MA DEP in 1990s• Printers, dry cleaners, photo processors

• Integrates proven tools to cost-effectively improve performance in sectors characterized by large numbers of small sources of pollution

• To date, average initial sector improvements from 5% to 30%• Improvement continues in later years• Stabilizes at high levels

ERP: Interlocking Tools, Integrated System

ERP: For Large Numbers of Small Pollution Sources

• Developed by MA in 1990s for printers, dry cleaners and photo processors• Hundreds of facilities per sector• Significant aggregate environmental footprint• Traditional approaches untenable• DEP resources shrinking

• ERP sectors have large numbers of facilities nationally• e.g., over 30,000 auto body/paint shops in U. S. • So far, six states with auto body ERPs

Current State of ERP

• Endorsed for "scale up" by EPA Innovation Action Council (IAC) in 2000• Based on MA results, NAPA evaluation and potential to

address significant problems

• Tangible EPA support since then• Grants, technical support, tools and resource flexibility

• States ERP Consortium formed in 2006• Organized as "forum" under ECOS• Promotes and supports use of ERP approaches• EPA NCEI is a steering committee member

Who's Doing ERP? (18 States, 8 EPA Regions)

May 2007

ERP Now Covers 11 Sectors/Groups

Small Animal Feedlots (MN)

Auto Body Shops (DE, MD*, ME, NY, RI, WA)

Dry Cleaners (MA, MI, NH, NV)

Oil & Gas Extraction Facilities (LA)

Printers (MA, NY, WI)

Underground Injection Wells (IL)

*FL/MD no longer implementing ERP. MD had one ERP that covered both auto body and auto repair shops

ERP Now Covers 11 Sectors/Groups (Cont.)

Auto Salvage Yards (IN, RI)

Auto Repair Shops (FL*, MD*)

Gas Stations (RI, VA, VT)

Photo Processors (MA)

Stormwater Dischargers (ME, RI)

*FL/MD no longer implementing ERP. MD had one ERP that covered both auto body and auto repair shops

How ERP Works: A Typical ERP Cycle

Step 1:Inventory

Step 2:Statistical BaselineInspections

Step 3:Compliance Assistance

Step 4:Self-Certification

Step 7:InformedDecision-Making

Step 6:Statistical Post-CertificationInspections

Step 5:TargetedFollow-Up

Renew Assistance and Certification (As Deemed Necessary)


Identify the myriad small facilities that are sources of pollution, many of which are often unknown to regulators.

Step 1:Inventory


Conduct random inspections to accurately measure existing environmental performance and focus outreach on the biggest problems.

Step 2:Statistical BaselineInspections


Work with trade associations to create and provide plain-language, user-friendly assistance that improves compliance and promotes pollution prevention.

Step 3:Compliance Assistance


Facilities conduct self-assessments using a detailed checklist closely linked to assistance materials.

Responsible officials certify to their facilities’ environmental performance on each item. If necessary, they submit plans to return to compliance.

In some cases, certification has been mandatory; in others, voluntary. In all cases, compliance is required.

Step 4:Self-Certification


Identify potential problem facilities via certification analysis, and target them for inspections, correspondence or phone calls. Provide assistance and/or initiate enforcement, as needed.

Step 5:TargetedFollow-Up


Conduct random inspections to accurately estimate performance changes and verify facility certifications.

Step 6:Statistical Post-CertificationInspections


Assess performance data and consider whether to adjust compliance assistance or other strategies directed at the sector or, if sufficient progress has been made over time, target resources elsewhere.

Step 7:InformedDecision-Making

Renew Assistance and Certification (As Deemed Necessary)

How ERP Measurement Works

• Performance assessment is largely based upon random, inspector-collected data before and after self-certification• Data are representative and objective

• Recognize that baseline inspections may have an effect on performance • It’s part of what you are measuring

• Critical to follow principles of good data quality, such as:• Identify as complete a list of facilities as possible• Design a sound inspection checklist (garbage in, garbage out)• Train inspectors to follow sampling protocol & collect good and

complete data• Quality Assurance Project Plans (QAPPs) help many states

Who Gets Inspected?

• Random samples of all facilities• Typically about 35-50 per round of inspections, sometimes

higher

• Even in voluntary certification programs, you don’t just inspect the volunteers

• Quality issues with just sampling volunteers: • missing the big picture, • self-selection bias, and • potential to miss spillover effects

What Gets Measured in ERP?

• States have a lot of flexibility in choosing their measures, but there is convergence

• Environmental Business Practice Indicators (EBPIs) are basic measurement building blocks• EBPIs represent what an agency considers to be the most

important environmental aspects for the group• Typically multimedia, including both compliance and beyond-

compliance issues• A multimedia inspection checklist may have well over 100

questions, but most states focus on 10-30 EBPIs• Statistical approaches can be used to infer how well an overall

group is performing on EBPIs, based upon just the results from random inspections

EBPIs: Roll-up and Stand-Alone

• Roll-up indicator • Summarizes the results of several questions• E.g., a Maryland EBPI tracked the implementation of several

emission-reducing painting practices at auto body shops. Maryland observed an increase in use of such painting practices from 40% to 62% of inspected facilities.

• Stand-alone indicator• Linked to one important question on the facility checklist• E.g., a Rhode Island auto body EBPI tracked the usage of

methylene chloride, a dangerous paint-stripping chemical. Rhode Island observed a decrease in usage from 33% of inspected facilities to 5% of inspected facilities.

Other ERP Measures

• For simplicity, statistical analysis of EBPIs is the focus of this training course

• ERP states use statistics to calculate other important measures, too. E.g.,• Group compliance score: Extent to which the average facility is

achieving all relevant compliance-related EBPIs• Certification accuracy: Extent to which facility self-certification

responses agree with observations of inspectors• Change in environmental condition: Extent to which the

environment has changed, pollution has decreased, etc., since ERP implementation

Coming Up: A Closer Look at theTwo Main ERP Analyses

• Module 2: Current state of performance• Looking at a single random sample

• Module 3: Difference over time• Looking at 2 random samples, before and after certification• Difference between states is very similar, but not our focus (for

simplicity)

For more information…

Steve DeGabriele

Director, Business Compliance Division

Massachusetts DEP

[email protected]

Work: 617-556-1120

Module 1: Why Use Statistics? An Introduction to the ERP Case Study Steve DeGabriele (Massachusetts...

Documents

Transcript of Module 1: Why Use Statistics? An Introduction to the ERP Case Study Steve DeGabriele (Massachusetts...