Practical, Affordable Performance Measurement: Why and How? Shelley H. Metzenbaum...
-
Upload
duane-howard -
Category
Documents
-
view
216 -
download
1
Transcript of Practical, Affordable Performance Measurement: Why and How? Shelley H. Metzenbaum...
Practical, Affordable Practical, Affordable Performance Performance Measurement:Measurement:Why and How?Why and How?
Shelley H. Metzenbaum
[email protected] www.complianceconsortium.orgwww.facilityreporting.org
Northeast Environmental Summit September 2005
OverviewOverview• Why Measure?Why Measure?
• How to Measure and Use the How to Measure and Use the Measures?Measures?
• ExamplesExamples
• Issues to ConsiderIssues to Consider
• Workshop TomorrowWorkshop Tomorrow
Why Measure?Why Measure?
• Increase Increase AccountabilityAccountability
• LearnLearn to to Achieve Better Achieve Better OutcomesOutcomes
• Strengthen Strengthen DemocracyDemocracy
Measurement for Measurement for AccountabilityAccountability
(Program Justification?)(Program Justification?)
• EthicalEthical: : Self-Dealing/Fraud (tSelf-Dealing/Fraud (track rack $ flows to whom for what – accounting $ flows to whom for what – accounting & fiscal audits)& fiscal audits)
• FiscalFiscal: Unallowed : Unallowed spending/Waste/Efficiency spending/Waste/Efficiency ($ ($ spent/$ allocated & goods or services spent/$ allocated & goods or services bought per input cost)bought per input cost)
• PerformancePerformance: Improve social : Improve social outcomes/reduce risk/more outcomes/reduce risk/more cost-effective cost-effective (better outcomes; (better outcomes; better outcomes at lowest cost)better outcomes at lowest cost)
Measurement to Measurement to LearnLearn so so Achieve Better Achieve Better
OutcomesOutcomes• DETECTIONDETECTION: Detect : Detect problemsproblems needing needing
attention; attention; progressprogress deserving replication- deserving replication- what works and what doesn’t?what works and what doesn’t? – Look for anomalies, unexpected direction, Look for anomalies, unexpected direction,
speed ups, slow downs, etc.speed ups, slow downs, etc.– Use measurement to trigger focused follow Use measurement to trigger focused follow
up up
• DIAGNOSTICSDIAGNOSTICS: : Why? What is the reason Why? What is the reason for variation? Is it a problem or lesson?for variation? Is it a problem or lesson?
• ASSESSMENTASSESSMENT: : Did it seem to work? Did it seem to work? Replicate, expand, or stop and fix?Replicate, expand, or stop and fix?
Goals/Measures to Goals/Measures to Strengthen DemocracyStrengthen Democracy
• Cannot Do Everything: Goals/ Cannot Do Everything: Goals/ Targets Communicate: Targets Communicate: – What You Will DoWhat You Will Do– What You Won’t DoWhat You Won’t Do
• Are You Doing Right Thing? Are You Doing Right Thing? DemocraciesDemocracies can respond when can respond when unhappy with agency choicesunhappy with agency choices
• How Well Are You Doing It? How Well Are You Doing It? MeasurementMeasurement
Start with Start with Learning Learning Emphasis of Emphasis of MeasurementMeasurement
• Detection, Diagnostics, Detection, Diagnostics, AssessmentAssessment
• Do this well … Accountability & Do this well … Accountability & Democracy will Rise, as wellDemocracy will Rise, as well
• Practical, Affordable ExamplesPractical, Affordable Examples
DETECTION DETECTION : Look for Pattern : Look for Pattern Changes to Sense ProblemsChanges to Sense Problems
Why the Downturn ?
Source: http://www.chesapeakebay.net/status/status_print.cfm?sid=101&print=yes
DETECTION: Volunteer Data -- Look for Unexplained VariationsDescription Town Riv
er Mile
2/ 15/ 00 3/ 21/ 00
5/ 16/ 00
Central Street Bridge
Milford 3.5 1100 380 120
Discharge Pipe @ Central Street
Milford 3.5 1470 2100 3400
2d Discharge Pipe @ Central Street
Milford 3.5 <10 260
Mellen St. Bridge
Bellingham
5.9 490 140
Rt. 126, N. Main
Bellingham
9.0 340 10 120
DETECTION DETECTION : Look for Pattern : Look for Pattern Changes to Find ProgressChanges to Find Progress
Recognize Progress !
Source: http://www.chesapeakebay.net/status/status_print.cfm?sid=101&print=yes
DETECT: Before & After DETECT: Before & After PhotosPhotos
DETECTION/DIAGNOSTICS: DETECTION/DIAGNOSTICS: Inspec. DataInspec. Data“Scout & “Scout & Survey”Survey”% Compliance by Violation Type
0102030405060708090
100
Train
ing
Insp
ectio
ns
Label
s
Aisle
Spac
e
Close
d
Postin
gs
Charac
teri.
..
NH FQG
NH SQG
NH, CO
Detect low comp. rateDiagnose: Is it a problem?
Why is it happening?
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
8 10a 13 18 22 35 36i 38 50
(10) Leak Checks
(18-19) Controls
(36i) Emergency Procedures
Diagnostics: MA ERP Self-Audit Data 1997 & 1998
DETECT, DIAGNOSE, ASSESS: Self-Reported Practices Trends (97/98)
MA ERP
Lower than Other Areas:
Problem?
Changes over Time:
Progress?
1996 DRY 1996 DRY WEATHER WEATHER BACTERIABACTERIA
Swim StandardSwim Standard
Some FailuresSome Failures
FailedFailed
1996/97 WET WEATHER 1996/97 WET WEATHER BACTERIABACTERIA
DETECTION and DIAGNOSTICS: Mapping/Segmentation HighlightsProblem Areas and Progress
DIAGNOSTICS/ASSESSMENT: Slice & Dice by Categories to Identify Cause of Problems, Progress
Detection: # Detection: # ComplaintsComplaints(Cleveland Air (Cleveland Air Pollution)Pollution)• General Environmental Mgt. – 64General Environmental Mgt. – 64
• ISG – 52ISG – 52
• Ford Motor Company – 29Ford Motor Company – 29
• Kirkwood Industries – 18Kirkwood Industries – 18
• Landfill – 14Landfill – 14
• Wabash Alloys -12Wabash Alloys -12
DIAGNOSTICS: Complaint DIAGNOSTICS: Complaint Relevance?Relevance?
Source: PA DEP
ASSESSMENT: Pictures ASSESSMENT: Pictures ConsolidatedConsolidated Tailings Pile (CO)Tailings Pile (CO)
Prior to 1989 After Reclamation
1999
Lots of Affordable Ways to Lots of Affordable Ways to DETECT, DIAGNOSE, ASSESSDETECT, DIAGNOSE, ASSESS• Air and Water Quality DataAir and Water Quality Data
– Volunteer & govt collected data usefulVolunteer & govt collected data useful– Water, Energy Supply – Boston Water Water, Energy Supply – Boston Water
and Sewer Commission use data to cut and Sewer Commission use data to cut water use and costswater use and costs
• Cameras – Texas detects problem air Cameras – Texas detects problem air plumes; Rutgers extension helps farmers plumes; Rutgers extension helps farmers cut pesticides and costscut pesticides and costs
• Incident and Inspection Data: Incident and Inspection Data: – Repeat Violators - Accidents Repeat Violators - Accidents – Nature of Non-Compliance – ComplaintsNature of Non-Compliance – Complaints– FiresFires
DETECTION, DIAGNOSTICS, DETECTION, DIAGNOSTICS, ASSESSMENT: Interim ASSESSMENT: Interim Outcome Indicators IdeasOutcome Indicators Ideas
Attention
Comprehension
Beliefs/Attitudes
Motivation
Information
Behavior
Measure Awareness (OR)
Measure Understandin
gMeasure Attitudes
Measure Motivation
Measure Behavioral Changes (ERP)
Newspaper Content Newspaper Content Analysis, Surveys, Analysis, Surveys, Observational Observational Measurement Measurement
0
10
20
30
40
50
60
70
1962-1970 1971-1982 1983-1988 1989-1993
Technology
Regulation
Enforcement
Management
Chemical Industry Trade Press Content Analysis (Hoffman 1999)
STRENGTHEN DEMOCRACY: Specific Goals Clarify What Govt Will Do and What It Won’t
• Where: Lower Charles River
• When: 10 years
• How much: Swimmable
• Who: All Sources
• Invite public attention
• Public can respond
Strengthen Democracy: Make Strengthen Democracy: Make Goals, Progress, Problems Goals, Progress, Problems ClearClearGoal Not Met,
Past and Future
Strategy Described
Goal Met; Commitment to New
Goal Described
Issues to ConsiderIssues to Consider• Data AccuracyData Accuracy: Match to need. Can be lower : Match to need. Can be lower
for Detection; higher for Rules and for Detection; higher for Rules and Enforcement. For assessment, consider Enforcement. For assessment, consider program $program $
• Project and Measurement Project and Measurement ScaleScale: indiv. facility : indiv. facility level work OK only if have context/control. level work OK only if have context/control. Count costs and cost-effectiveness when Count costs and cost-effectiveness when project level. project level. Start with env. problemStart with env. problem..
• Comp asstc and P2 activity counts not useful Comp asstc and P2 activity counts not useful indicator; may have occasional diagnostic indicator; may have occasional diagnostic valuevalue
Tomorrow’s Tomorrow’s Workshop: Workshop:
Brainstorming Your Specific Measurement Challenges Together
Brainstorming Your Specific Measurement Challenges Together
1:30 - 3:30
Shelley H. Metzenbaumsmetzenbaum@verizon.netwww.complianceconsortium.orgwww.facilityreporting.org