Level 1 Water Audit Validation - CA-NV AWWA · ―Standardize water audit data collection and...
Transcript of Level 1 Water Audit Validation - CA-NV AWWA · ―Standardize water audit data collection and...
-
Level 1 Water Audit Validation
North American Water Loss Conference 2017
Lucy Andrews
-
What Is Typical Water Loss in the US?
-
Collecting Water Loss Data
CA DRBC GA TN TX
total audits 300 517 452 629 2,646
# of unrealistic audits 100 130 74 122 1,065
% of unrealistic audits 33% 25% 16% 19% 40%
R. Sturm, K. Gasner, and L. Andrews. 2015. Water Audits in the United States: A Review of Water Losses and Data Validity (Project 4372B). Denver, CO: Water Research Foundation.
-
Something Is Amiss…
What’s the problem?
How do I fix it?Water Research Foundation 4639: Level 1 Water Audit Validation
-
Data Quality Matters!
inaccuracy & uncertainty in
inputs
inaccuracy & uncertainty in
results
• Instruments
• Databases
• People
• Missing information
Sources of error:
-
What Does Validation Look Like Abroad?
― IWA – validation is recommendedconsider reliability (qualitative) and accuracy (quantitative)
― Australia – third-party validation every three yearsprompted by drought (2004)
― Quebec – minimum night flow analysisannual Ministry validation interview with all agencies
Audit collection programs also exist in Austria, England, and Denmark.
-
What Does Validation Look Like in the US?
― Georgia – level 1 validation is required
started in 2012; first three years of data were validated by a state-funded third party
― California – level 1 validation is requiredstarted in 2017; first year of data was validated by state-funded third party
― AWWA WADI – third-party validation of annual national dataset
started before validation methodology had been standardized
Additional states are considering validation programs to improve the water audit data informing resource decisions.
-
Programmatic Recommendations
― Standardize water audit data collection and validation.
― Insist on transparent and neutral data validity grades.
― Approach regulating data validity grades and scores with caution.
― Focus on the role of data validity in water loss control and utility management.
― Establish a legislative mandate and funding.
-
What Does Validation Accomplish?
Validation aims to:
1. Identify and appropriately correct errors in data and application of methodology.
2. Evaluate and communicate uncertainty in water audit data inputs.
-
How Much Effort?
Level 1: data validity and methodology interview
Level 2: desktop analysis and raw data work
Level 3: field data collection
Validation doesn’t necessarily fix all errors –
auditing and validation are retrospective.
-
Level 1 Validation
Goals: Confirm interpretation of methodologyIdentify evident errorsAssign appropriate data validity grades
Outcomes: Appropriate data validity gradesRecommendations for further validation
Limitations: Does not correct errors in raw dataDoes not study instrument performance
-
Level 1 Validation
The Validator:
― Knowledgeable – understands what can go wrong
― Objective – keeps the goal in mind
― Diplomatic – avoids blaming
― Systematic – doesn’t miss anything
Not the auditor!
-
Level 1 Validation
The Method:
1. Collect the audit and required supporting documents.
2. Examine initial performance indicators.
3. Validate audit inputs (volumes, values, validity grades).
4. Re-examine final performance indicators.
5. Document results and propose next steps.
-
Level 1 Validation: Collect
The Required Materials:
― A water audit!
― Summary tablesWater supplied – by month and meter
Authorized consumption – by month and account class
― Supply meter test and calibration results (if applicable)
― Whatever else the audit compiler can pass along!
-
Level 1 Validation: Examine
Financial Indicators Check
NRW volume as % of Water Supplied > 0%
NRW value as % of operating cost < 100%
Operational Efficiency Indicators Check
Apparent Losses per Service Connection per Day > 0
Real Losses per Service Connection per Day > 0
Real Losses per Service Connection per Day per PSI > 0
Infrastructure Leakage Index > 1.0
Anything funky in the performance indicators, audit, or supporting documentation?
-
Level 1 Validation: Validate
For each input:
― How did the auditor arrive at the input? Is this corroborated by supporting documentation?
― How did the auditor interpret methodology?
― Which data validity grade describes operational practices?
― Does anything need to be changed to make the audit more accurate?
-
Level 1 Validation: Re-Examine
After validation, did the performance indicators change?
Financial Indicators Check
NRW volume as % of Water Supplied > 0%
NRW value as % of operating cost < 100%
Operational Efficiency Indicators Check
Apparent Losses per Service Connection per Day > 0
Real Losses per Service Connection per Day > 0
Real Losses per Service Connection per Day per PSI > 0
Infrastructure Leakage Index > 1.0
-
Level 1 Validation: Document
― People – auditor and validator
― Initial performance indicators― Validated performance indicators
― Recommended changes (audit inputs, data validity grades)
― Remaining questions― Recommendations for further validation― Overall impression
-
Success Story: CA Water Loss TAP
2015 CA UWMP submissions:
46% pass rate
-
What Happens After Level 1 Validation?
Audits still aren’t perfect! Inaccuracy can persist.
― Level 2 validation – raw data, data transfer
― Level 3 validation – instrument tests, leakage investigation
― Subsequent audits
― Water loss intervention
-
Thanks!
Comments? Questions? Brilliant ideas?
Lucy Andrews - [email protected]
L. Andrews, K. Gasner, R. Sturm, G. Kunkel, W. Jernigan, and S. Cavanaugh. 2016. Level 1 Water Audit Validation: Guidance Manual (Project 4639A). Denver, CO: Water Research Foundation.
L. Andrews, K. Gasner, R. Sturm, G. Kunkel, W. Jernigan, and S. Cavanaugh. 2017. Utility Water Audit Validation: Principles and Programs (Project 4639B). Denver, CO: Water Research Foundation.