Percentile Standards: Rationale,

Implementation and a Few

Misconceptions

Graham McBride

Graham.McBride@niwa.co.nz

Hamilton

1

Colours

Key terms, blue

Bad ideas, red

Good things, green

2

Why consider statistics now?

To put a monitoring-for-compliance on a sound footing

Identify efficient and transparent approaches

Avoid misconceptions

– e.g., need at least 19 samples to estimate a 95%ile

Examine and cope with “tricky issues”

– rolling compliance assessment periods (drop a month, add another)

“The statistically correct method” doesn’t exist

– Need to identify the most appropriate and consistent one

– Plenty of inappropriate ones!

Compliance assessment is a statistical issue (among others)

3

95th percentile vs. 95% confidence

Can get confusing (two “95”s)! Just know that we can have

1) The 95th percentile (= “95%ile”)

o The value of a discharge’s water quality variable that is not exceeded for 95% of the time

or, equivalently

o The value of a discharge’s water quality variable that is exceeded for 5% of the time

2) With 95% confidence

o That the discharge’s 95%ile water quality limit is below the consent’s limit

4

Examples

MoH Drinking-Water Standards’ E. coli compliance rules are based on a requirement that sampling be sufficient to allow 95% confidence that a stated limit (zero E. coli) has not been exceeded for at least 95% of the time

– or that the limit (zero E. coli) has been exceeded for no more than 5% of the time

Those Standard’s turbidity rules also use 99%iles. They are based on a requirement that sampling be sufficient to allow 95% confidence that a turbidity limit has not been exceeded for at least 99% of the time

– or that the limit has been exceeded for no more than 1% of the time

WaterNZ/MfE New Zealand Municipal Wastewater Monitoring Guidelines, Table 13.2. Based on 90% confidence (more later)

5

Why percentiles?

For most water quality variables a feasible maximum will inevitably be exceeded once enough samples are taken

– a deterrent to sampling

– and if exceeded, so what?

Medians are useful for stating expected performance of a WWTP in the normal-course-of-events

Other percentiles (e.g., 90%ile, 95%ile, 98%ile) useful in accounting for unusual events

– not a deterrent to sampling

o estimated 95%iles tend to decrease with sample size.

6

Fundamentals

A discharge percentile compliance rule should state:

1. A compliance assessment period

2. The sample statistics to be used

o tolerance limit, percentile, number of exceedances,…

3. Compliance definition (in terms of those statistics)

7

Fundamentals

A discharge percentile compliance rule should state:

1. A compliance assessment period

2. The sample statistics to be used

o tolerance limit, percentile, number of exceedances,…

3. Compliance definition (in terms of those statistics)

The rule should be derived from percentiles of time

– Statistics 101: Take samples to estimate the frequency characteristics of a population

– Therefore must consider “statistical sampling error”

o “Slings and arrows of outrageous fortune” (pers. comm., W. Shakespear)

8

So? (using 90%ile as an example)

The compliance rule for a 90%ile discharge standard isn’t necessarily based on the 90%ile of the samples

– Only would do so if an “even-handed” burden-of-proof were adopted

– Precautionary ⟹ higher sample percentile (minimise consumer’s risk)

– Permissive ⟹ lower sample percentile (minimise producer’s risk)

9

So? (using 90%ile as an example)

The compliance rule for a 90%ile discharge standard isn’t necessarily based on the 90%ile of the samples

– Only would do so if an “even-handed” burden-of-proof were adopted

– Precautionary ⟹ higher sample percentile (minimise consumer’s risk)

– Permissive ⟹ lower sample percentile (minimise producer’s risk)

More formally (using 90%iles as an example)

– Precautionary ⟹ compare discharge percentile limit with upper one-sided 95% tolerance limit

– Permissive ⟹ compare discharge percentile limit with lower one-sided tolerance limit

10

So? (using 90%ile as an example)

11

90%ile

Permissive

Upper one-sided 95% tolerance

interval on the 90%ile

Precautionary

Lower one-sided 95% tolerance

interval on the 90%ile

So? (using 90%ile as an example)

12

90%ile

Permissive

Upper one-sided 95% tolerance

interval on the 90%ile

Precautionary

Lower one-sided 95% tolerance

interval on the 90%ile

So? (using 90%ile as an example)

13

90%ile

Permissive

Upper one-sided 95% tolerance

interval on the 90%ile

Precautionary

Lower one-sided 95% tolerance

interval on the 90%ile

So? (using 90%ile as an example)

14

90%ile

Permissive

Upper one-sided 95% tolerance

interval on the 90%ile

Precautionary

Lower one-sided 95% tolerance

interval on the 90%ile

Interpretation

Permissive: I would only be 95% confident that the red line has been breached if

calculations from my samples result in the upper blue line.

Precautionary: I would only be 95% confident that the red line has not been breached if

calculations from my samples result in the lower blue line.

But…

There are many ways to estimate percentiles

– Excel’s is very unusual

– Most software manuals don’t tell you what estimation method is used

Can be issues in calculating tolerance limits on percentiles

– Often based on assumptions that are hard to test

o especially concerning the water quality variable’s statistical distribution

– Simpler methods can be “biased”

You often don’t know if a breach has occurred until the end of the compliance assessment period

– Can’t calculate the tolerance limit until all data are to hand 15

A better way

Base the compliance rule on the number of exceedances of the discharge percentile limit from random sampling in an assessment period

– No distributional assumptions needed

– Probability that a single sample exceeds the percentile limit follows the binomial distribution

– Simple!

Next slide contains all the information used to develop the MoH NZ Drinking-Water Standards and the WaterNZ/MfE New Zealand Municipal Wastewater Monitoring Guidelines

– I will demonstrate how that was done (simple reading of the appropriate graph)

16

Sources: : McBride, G.B.; Ellis, J.C. 2001. Confidence of Compliance: a Bayesian approach for percentile

standards. Water Research 35(5): 1117–1124.

McBride, G.B. (2005). Using statistical methods for water quality management, Wiley, NY,

17

Source. MoH (2005). Drinking-water Standards for New Zealand 2005 (Revised 2008). MoH.

http://www.health.govt.nz/publication/drinking-water-standards-new-zealand-2005-revised-2008-0

Drinking-water Standards 95%ile look-up table

Precautionary, minimises consumer’s risk

18

Source: NZWERF/MfE. New Zealand Municipal Wastewater Monitoring Guidelines.

http://www.nzwwa.org.nz/Category?Action=View&Category_id=106#guidelines

,

Municipal Wastewater Monitoring Guideline look-up table

Permissive, minimises producer’s risk

19

Take-home messages

Percentile standards’ compliance rules should be derived from requirements on percentiles of time

Simplest compliance rules are based on number of exceedances of a percentile limit in a required number of samples in an assessment period

You must adopt (and should state) a burden-of-proof

o permissive, even-handed, precautionary

Can infer breach before the end of the assessment period

Simple! (Lookup tables and graphs) 20