ATTCH 15 11-032redacted ( $ 1 BILLION WRONGLY GIVEN TO DYNCORP SOLE SOURCE )
Guide to Selecting Qualified WET Laboratories Robert N. Brent, Ph.D. DynCorp Science and Engineering...
-
Upload
brian-wood -
Category
Documents
-
view
217 -
download
1
Transcript of Guide to Selecting Qualified WET Laboratories Robert N. Brent, Ph.D. DynCorp Science and Engineering...
Guide to Selecting Qualified WET Laboratories
Robert N. Brent, Ph.D. DynCorp Science and Engineering Group
Why is Lab Selection Important?
You, the permittee, are responsible
for:
• Meeting monitoring frequency requirements
• Certifying the quality of data
– “I certify under penalty of law …the information submitted is, to the best of my knowledge and belief, true, accurate, and complete…”
• Living with the test results
– Test failures can initiate additional testing, TIEs, TREs, enforcement actions, fines, public scrutiny
Why is Lab Selection Important?
You are the one paying
• In a 1998 WERF Study
– C.dubia chronic test: $100 - $2,300
– Fathead minnow chronic test: $238 - $5,500
• With multiple outfalls, monthly testing requirements, and multi-species monitoring, test costs can be significant
– 3 outfalls X 12 samples/yr X 3 species X $1000/test = $108,000
• Know what you are paying for!
Why is Lab Selection Important?
Quality does matter
• There is a wide range of laboratory performance
– Ex: EPA’s Variability Guidance Document
Percentiles of Intralaboratory CVs (coefficient of variation) for C. dubia Chronic Test
25th percentile CV17%
0
5
10
15
20
25
0 10 20 30 40
Test
Res
ult
50th percentile CV27%
0
5
10
15
20
25
0 10 20 30 40
Test
Res
ult
75th percentile CV45%
0
5
10
15
20
25
0 10 20 30 40
Test
Res
ult
A Tale of Two Labs
Ex: WET Interlaboratory Variability Study
0
10
20
30
40
50
60
70
80
1 2 3 4 1 2 3 4SampleLab
Va
ria
bili
ty (
Co
ntr
ol %
CV
)
0
5
10
15
20
25
30
Co
ntr
ol P
erf
orm
an
ce
(M
ea
n)
Variability
ControlPerformance
inva
lid
inva
lid
inva
lid
false +
-------------A------------- -------------B-------------
A Tale of Two Labs
Ex: WET Interlaboratory Variability Study
0
10
20
30
40
50
60
70
80
1 2 3 4 1 2 3 4SampleLab
Va
ria
bili
ty (
Co
ntr
ol %
CV
)
0
5
10
15
20
25
30
Co
ntr
ol P
erf
orm
an
ce
(M
ea
n)
Variability
ControlPerformance
inva
lid
inva
lid
inva
lid
false +
-------------A------------- -------------B-------------
$936 / test
$936 / test
$500 / test
$500 / test
Obstacles to Selecting Qualified Labs
Procurement Regulations
• Many permittees are required to select lowest responsive and responsible bidder
• This doesn’t always mean selecting lowest bidder
– Must be responsive to everything that you request in the solicitation
– Must be responsible for delivering exactly what you specify in the contract
• Solution: You determine the exact specifications for what you want
– Government Ex: $600 hammer
– WET Ex: lab must demonstrate intralab variability of <27%
• Talk to your procurement specialist about incorporating detailed specifications into the lab solicitation process
Obstacles to Selecting Qualified Labs
Insufficient knowledge and resources for identifying qualified laboratories
• Solution: This workshop will provide
– Practical tools for evaluating laboratory quality
– Things to look for in a qualified lab
– Questions to ask
– Sample test data sets to quiz your lab
General Evaluation Criteria
Capacity
• Will the lab be able to test your samples when you need them tested?
• What is the labs maximum and typical capacity (tests per week)?
– Look for maximum capacity to greatly exceed typical capacity
– Quality usually decreases when operating near capacity limits
• What limits capacity? (space, staff, organisms, equipment)
– Look for limits that can be easily remedied
General Evaluation Criteria
Staff
• What is the level of experience and education of staff from top to bottom?
– Remember: it is the staff at the bottom (technicians) that will have the most contact and interaction with your samples
– Experienced and well-trained staff is critical to consistently generating high quality data
• What is the turnover rate?
– Look for low turnover at the top
– Turnover at the bottom may be high, but look for rigorous training program
General Evaluation Criteria
Organizational Structure
• What is the chain of command?
• Who will you be interacting with?
– Look for a dedicated account manager
– Look for a laboratory contact as well
Certification
• Does the lab carry any State or national certifications?
– Not all States have lab certification programs, but beware of labs without certification in a State where there is a certification program
General Evaluation Criteria
Historical Performance
• How many tests does the lab run annually?
– Look for 50 – several hundred (for common test methods)
• What percentage of tests were successfully completed (met TAC) without retesting?
– Look for 90 -100%
• How many other clients do they have with similar waste streams (municipal, industrial, etc.) to yours?
– Familiarity with similar waste streams may lead to better, quicker problem resolution
General Evaluation Criteria
Reporting
• Does the lab’s reporting standards and format meet the requirements of your regulatory authority?
• Does the lab’s reporting format meet your needs
– Look for simple but detailed reports
Ex: You should be able to find the test result (NOEC, IC25, etc.) within 60 seconds
Ex: Report should be detailed enough for someone to completely recalculate the test results
Ex: You should be able to find the temperature of treatment A on Day 4 of the test
General Evaluation Criteria
Awareness
• Is the lab aware of recent developments in WET testing? – Look for lab to be aware of:
Proposed WET method changes and upcoming new versions of WET manuals
WET Method Guidance Document (EPA 821/B-00/004)
WET Variability Guidance Document (EPA 833/R-00/003)
Quality Control Evaluation Criteria
Organism Source and Quality
• Where does the lab get their organisms?
– In-house cultures vs. commercial suppliers
– Look for consist source
• How many organisms are available on a daily basis?
– Look for capacity to exceed usage
• What QC measures are used to assess organism health?
– Look for reference toxicant testing and other measures (reproduction and survival in cultures)
• How often are culture crashes experienced?
Quality Control Evaluation Criteria
Dilution Water Source and Quality
• What dilution water types does the lab use?
• Where does the dilution water come from or how is it prepared?
– Look for consistent high quality source
– Look for experience with different dilution water types
• How does the lab assess dilution water quality?
– Look for chemical testing and toxicity testing of new dilution water batches
Quality Control Evaluation Criteria
Reference Toxicant Control Charts
• Reference toxicant control charts are one of the best ways to assess laboratory performance (if you know what to look for)
• Labs are required to conduct monthly reference toxicant tests
• Test results are plotted on a chart to show variability over time
0
0.5
1
1.5
7/24/98 2/9/99 8/28/99 3/15/00 10/1/00 4/19/01
Date
Ref
. To
x. C
on
c. IC25
Mean
-2STD
+2STD
Evaluating Reference Toxicant Control Charts
Look for control charts of LC50 (for acute methods) or IC25 (for chronic methods), not NOECs
Look for 20 points
Look for curved limits
0
0.5
1
1.5
7/24/98 2/9/99 8/28/99 3/15/00 10/1/00 4/19/01
Date
Ref
.To
x. C
on
c. IC25
Mean
-2STD
+2STD
Evaluating Reference Toxicant Control Charts
Look at scale
0
0.5
1
1.5
7/24/98 2/9/99 8/28/99 3/15/00 10/1/00 4/19/01
Date
Re
f. T
ox
. Co
nc
. IC25
Mean
-2STD
+2STD
-0.50
0.51
1.52
2.53
7/24/98 2/9/99 8/28/99 3/15/00 10/1/00 4/19/01
Date
Re
f. T
ox
. Co
nc
. IC25
Mean
-2STD
+2STD
0
0.5
1
1.5
2
7/24/98 2/9/99 8/28/99 3/15/00 10/1/00 4/19/01
Date
Re
f. T
ox
. Co
nc
. IC25
Mean
-2STD
+2STD
0
0.5
1
1.5
2
7/24/98 2/9/99 8/28/99 3/15/00 10/1/00 4/19/01
DateR
ef.
To
x. C
on
c. IC25
Mean
-2STD
+2STD
DifferentScale
SameScale
Evaluating Reference Toxicant Control Charts
Look at CV
• CV = Standard deviation / Mean * 100%
• Compare CV to national percentiles (Variability Guidance Doc Table B-1)
0
0.5
1
1.5
2
7/24/98 2/9/99 8/28/99 3/15/00 10/1/00 4/19/01
Date
Re
f. T
ox
. Co
nc
. IC25
Mean
-2STD
+2STD
0
0.5
1
1.5
2
7/24/98 2/9/99 8/28/99 3/15/00 10/1/00 4/19/01
Date
Re
f. T
ox
. Co
nc
. IC25
Mean
-2STD
+2STD
CV = 17% CV = 40%
0
0.5
1
1.5
2
7/24/98 2/9/99 8/28/99 3/15/00 10/1/00 4/19/01
Date
Re
f. T
ox
. Co
nc
. IC25
Mean
-2STD
+2STD
Evaluating Reference Toxicant Control Charts
Look for results beyond control limits
• 1 in 20 will exceed limits by chance alone
• Beware of >2 exceedances in 20
Quality Control Evaluation Criteria
Control Charts for Additional QC Measures
• Look for control charts of:
– Control performance over time (i.e., control reproduction)
– Control CV over time
– PMSDs (percent minimum significant difference) over time
• Compare with national percentiles from Variability Guidance Document (Tables B-7 to B-8)
Evaluation of PMSD Control Charts
Look for PMSD mean versus national percentiles
Look for PMSDs that exceed 90th percentile
0
10
20
30
40
50
7/24/98 12/6/99 4/19/01 9/1/02
Date
PM
SD
(%
)
PMSD
Mean
50th %tile
90th %tile
Quality Control Evaluation Criteria
Statistical Analysis and Endpoint Calculation
• Can the laboratory correctly perform the recommended statistics and properly calculate test endpoints?
– EPA found that a large percentage of laboratories in the WET Interlaboratory Variability Study made one or more errors in the calculation of test results
• Test laboratories with sample data sets
– 3 sample data sets provided
– Ask laboratories to calculate LC50, survival NOEC, sublethal NOEC, and IC25 using the recommended statistical flowcharts and concentration-response evaluation guidance (where appropriate)
Sample Test Data Set #1
Sample Test Data Set #1 - Ceriodaphnia dubia Survival and Reproduction Test
Survival NOEC 25LC50 40.613
Reproduction NOEC 12.5Reproduction IC25 22.969
1-tail, 0.05 levelof significance
0
5
10
15
20
25
30
D-C
ontr
ol
6.25
12.5
*25 50 100
Da
y 6
Re
pro
du
cti
on
Sample Test Data Set #2
Requires use of concentration-response guidance
Sample Test Data Set #2 - Ceriodaphnia dubia Survival and Reproduction Test
Survival NOEC 100LC50 >100
Reproduction NOEC 100Reproduction IC25 >100*
* IC25 calculated as 86.82, but based on concentration-response evaluation, this result was determined to be anomalous and IC25 reported as >100
0
5
10
15
20
25
30
35
40
D-C
ontr
ol
6.2
5
12.5 25
50
100
Da
y 6
Re
pro
du
cti
on
Sample Test Data Set #3
Requires use of concentration-response guidance
Sample Test Data Set #3 - Fathead Minnow Larval Survival and Growth Test
Survival NOEC 50LC50 51.404
Growth NOEC 50*Growth IC25 51.446
* 6.25% treatment was significantly different from control, but based on concentration-response evaluation, this treatment was determined to be anomalous and NOEC reported as highest concentration not significantly different from the control
0
0.1
0.2
0.3
0.4
0.5
0.6
D-C
ontr
ol
*6.2
5
12.5 25 50 100
7 D
ay B
iom
ass
Conclusions
It’s your money and your reputation at stake:
• Choose a quality laboratory
When you find a quality laboratory, stick with them
When you don’t know, ask