Building Credibility: Quality Assurance & Quality Control ...€¦ · Building Credibility: Quality...
Transcript of Building Credibility: Quality Assurance & Quality Control ...€¦ · Building Credibility: Quality...
![Page 1: Building Credibility: Quality Assurance & Quality Control ...€¦ · Building Credibility: Quality Assurance & Quality Control for Volunteer Monitoring Programs Elizabeth Herron](https://reader031.fdocuments.us/reader031/viewer/2022013010/5faabb2736ebb305481b746b/html5/thumbnails/1.jpg)
Building Credibility: Quality Assurance & Quality Control for Volunteer Monitoring Programs
Elizabeth Herron URI Watershed Watch/ National Facilitation of CSREES Volunteer Monitoring
Ingrid HarraldCook Inletkeeper
![Page 2: Building Credibility: Quality Assurance & Quality Control ...€¦ · Building Credibility: Quality Assurance & Quality Control for Volunteer Monitoring Programs Elizabeth Herron](https://reader031.fdocuments.us/reader031/viewer/2022013010/5faabb2736ebb305481b746b/html5/thumbnails/2.jpg)
Purpose of this Workshop
Overview of QA/QC conceptsHow they apply to volunteer monitoringNot a workshop on writing a QAPPPlease attend “Assuring Credible Volunteer Data”Room A6, 1:30 – 3:00for more details
![Page 3: Building Credibility: Quality Assurance & Quality Control ...€¦ · Building Credibility: Quality Assurance & Quality Control for Volunteer Monitoring Programs Elizabeth Herron](https://reader031.fdocuments.us/reader031/viewer/2022013010/5faabb2736ebb305481b746b/html5/thumbnails/3.jpg)
Workshop AgendaVERY Brief IntroductionsQuality Assurance – Before/Planning
QAPP Quality Control – During/Implementation
QC Tools (Including examples from the group)
Quality Assessment – After/AssessingDiscussion – What can we do to better demonstrate the credibility of volunteer generated monitoring data??
![Page 4: Building Credibility: Quality Assurance & Quality Control ...€¦ · Building Credibility: Quality Assurance & Quality Control for Volunteer Monitoring Programs Elizabeth Herron](https://reader031.fdocuments.us/reader031/viewer/2022013010/5faabb2736ebb305481b746b/html5/thumbnails/4.jpg)
BRIEF Introductions
![Page 5: Building Credibility: Quality Assurance & Quality Control ...€¦ · Building Credibility: Quality Assurance & Quality Control for Volunteer Monitoring Programs Elizabeth Herron](https://reader031.fdocuments.us/reader031/viewer/2022013010/5faabb2736ebb305481b746b/html5/thumbnails/5.jpg)
Quality is Assured through:
Monitoring multiple indicatorsAdhering to established proceduresTrainingRepetition Routine samplingQA/QC field and laboratory testing
The most important factor determining the level of quality is the cost of being
wrong.
![Page 6: Building Credibility: Quality Assurance & Quality Control ...€¦ · Building Credibility: Quality Assurance & Quality Control for Volunteer Monitoring Programs Elizabeth Herron](https://reader031.fdocuments.us/reader031/viewer/2022013010/5faabb2736ebb305481b746b/html5/thumbnails/6.jpg)
Data Quality System
Before - Plan During –Implement
After - Assess
Quality Assurance
Quality Control Quality Assessment
Study DesignQuality Assurance Project PlanDevelop training program and materials
TrainingFollow WRITTEN monitoring manualFollow standard operating procedures (SOPs)Document changes
Data proofing/reviewOutside performance evaluationReconcile data with objectivesRevise SOPs as needed
![Page 7: Building Credibility: Quality Assurance & Quality Control ...€¦ · Building Credibility: Quality Assurance & Quality Control for Volunteer Monitoring Programs Elizabeth Herron](https://reader031.fdocuments.us/reader031/viewer/2022013010/5faabb2736ebb305481b746b/html5/thumbnails/7.jpg)
Study Design and QA/QC: Important Questions to Consider
Why do you want to monitor?Who will use the data?How will the data be used?How good do the data need to be?What resources are available?What type of monitoring will you do?
Modified from EPA Volunteer Stream Monitoring Methods
![Page 8: Building Credibility: Quality Assurance & Quality Control ...€¦ · Building Credibility: Quality Assurance & Quality Control for Volunteer Monitoring Programs Elizabeth Herron](https://reader031.fdocuments.us/reader031/viewer/2022013010/5faabb2736ebb305481b746b/html5/thumbnails/8.jpg)
Study Design: Matrix of Monitoring Activitieso Monitoring activities / Type of monitoringo Data objectiveso Example activitieso Equipment and supplieso Education and trainingo Frequency of monitoringo QA/QC level and standardso Duration of monitoring o Intensity of analysis / Complexity of approach
![Page 9: Building Credibility: Quality Assurance & Quality Control ...€¦ · Building Credibility: Quality Assurance & Quality Control for Volunteer Monitoring Programs Elizabeth Herron](https://reader031.fdocuments.us/reader031/viewer/2022013010/5faabb2736ebb305481b746b/html5/thumbnails/9.jpg)
![Page 10: Building Credibility: Quality Assurance & Quality Control ...€¦ · Building Credibility: Quality Assurance & Quality Control for Volunteer Monitoring Programs Elizabeth Herron](https://reader031.fdocuments.us/reader031/viewer/2022013010/5faabb2736ebb305481b746b/html5/thumbnails/10.jpg)
Education / Awareness
Problem ID,Assess
Impairment,Local
Decisions
Legal & Regulatory
Increasing Time Increasing Time -- Rigor Rigor -- QA QA -- Expense $$Expense $$
Geoff Dates, River Network
Selecting Methods :The Continuum of Monitoring Data Use
![Page 11: Building Credibility: Quality Assurance & Quality Control ...€¦ · Building Credibility: Quality Assurance & Quality Control for Volunteer Monitoring Programs Elizabeth Herron](https://reader031.fdocuments.us/reader031/viewer/2022013010/5faabb2736ebb305481b746b/html5/thumbnails/11.jpg)
QA/QC: Important Questions to Consider
What will the data be used for?Who will use the data?What quality do your data need to be?
PrecisionBiasAccuracyComparabilityCompletenessRepresentativeSensitivity
![Page 12: Building Credibility: Quality Assurance & Quality Control ...€¦ · Building Credibility: Quality Assurance & Quality Control for Volunteer Monitoring Programs Elizabeth Herron](https://reader031.fdocuments.us/reader031/viewer/2022013010/5faabb2736ebb305481b746b/html5/thumbnails/12.jpg)
QA/QC: Precision• The degree of agreement among
repeated measurements of the same characteristic or parameter
• Tells you how consistent or reproducible your methods are by showing you how close your measurements are to each other
• Measured by standard deviation and coefficient of variation
XX X
X
X XXX
![Page 13: Building Credibility: Quality Assurance & Quality Control ...€¦ · Building Credibility: Quality Assurance & Quality Control for Volunteer Monitoring Programs Elizabeth Herron](https://reader031.fdocuments.us/reader031/viewer/2022013010/5faabb2736ebb305481b746b/html5/thumbnails/13.jpg)
QA/QC: Bias• Systematic or persistent
distortion of a measurement process which causes errors in direction
• Produces measurements consistently higher or lower than the samples true value
X
XX
X
XXXX
![Page 14: Building Credibility: Quality Assurance & Quality Control ...€¦ · Building Credibility: Quality Assurance & Quality Control for Volunteer Monitoring Programs Elizabeth Herron](https://reader031.fdocuments.us/reader031/viewer/2022013010/5faabb2736ebb305481b746b/html5/thumbnails/14.jpg)
QA/QC: Accuracy• Accuracy (or bias) is a
measure of confidence that describes how close a measurement is to its “true” or expected value
• Measured by (true value-found value)/(true x 100)
XX XX
![Page 15: Building Credibility: Quality Assurance & Quality Control ...€¦ · Building Credibility: Quality Assurance & Quality Control for Volunteer Monitoring Programs Elizabeth Herron](https://reader031.fdocuments.us/reader031/viewer/2022013010/5faabb2736ebb305481b746b/html5/thumbnails/15.jpg)
QA/QC: Comparability
Comparability is the extent to which your data can be compared to directly to:
• Past data from the current project or• Data from another study• Prescriptive• Performance
![Page 16: Building Credibility: Quality Assurance & Quality Control ...€¦ · Building Credibility: Quality Assurance & Quality Control for Volunteer Monitoring Programs Elizabeth Herron](https://reader031.fdocuments.us/reader031/viewer/2022013010/5faabb2736ebb305481b746b/html5/thumbnails/16.jpg)
How Do You Define Comparability? And What is the Cost/Benefit of Getting Right?
![Page 17: Building Credibility: Quality Assurance & Quality Control ...€¦ · Building Credibility: Quality Assurance & Quality Control for Volunteer Monitoring Programs Elizabeth Herron](https://reader031.fdocuments.us/reader031/viewer/2022013010/5faabb2736ebb305481b746b/html5/thumbnails/17.jpg)
QA/QC: Completeness
• Completeness is the comparison between the amount of data you planned to collect versus how much usable data you collected
![Page 18: Building Credibility: Quality Assurance & Quality Control ...€¦ · Building Credibility: Quality Assurance & Quality Control for Volunteer Monitoring Programs Elizabeth Herron](https://reader031.fdocuments.us/reader031/viewer/2022013010/5faabb2736ebb305481b746b/html5/thumbnails/18.jpg)
QA/QC:RepresentativenessThe extent to which measurements actually
depict the true environmental condition or population being evaluated
• data collected at a site just downstream of an outfall may not be representative of an entire stream, but it may be representative of sites just below outfalls
![Page 19: Building Credibility: Quality Assurance & Quality Control ...€¦ · Building Credibility: Quality Assurance & Quality Control for Volunteer Monitoring Programs Elizabeth Herron](https://reader031.fdocuments.us/reader031/viewer/2022013010/5faabb2736ebb305481b746b/html5/thumbnails/19.jpg)
QA/QC: Sensitivity
• The sensitivity of a given method• pH assessment using litmus paper
(acid vs. base)• pH assessment using pH paper
with unit intervals(pH 3,4,5,etc.)
• pH color comparator(pH 3.5, 8.0, 9.5, etc.)
• pH meter (pH to 0.1 or 0.01)(caution, standard buffers are only sensitive to 2 decimal places)
![Page 20: Building Credibility: Quality Assurance & Quality Control ...€¦ · Building Credibility: Quality Assurance & Quality Control for Volunteer Monitoring Programs Elizabeth Herron](https://reader031.fdocuments.us/reader031/viewer/2022013010/5faabb2736ebb305481b746b/html5/thumbnails/20.jpg)
QAPPQuality Assurance Project Plan
Documents the Why, Who, What, When, Where and How of your monitoring effortUSEPA approved QAPP has fairly rigid format
Guidance documents are availableApproved Volunteer QAPPs on-line
Don’t Just Copy – the Process is as Important as the Document
![Page 21: Building Credibility: Quality Assurance & Quality Control ...€¦ · Building Credibility: Quality Assurance & Quality Control for Volunteer Monitoring Programs Elizabeth Herron](https://reader031.fdocuments.us/reader031/viewer/2022013010/5faabb2736ebb305481b746b/html5/thumbnails/21.jpg)
URI Watershed WatchAnalytical Laboratory QAPPField Monitoring QAPPProject Specific QAPPs
Tap Water QAPPBlock Island / Green Hill Pond QAPP
www.uri.edu/ce/wq/ click on Watershed Watch
![Page 22: Building Credibility: Quality Assurance & Quality Control ...€¦ · Building Credibility: Quality Assurance & Quality Control for Volunteer Monitoring Programs Elizabeth Herron](https://reader031.fdocuments.us/reader031/viewer/2022013010/5faabb2736ebb305481b746b/html5/thumbnails/22.jpg)
Cook Inlet Keepers
USEPA approved marine ecosystem baseline monitoring planConcise (20 pgs)Laboratory QAPP alsoProject specific QAPPs
www.inletkeeper.org/
![Page 23: Building Credibility: Quality Assurance & Quality Control ...€¦ · Building Credibility: Quality Assurance & Quality Control for Volunteer Monitoring Programs Elizabeth Herron](https://reader031.fdocuments.us/reader031/viewer/2022013010/5faabb2736ebb305481b746b/html5/thumbnails/23.jpg)
Other Resources
Virginia Dept of Environmental Quality, QAPP/Monitoring Plan templateswww.deq.virginia.gov/cmonitor/grant.html
National Facilitation Project – Getting Started and Building Credibility modules, active links to multiple siteswww.usawaterquality.org/volunteer
![Page 24: Building Credibility: Quality Assurance & Quality Control ...€¦ · Building Credibility: Quality Assurance & Quality Control for Volunteer Monitoring Programs Elizabeth Herron](https://reader031.fdocuments.us/reader031/viewer/2022013010/5faabb2736ebb305481b746b/html5/thumbnails/24.jpg)
QA/QC: Quality Control - Assuring Accuracy• Analyze blanks (field and lab)• Analyze samples of known concentrations
(standards)• Participate in performance audits (from
outside source)• Collect & analyze duplicates• Replicate 10-20% of samples• New analysis >= 7 times• Check against other methods
![Page 25: Building Credibility: Quality Assurance & Quality Control ...€¦ · Building Credibility: Quality Assurance & Quality Control for Volunteer Monitoring Programs Elizabeth Herron](https://reader031.fdocuments.us/reader031/viewer/2022013010/5faabb2736ebb305481b746b/html5/thumbnails/25.jpg)
QA/QC: CE Programs’ QA/QC Procedures - 21 responding programs
02468
1012
Replica
te sa
mples
Duplica
te mea
surem
ents
Known s
tanda
rds
Field b
lanks
Analyt
ical re
plica
tes
Calibra
tion s
ample
s
Pos.N
eg pl
ates
Externa
l field
dupli
cates
Split s
ample
s (2n
d lab
)
Duplica
te sa
mples (
2nd l
ab)
Spiked
sample
s
Calibra
tion b
lanks
Unkno
wn stan
dards
Num
ber o
f Pro
gram
s
QC MethodNational Facilitation Project Inquiry 2002
![Page 26: Building Credibility: Quality Assurance & Quality Control ...€¦ · Building Credibility: Quality Assurance & Quality Control for Volunteer Monitoring Programs Elizabeth Herron](https://reader031.fdocuments.us/reader031/viewer/2022013010/5faabb2736ebb305481b746b/html5/thumbnails/26.jpg)
US Geological Survey
National Field Quality Assurance ProgramNational Laboratory Quality Assurance ProgramA free standards program
USGS pays for shippingApproximate 3 week turn around time for resultsNeed to register volunteers/program in advance
Contact: Jonathan Currier
![Page 27: Building Credibility: Quality Assurance & Quality Control ...€¦ · Building Credibility: Quality Assurance & Quality Control for Volunteer Monitoring Programs Elizabeth Herron](https://reader031.fdocuments.us/reader031/viewer/2022013010/5faabb2736ebb305481b746b/html5/thumbnails/27.jpg)
Other Quality Control Tools??
Suggestions?Experiences
GoodBad
Training and recertification programs?
![Page 28: Building Credibility: Quality Assurance & Quality Control ...€¦ · Building Credibility: Quality Assurance & Quality Control for Volunteer Monitoring Programs Elizabeth Herron](https://reader031.fdocuments.us/reader031/viewer/2022013010/5faabb2736ebb305481b746b/html5/thumbnails/28.jpg)
Quality AssessmentReviewing the Data
Proof the data – compare what was entered versus what found
Raw data versus summarizedKeep original hard copies
Assess the data – does it make sense??Compare to data quality objectivesOutside performance evaluation
![Page 29: Building Credibility: Quality Assurance & Quality Control ...€¦ · Building Credibility: Quality Assurance & Quality Control for Volunteer Monitoring Programs Elizabeth Herron](https://reader031.fdocuments.us/reader031/viewer/2022013010/5faabb2736ebb305481b746b/html5/thumbnails/29.jpg)
“Where such situations seem to exist (that is, identical results) either the measurement is not sensitive enough to detect differences or the person making the measurement is not performing it properly.”
QA/QC: Variability Happens
IDENTICAL RESULTS DO NOT
MEAN PERFECTION
![Page 30: Building Credibility: Quality Assurance & Quality Control ...€¦ · Building Credibility: Quality Assurance & Quality Control for Volunteer Monitoring Programs Elizabeth Herron](https://reader031.fdocuments.us/reader031/viewer/2022013010/5faabb2736ebb305481b746b/html5/thumbnails/30.jpg)
“Any conclusions reached as a result of tests or analytical determinations are charged with uncertainty. No test or analytical method is so perfect, so unaffected by the environment or other external contributing factors, that it will always produce exactly the same test or measurement result or value.”
QA/QC: The Monitoring Conundrum
VARIABILITY HAPPENS
![Page 31: Building Credibility: Quality Assurance & Quality Control ...€¦ · Building Credibility: Quality Assurance & Quality Control for Volunteer Monitoring Programs Elizabeth Herron](https://reader031.fdocuments.us/reader031/viewer/2022013010/5faabb2736ebb305481b746b/html5/thumbnails/31.jpg)
The measurement of a single sample tells us nothing about its environment, only about the sample itself.
QA/QC:Quality is Mostly Assured by Repetition
![Page 32: Building Credibility: Quality Assurance & Quality Control ...€¦ · Building Credibility: Quality Assurance & Quality Control for Volunteer Monitoring Programs Elizabeth Herron](https://reader031.fdocuments.us/reader031/viewer/2022013010/5faabb2736ebb305481b746b/html5/thumbnails/32.jpg)
The most important factor determining the level of quality is the cost of being wrong.
QA/QC
![Page 33: Building Credibility: Quality Assurance & Quality Control ...€¦ · Building Credibility: Quality Assurance & Quality Control for Volunteer Monitoring Programs Elizabeth Herron](https://reader031.fdocuments.us/reader031/viewer/2022013010/5faabb2736ebb305481b746b/html5/thumbnails/33.jpg)
It meansdelivering on your promises,
no matter how small or large.-Meg Kerr, River Rescue
It meansdelivering on your promises,
no matter how small or large.-Meg Kerr, River Rescue
Credibility doesn’t mean having the most exacting techniques. Credibility doesn’t mean having the most exacting techniques.
![Page 34: Building Credibility: Quality Assurance & Quality Control ...€¦ · Building Credibility: Quality Assurance & Quality Control for Volunteer Monitoring Programs Elizabeth Herron](https://reader031.fdocuments.us/reader031/viewer/2022013010/5faabb2736ebb305481b746b/html5/thumbnails/34.jpg)
Questions
![Page 35: Building Credibility: Quality Assurance & Quality Control ...€¦ · Building Credibility: Quality Assurance & Quality Control for Volunteer Monitoring Programs Elizabeth Herron](https://reader031.fdocuments.us/reader031/viewer/2022013010/5faabb2736ebb305481b746b/html5/thumbnails/35.jpg)
Discussion
We know that volunteer programs can produce credible data – so how do we demonstration that more effectively??
![Page 36: Building Credibility: Quality Assurance & Quality Control ...€¦ · Building Credibility: Quality Assurance & Quality Control for Volunteer Monitoring Programs Elizabeth Herron](https://reader031.fdocuments.us/reader031/viewer/2022013010/5faabb2736ebb305481b746b/html5/thumbnails/36.jpg)
THANK – YOU!
Remember to fill out your workshop evaluation formStop by the Volunteer Monitoring exhibit booth and complete a Needs Assessment formCome to tonight’s Volunteer Monitors Coordinators Meeting
5:15 – 6:30 Room C1