Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach
description
Transcript of Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach
![Page 1: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/1.jpg)
Data Quality Indicators(DQIs)
What are they, and how do they affect me?An US-EPA Approach
A
CC
P
S R
![Page 2: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/2.jpg)
DQIs DefinedDQIs are quantitative and qualitative measures of principal quality attributes;Precision;Bias;Representativeness;Comparability;Completeness; andSensitivity.
Quantitative DQIsPrecision, bias, and sensitivity.
Qualitative DQIsrepresentativeness, comparability, and completeness.
![Page 3: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/3.jpg)
The Hierarchy of Quality Terms
DQOs Qualitative and quantitative study objectives
Attributes Descriptive qualitative and quantitative aspects of collected data
DQIs Indicators of the quality attributesMQOs Acceptance criteria for the quality attributes
measured by project DQIs
![Page 4: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/4.jpg)
Precision
Precision is the measure of agreement among repeated measurements of the same property under identical or substantially similar conditions.A precision DQI is a quantitative indicator of the random errors or fluctuations in the measurement process.e.g., standard deviation or variance
![Page 5: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/5.jpg)
BiasBias is systematic or persistent distortion of a measurement process that causes error in one direction.A bias DQI is a quantitative indicator of the magnitude of systematic error resulting from:biased sampling design;calibration errors;response factor shifts;unaccounted-for interferences; andchronic sample contamination.
e.g., instrument reads XX mg/L too high
![Page 6: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/6.jpg)
Accuracy
Accuracy is composed of precision and bias. Accuracy is a measure of the overall agreement of a measurement to a known value: when random errors are tightly controlled, bias dominates the overall accuracy; and
when random errors predominate, variance dominates the overall accuracy.
![Page 7: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/7.jpg)
Influence of Bias and Imprecision on Overall Accuracy
Imprecise andbiased
Imprecise andunbiased
Precise andbiased
Precise andunbiased
![Page 8: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/8.jpg)
RepresentativenessRepresentativeness is the measure of the degree to which data suitably represent a characteristic of a population, parameter variations at a sampling point, a process condition, or an environmental condition.Representativeness DQIs are qualitative and quantitative statements regarding the degree to which data reflect the true characteristics of a well defined population.e.g., these samples are representative of surface soil to be found in a specific area of XX square meters.
![Page 9: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/9.jpg)
ComparabilityComparability is a qualitative expression of the measure of confidence that two or more data sets may contribute to a common analysis.a comparability DQI is a qualitative indicator of the similarity of attributes of data sets. e.g., soil salinity or soil acidity data sets are comparable as they share a common preparation and analytical method operated under similar conditions.
![Page 10: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/10.jpg)
Completeness
Completeness is a measure of the amount of valid data obtained from a measurement system, expressed as a percentage of the number of valid measurements that should have been collected.the DQI for completeness is often expressed as a percentage.e.g., the percentage of valid samples for which data for all analytes of interest were reported.
![Page 11: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/11.jpg)
SensitivitySensitivity is the capability of a method or instrument to discriminate between measurement responses representing different levels of the variable of interest.Sensitivity can be regarded as detection limitbut this term is often used without defining what is intended (minimum detection or quantitation).
A sensitivity DQI describes the capability of measuring a constituent at low levels.a Practical Quantitation Level (PQL) describes the ability to quantify a constituent with known certainty. e.g., a PQL of .05 g/L for mercury represents the level where a precision of +/- 15% can be obtained.
![Page 12: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/12.jpg)
VerificationData verification refers to the procedures needed to ensure that a set of data is a faithful reflection of all the processes and procedures used to generate the data.verification involves the examination of objective evidence that the specified method, procedures, and contractual requirements were fulfilled.
![Page 13: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/13.jpg)
ValidationData validation is an analyte and sample matrix-specific process to determine the analytical quality of a specific data set.validation entails the inspection of data handling practices for deviations from consistency, the review of quality control (QC) information for deviations, assessment of deviations, and assignment of data qualification codes.
Validation can entail the examination of the data with respect to the QA Plan.
![Page 14: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/14.jpg)
IntegrityLack of integrity affects all aspects of data interpretation, especially data used for decision making; andLack of integrity includes:manipulation of QC measurements;Dry-labbing (complete falsification of data);manipulation of results during analysis;failure to conduct required analytical steps; andpost-analysis alteration of results.
![Page 15: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/15.jpg)
After Verification and ValidationThe set of data are then analyzed by comparing the results to the original objectives. In many cases this is a comparison of the results to the DQOs using data quality assessment;Data quality assessment, a five step process:Review of DQOs and sample designPreliminary data reviewSelection of statistical testVerification of assumptionsDrawing conclusions from the data
But that is another course all together!
![Page 16: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/16.jpg)
RepresentativenessStatistical and ConceptualModel-Based Approaches
![Page 17: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/17.jpg)
Representativeness
Representativeness is the measure of the degree to which data suitably represent a characteristic of a population, parameter variations at a sampling point, a process condition, or an environmental condition:Representativeness DQIs are qualitative and quantitative statements regarding the degree to which data reflect the true characteristics of a well- defined population.
![Page 18: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/18.jpg)
What Does "Representativeness" Mean?
Very vaguely defined in working English:"seal of approval" by simple statement of writer.there is an absence of biasing forces.it is a miniature or replica of the population.it is a typical or ideal case.there is wide coverage of a population.it enables good estimation.it is good enough for the purposes of the study.statistically based sampling method.
![Page 19: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/19.jpg)
Different definitions of "Representativeness”
"...expected to exhibit the average properties of the universe or whole" "...should be selected on the basis of spatial and temporal representativeness" "...samples should be representative of daily operations”
![Page 20: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/20.jpg)
Achieving Representativeness Involves a Process
Planning, design, and assessment careful attention to measurement and analytical process;consideration of the size (amount of material) and method for sample collection and handling;
determination of adequate type, location, timing, and number of samples to be taken; and
defensible approach for drawing inferences from sample data to the target population.
Sample design and measurement processes should minimize unintentional bias.
![Page 21: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/21.jpg)
The Process Involves Evaluating Both Micro and Macro Scales
Micro scalehow well measurements taken within a sampling unit reflect that unit (e.g.,"parameter variations at a sampling point")
Macro scaledegree to which measurements from a set of sampling units reflect the population of interest (e.g., "accurately and precisely represent a characteristic of a population")
![Page 22: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/22.jpg)
Micro Scale (Within-Sampling-Unit) Representativeness
An appropriate quality system to ensure quality implementation and sample integrity;Carefully defined sampling units with correct sampling procedures and equipment;Adequate sample support (amount of material) to make inferences about the characteristics within the sampling unit; andAppropriate analytical methods (including sample preparation), designed to achieve MQOs for measurement precision, bias, and sensitivity.
![Page 23: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/23.jpg)
What is a Sampling Unit?A sampling unit (SU) can be defined as the portion of the environment for which a measurement has meaning for its intended use.Defining SUs for a project allows us to communicate more clearly about components of total-study precision.
![Page 24: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/24.jpg)
SUs can vary depending on the specific problem; they can be: as small as the physical sample itself;something encompassing multiple physical samples; or
something much larger.In classical survey design (e.g., opinion survey) the SU is typically an individual.
Specifying Sampling Units
![Page 25: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/25.jpg)
Specifying Sampling Units
SUs are less well defined in other types of surveys (e.g., in a survey to determine soil salinity levels)in this case, a soil sample is much smaller than the area it represents - is the sampling unit topsoil sample, a pedon, or the farm as a whole?
Consider how data will be usedaverage over multiple units, the spatial distribution of units, or some combination?
![Page 26: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/26.jpg)
Default SU Definitionequivalent to the physical sample (soil, water, or plant specimen) taken.
Alternative SU Definitionsunits comprised of multiple samples to obtain enough of the medium to perform all desired analyses.
units of a size adequate to collect multiple specimens (such as composite samples).
units defined to include a group of samples when individual samples are not the unit of interest.
Alternative Sampling Unit Definitions
![Page 27: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/27.jpg)
Choice of Sampling Unit -What Does a Sample Represent?
7.5 cm core
A small farm
1-ha area
![Page 28: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/28.jpg)
Sampling Theory: Within-SU ErrorTo what degree is heterogeneity within a sampling unit inherent?Gy refers to this as the “constitution heterogeneity.” No amount of mixing or homogenization can reduce this.Constitution heterogeneity leads to fundamental errorFundamental errors are negligible for liquids and gases without suspended solids, but are significant in soil and any other solids.
![Page 29: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/29.jpg)
Sampling Theory: Within-SU Error
What is the distribution and variance between small increments of the media?Gy refers to this as “distribution heterogeneity” which reflects the distribution of groups of some number of neighboring fragments.
Grouping and segregation errors result from distribution heterogeneityminimize these errors by taking more increments to form a sample of the required weight
![Page 30: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/30.jpg)
Heterogeneity of Pollutants Can Lead to Sampling Errors
h1 = small scale (random fluctuations)
h2 = large scale (trends, nonrandom, bias)
h3 = cyclic phenomena
h = h1 + h2 + h3
Each of these components of heterogeneity lead to errors
Experiments to characterize these components (using variograms) allow one to optimize a design
![Page 31: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/31.jpg)
Ensure the field sampling protocol does not distort or bias sampleshould be capable of ensuring all parts of the media (e.g., all particle sizes) have the same probability of being included in the increment obtained to form a sample
Ensure the laboratory subsamples represent all the particle-size fractionssubsamples must be large enough (optimal sample weight to accommodate the range of particle sizes
samples and subsamples should be comprised of as many correctly obtained increments as possible
Controlling Sampling Errors
![Page 32: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/32.jpg)
Questions raised by Sampling Theory Related to “Within-SU Error”
What is the correct scale at which to sample?What is the correct protocol for obtaining increments to form samples of the media of interest?
![Page 33: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/33.jpg)
Questions raised by Sampling Theory Related to “Within-SU Error” (cont.)
pilot studies needed to determine the nature of the heterogeneity.If, for example, soil salinity areas are highly clustered on a scale smaller than the scale of real concern, small grabs will reveal varied results
If homogenization and sub-sampling does not remove clustering, representation of a sampling unit from a single sample will not be achievable
sampling protocols should be selected that do not alter the characteristics of the media (e.g., particle- size composition)
![Page 34: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/34.jpg)
Classical Statistical ApproachDefine the population of interestspatial and temporal boundaries; andsampling units
Develop a statistical sampling planprobability-based design, every sampling unit has a known probability of inclusion.
Evaluate process for drawing inferences from datahow well the sampling units selected represent the population under study;
how data will be used to estimate target population parameters such as the mean and variance; and
how well the sampled population provide information on subject in question.
![Page 35: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/35.jpg)
Strategies for Improving Within SU Representativeness
Utilize within-sampling-unit replicationcan reduce the variability of the average by a factor of n-1/2.
Utilize within-sampling-unit compositingincreasing the number of increments in the sample reduces the variability of the unit average.
Increase the sample support area or volumeexpanding the definition of what area or volume the analytical measurement will represent can alleviate small-scale (or short-term) fluctuations.
![Page 36: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/36.jpg)
Statistical Strategies for Improving Between-Sampling-Unit Representativeness
Statistical sampling schemessimple random samplingsystematic (grid) samplingstratified random samplingranked set samplingcluster samplingbetween-sampling-unit composite sampling
![Page 37: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/37.jpg)
Balanced Design to Achieve Representativeness
Understanding the relative contribution of within-sampling-unit and between- sampling-unit variancefocus on components of variance to which the total variability is most sensitive
More samplesto lower between-sampling-unitvariance
More precisemeasurementsto lower within-sampling-unitvariance
![Page 38: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/38.jpg)
Assessing RepresentativenessEvaluating existing datarepresentativeness affects the degree to which a data set can be used for a purpose other than originally intended.
Use of a checklist promotes a thorough evaluation of the attributes of representativeness.
Use of quality assessment samples such as dups, splits, or other replicates can assist in answering questions about within-sampling-unit representativeness.
![Page 39: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/39.jpg)
Important Attributes (Micro-level)Was a rationale provided to support the selection of sampling equipment and handling procedures?correct choice of equipment and handling procedures directly affect degree to which the increments and samples reflect the characteristics of the matrix.
Was the rationale to support selection of analytical methods provided?choice of sample preparation and analytical instrument is critical.
Were samples collected from all selected sampling units? incomplete sampling, if biased due to the lack of completeness, can lead to incorrect conclusions
![Page 40: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/40.jpg)
Important Attributes (Macro-level)Were study objectives adequately defined using the DQO process or equivalent planning process?intended use of data provides the context for evaluating representativeness.
Was the population of interest clearly defined?probability-based designs require the population to be defined as a set of sampling units.
Was the statistical basis for the sampling plan explained (number of samples, their allocation)?representativeness hinges on adequate number of samples.
different sample allocation approaches can maximize effectiveness.
![Page 41: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/41.jpg)
Precision Indicators Reflective of the Data Collection Life Cycle
Planning
ImplementationAssessment
![Page 42: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/42.jpg)
Precision"Precision is the measure of agreement among repeated measurements of the same property under identical or substantially similar conditions."properties in soil studies
concentration of a constituent, say nitrogenphysical measurement (e.g., grain size) of soil media
a precision DQI is a quantitative indicator of the random errors or fluctuations in the measurement process
![Page 43: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/43.jpg)
Common Indicators of PrecisionRangedifference between largest and smallest values
Variance or standard deviationa statistical measure of the spread of data calculated from two or more measured values
the standard deviation is the square root of the variance
Relative rangethe Range divided by the mean of the data set
Relative standard deviation (CV)the standard deviation calculated from two or more values divided by the mean of those values
![Page 44: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/44.jpg)
Framework for EvaluatingIndicators of Precision
A simple model allows us to evaluate the components and indicators of total-study variabilitywithin-sampling-unit variability:
measurement processsmall-scale variabilitysample acquisition
between-sampling-unit variability:
inherent spatial variabilitysampling design error
Total-Study Variability
Within- Sampling-Unit
Variability
Between- Sampling-Unit
Variability
![Page 45: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/45.jpg)
Simple Total-Study Variability Model
Total-Study Variability
Within- Sampling-Unit
Variability
Between- Sampling-Unit
Variability
Small-Scale Variability
(within unit)
Sample Collection and Measurement Process Variability
Inherent Spatial Variability
(among units)
Sampling Design Error
![Page 46: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/46.jpg)
Sampling Units
A sampling unit (SU) can be defined as the portion of the natural environment (soil, water, plant) for which a measurement has meaning for its intended use.Defining SUs for a soil, water or plant sampling project allows us to communicate more clearly about components of total-study precision.
![Page 47: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/47.jpg)
SUs can vary depending on the specific problem; they can be:as small as the physical sample itself;something encompassing multiple physical samples; orsomething much larger.
In classical survey design (e.g., opinion survey) the SU is typically an individual.SUs are less well defined in other types of surveys (e.g., in a survey to determine soil salinity levels).in this case, a soil sample is much smaller than the individual - is the sampling unit the soil sample, the pedon, the farm, or the project?
Consider how data will be used.average over multiple units, the spatial distribution of units, or some combination?
Specifying SUs
![Page 48: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/48.jpg)
Default SU Definitionequivalent to the physical sample taken
Alternative SU Definitionsunits comprised of multiple samples to allow for obtaining enough of the medium to perform all desired analyses
units of a size adequate to collect multiple samples (such as collocated samples)
units uniquely defined to measure properties of interest when a sample is not the unit of interest, nearby samples are highly correlated, or there is an explicit desire to control the precision within the unit
Alternative Sampling Unit Definitions
![Page 49: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/49.jpg)
Evaluating Sampling Unit Definitions
Defining SUs larger than the physical sample has some potential benefits.clarifies whether collocated samples should be treated as additional field samples or replicates;
forces to consider the scale at which measurements have meaning; and
facilitates a more comprehensive consideration of sources of error affecting our understanding of properties of interest, and sources of variability affecting individual measurements.
Most study designs do not account for within-sampling-unit variability in any explicit way.tradeoffs between fewer precise measurements versus more imprecise measurements begin to address the issue.
![Page 50: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/50.jpg)
Sampling Theory Raises Important Questions Related to Within-SU Error
What is the correct scale at which to sample?What is the correct protocol for obtaining samples?pilot studies needed to determine the nature of the heterogeneity if concentration of analytes is highly clustered on a scale smaller than the scale of real concern, small grabs will reveal varied results.
if homogenization and sub-sampling does not remove clustering, representation of a sampling unit from a single sample will not be achievable.
sampling protocols should be selected that do not alter the characteristics of the media (e.g., particle- size composition).
![Page 51: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/51.jpg)
Components ofWithin-Sampling Unit Precision
Sample Collection and Measurement Process
Within Sample Variability
Measurement Method
Imprecision
Inherent small-scale variability
Subsampling or homogenization
Sample handling and preparation
Analytical instrument
Within- Sampling-Unit Variability
Small-Scale Variability(within unit)
![Page 52: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/52.jpg)
QA Samples Used to Evaluate Components of Total-Study Variability
![Page 53: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/53.jpg)
Total Within-Sampling-UnitPrecision Pyramid
![Page 54: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/54.jpg)
QC SampleComponents of
Variability Captured
Estimated Standard Deviation
Instrument replicate Instrument response (IR) 0.046
Laboratory replicate IR + subsampling and extraction/digestion (S&E)
0.11
Laboratory split IR + S&E + lab homogenization (LH) 0.12Field split IR + S&E + LH + sample handling (SH) 0.12Collocated samples IR + S&E + LH + SH + field sample acquisition
and small-scale variability (A&SV)0.15
Field samples IR + S&E + LH + SH + A&SV + between- sampling-unit variability
1.11
Example of Total-Study Variability Components
* Data is from actual lab/field study in New Mexico lab in the USA.
![Page 55: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/55.jpg)
A Simple Additive Variance Model
2t = 2
b + 2w
2t = 2
b + 2m + 2
st = total studyw = within-sampling-unitb = between-sampling-unitm = measurements = small-scale variability (eg. field duplicate) = Variance2
![Page 56: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/56.jpg)
Visualizing the Contribution of Components of Total-Study Variance
Total-Study(Field Samples) Within-Sampling-
Unit
Between-Sampling-Unit
t(Collocates)
b
w
![Page 57: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/57.jpg)
Total-Study(Field Samples)
Within-Sampling-
Unit Variability
Between-Sampling-Unit
w = .15
(Collocates) 0.10Small-Scale Variability
0.11
0.046 Sample-Preparation Variability 0.10
Analytical-InstrumentVariability
(Instrument Replicates)
Measurement Variability(Lab Replicates)
Visualizing the Within-Sampling-Unit Components of Variability
t
b
![Page 58: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/58.jpg)
Calculating Variance
Total variance:
Within-sampling-unit variance:
for duplicates: for multiple replicates:
Between-sampling-unit variance:
![Page 59: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/59.jpg)
Calculating Components of Variance from an Existing Data Set
Sample IDArsenic(ppm)
Cadmium(ppm)
Lead(ppm) Sample Type
99-7510 3.9 <0.07 22
99-7510 2.2 <0.06 20 lab replicate
99-7511 3.1 <0.07 104
99-7512 2.6 5.3 37.6
99-7512a 2 5.9 34.8 field split
99-7513 2.4 <0.07 782
99-7513 4.6 <0.07 829 lab replicate99-7514 2.5 0.47 35.999-7514a 3.1 <0.07 37.7 field split
99-7515 4.4 1.4 17.5
99-7515a 2.9 1.6 28.3 field split
99-7516 3.2 4.5 55.2
99-7517 2.8 5.1 921
99-7517 3.5 5.6 902 lab replicate
Standard deviation 0.78 2.5 390
![Page 60: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/60.jpg)
Calculating Components of Variance from an Existing Data Set (cont.)
Within-sampling-unit variance for lead may be calculated using the field split data
Based on all samples, the total-study variance for lead is estimated by:
Between-sampling-unit variance for lead is calculated as:
![Page 61: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/61.jpg)
Using Indicators of Variance
Arsenic Cadmium Leads2
w 0.25 0.13 21.3s2
b 0.36 6.01 148,876s2
t 0.61 6.14 148,897
For cadmium and lead, the total variability is swamped by between-unit-sampling variability (i.e., site heterogeneity).Arsenic probably near background - most variability comes from measurement process.
![Page 62: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/62.jpg)
Establishing MQOs
Decomposing total-study variance facilitates the identification of the relative importance of components of total error.this exercise also helps determine what kind of QA samples to employ.
Total-study variance estimates are plugged directly into sample-size calculations.Individual “Measurement Quality Objectives” (MQOs) should be established for components of variance that primarily drive the total variability.
MQOs on specific measurement components must reflect the requirements for total-study error.
![Page 63: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/63.jpg)
Strategies for Reducing Within-Sampling-Unit Variance
ReplicationSmall-scale compositingIncreasing sample supportMore precise measurement method
![Page 64: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/64.jpg)
BiasAnalysis and Prevention
![Page 65: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/65.jpg)
Bias
Bias = measured result - true value
Relative bias = measured result - true value
true value
When dealing with recovery rates:
Recovery = 1 + measured result - true value
true value
and expressed as a percentage
![Page 66: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/66.jpg)
Principal Causes of Bias
Incomplete dataAnalyticalcalibration errorsample contaminationmatrix effectsinterferences
Samplingincorrect location identificationjudgmental sampling scheme
![Page 67: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/67.jpg)
Bias Due to Incompleteness
Example: the objective is to estimate the percentage of correctly documented permits for exemptionData obtained by requesting the holders of permits to respond, 60% responded to the request.
Of these responses, 70% were correctly documented, does the 40% non-response rate really matter?
True percentage = (respondents x their percentage) + (non-respondents x their percentage)
Bias = non-respondents x (difference in percentages)
![Page 68: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/68.jpg)
If non-responses were 70% correctly documented
Bias = 0%, correct estimate is 70%If non-responses were 50% correctly documented
Bias = 8%, correct estimate is 62%If non-responses were 30% correctly documented
Bias = 16%, correct estimate is 54%If non-responses were 10% correctly documented
Bias = 24%, correct estimate is 46%
Bias Due to Incomplete Response
![Page 69: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/69.jpg)
Calibration Errors Leading to Bias
![Page 70: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/70.jpg)
Matrix Effects Leading to Bias
The composition of the matrix can influence both preparation and analysis.Non-ideal chemical behavior influences samples differently than standards.
![Page 71: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/71.jpg)
Method of Standard Addition (MSA)
Concentration in sample
![Page 72: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/72.jpg)
Atomic Absorption: spectral: cannot resolve analyte from other species chemical: chemical processes alter absorption characteristics of analyte
Possible resolution: successive serial dilutions matrix modification MSA
Interferences Leading to Bias
![Page 73: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/73.jpg)
Sample Handling Errors Leading to Bias
Loss of sample during collection, storageinadequate preservation (acid, darkness, cool, excessive holding time).
Examples:metals require acidification to prevent or minimize precipitation and adsorption to sample container.
![Page 74: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/74.jpg)
Why Bias, Why Not Accuracy?
Accuracy includes both precision (random error that could be positive or negative for each individual reading) and bias (systematic error that is either positive or negative for all readings)Accuracy (mean square error) = variance + bias2
Precision is estimated through replicate measurementsBias is estimated by comparison of the mean of replicate measurements to a known standardWithout standards bias cannot be estimated with confidence, only a reduction in bias is possible
![Page 75: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/75.jpg)
Bias Hidden as Variability
x
xx
x
x
x
xxx
xx
x x
x
x
x
xx
x
x
x
x
xx
x
x
0
50
40
30
20
Is data set A or B a better representation of the population?
x
xx
x
x
x
xxx
xx
x x
x
x
x
xx
x
x
x
x
xx
x
x
A B
10
![Page 76: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/76.jpg)
Both data sets have similar variability. Data set B is a biased representation of the population of interest.
=38.5
Bias Hidden as Variability (cont.)
x
xx
x
x
x
xxx
xx
x x
x
x
x
xx
x
x
x
x
xx
x
x
0
50
40
30
20
x
xx
x
x
x
xxx
xx
x x
x
x
x
xx
x
x
x
x
xx
x
x
A B
10
![Page 77: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/77.jpg)
SensitivityDiscerning the Signal in the Noise
Concentration
Response
![Page 78: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/78.jpg)
SensitivitySensitivity is the capability of a method or instrument to discriminate between measurement responses representing different levels of the variable of interest.the term "detection limit" is often used without consideration of what is really meant.
there are several sensitivity DQIs including IDLs, MDLs, and PQLs.
A sensitivity DQI describes the capability of measuring a constituent at low levels.a PQL describes the ability to quantify a constituent with known certainty.
• e.g., a PQL of 0.05 mg/L for mercury represents the level where a precision of +/- 15% can be obtained.
![Page 79: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/79.jpg)
Calibration Standards
Samples containing the analytes of interest are generally prepared in a clean matrix, to develop a relationship between concentration and instrument response. This can cause a problem if the matrix under investigation interacts differently with the analyte compared to a clean matrix.The relationship between concentration and instrument response is normally used to predict the unknown concentration found in samples of interest.Calibration allows for the determination of theoretical detection and quantification limits.
![Page 80: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/80.jpg)
Calibration Standards - graphically
1 g/L 2 g/L 3 g/L 3.5 g/L 4 g/L 4.5 g/L
This graph shows:Instrument response for six calibration levelsEach response represents an average of multiple runsWhere are the quantification levels?
![Page 81: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/81.jpg)
Calibration Curve from Standards
By graphing the calibration standards we can see three regions of interest in the relationship: below the linear rangethe linear range above the linear range
Concentration (g/L)
![Page 82: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/82.jpg)
Relationship of Instrument Calibration Curve and Analyte Detection/Quantification
Instrument Response
Concentration
Region of unknown identification and quantitation
Region of less certain quantification
Region of less certain quantification
Region of less certain identification
Region of knownquantification
ConcentrationIDL MDL PQL LOL
12
3
4
5 1
2
3
4
5
IDL = instrument detection limit MDL = method detection limit PQL = practical quantitation limit LOL = limit of linearity
![Page 83: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/83.jpg)
Commonly Used Sensitivity Indicators
Sensitivity Indicator Numerical Definition Definition Common UseInstrument Detection Limit (IDL)
Usually 3 times the instrument noise level
Lowest value at which instrument can distinguish from zero
Provides basis for determining an MDL
Method Detection Limit (MDL)
MDL = t (n-1, 0.99) x s
s = standard deviation
for 7 aliquots:t (n-1, 0.99) = 3.14
Defined 40 CFR Part 136 Appendix B
Determines the theoretical detection limit
Practical Quantitation Limit (PQL)
PQL = 5 x MDLorPQL = 10 x MDL
(more precisely defined as the lowest standard on the instrument calibration curve)
"the lowest concentration of an analyte that can be reliably measured within specified limits of precision and accuracy during routine laboratory operating conditions"
Provides numerical lower limit for critical data
Reporting Limit (RL) Laboratory defined(often the RL = PQL)
Lowest value reported by laboratory without a "J" flag
Laboratory basis for data reporting
![Page 84: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/84.jpg)
MDL: Controls the Type I Error
USEPA Definition of MDL:
“Concentration where we have 99% confidence that value of analyte concentration is greater than zero.“
By accepting 1% chance of type I error (false positive), we define the MDL as equal to:
MDL = t * Sd.
Type I error (false positive): concludes that the analyte is present when in fact it is absent (zero).
![Page 85: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/85.jpg)
MDL: Does Not Control the Type II Error
If the MDL is chosen as the reporting limit, then by default there is a 50% probability of a type II (false negative) error.
This means a sample that is truly at the MDL will be considered below the MDL 50% of the time.
Type II error (false negative): concluding that the analyte is absent when in fact it is present.
![Page 86: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/86.jpg)
PQL Relationship to MDL
PQL: Multiple definitions, generally 5-10 times MDL, or lowest point on calibration.
Requiring 5-10 times MDL usually provides precision of less than 20% RSD (quantification).
If the PQL is 5 times the MDL, and the MDL is about 3 times standard deviation, then the PQL is approximately 15 times the standard deviation.
![Page 87: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/87.jpg)
Quality Associated with Calibration Regions
3
Zero AnalyteConcentration
Region of high uncertainty
Region of certain detection
Region of less certain quantification
ApproximateMDL Level
Region of certainquantification
Instrument signal, standard deviation units
LODMatrix/method blank
Approximate PQL Level
LOQ
LOD = limit of detectionLOQ = limit of quantitationpopulation standard deviation
![Page 88: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/88.jpg)
Project Management Perspective
There are major differences between the various sensitivity DQIs (RL, IDL, MDL, and PQL) even when you specify a particular indicator, it is very important to get a precise definition, description of the process and formula to know what it really means!
it is important to know if the indicator reflects the detection limit in a clean matrix, or in an actual sample
The PQL is usually the most useful indicatorwhen selecting an analytical method or laboratory
![Page 89: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/89.jpg)
It is important to specify what results should be reported and how.all results above the MDL should be reported.
values that fall between the MDL and PQL should be "flagged" to indicate uncertainty in the value; and
values below the MDL should be reported as Non Detects and the MDL included.
Project Management Perspective (cont.)
![Page 90: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/90.jpg)
General Advice to Lab/Project Managers
Have a very specific understanding of the indicator(s) you choose to use;
Do not make any assumptions regarding the adequacy of a method, or of a data set, based on detection limits that have not been carefully defined.
The MDL and PQL should be reported, as well as all concentrations above the MDL.
Avoid censoring data at the PQL or RL. Get all values down to the MDL, but have the values
between the MDL and PQL flagged (they are estimates and there is no confidence in their exact concentration).
![Page 91: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/91.jpg)
What Drives The "Detection Limits"?
Regulatory requirementsprimary drinking water quality requirementsrisk based goals (nutrients, toxic elements)Irrigation water quality requirements
Background values for comparisonproject-wide site-specific
![Page 92: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/92.jpg)
Examples of Other Sensitivity Indicators
Detection limit (DL)commonly seen in regulations, no rigorous definition
Limit of detection (LOD)similar to MDL, however statistical formula modified (set at 3 times Sd.)
Reliable detection limit (RDL)level where detection is extremely likely (set at 6 times Sd)
Limit of quantification (LOQ)set at 10 times Sd.
Contract required detection limits (CRDL)Contract required quantification limits (CRQL)
![Page 93: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/93.jpg)
Controversies
Definition of MDL, provided by USEPA, is statistically incorrect/confusing. However, it remains the most widely documented DQI for sensitivity and one of the simplest ways to calculate a detection limitThere are too many definitions and indicators:
DL, MDL, CRDL, PQL, LOQ, ML, …
Censoring of low level values results in lost information because by not reporting data below a sensitivity indicator (e.g., PQL), information that could potentially be utilized in decision making is lost.
The correct approach is dependent upon how the data will be interpreted and used.
![Page 94: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/94.jpg)
Techniques for Lowering the Method Detection Limit
Use more sample materialIf possible, improve detector sensitivityReduce interferences
During sample preparation:instrumentation:
selective AA: graphite furnace versus flame matrix modifiers in AA to alter volatility
![Page 95: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/95.jpg)
Case Study 1: Calculation of MDL
Preliminary sampling has resulted in request for lower detection limit for obtained data.What are options?
more sample material;reduced interferences; ormore sensitive detector.
Elect to choose more sensitive detector .Use of graphite furnace AAS instead of AAS (more sensitive).
![Page 96: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/96.jpg)
Initial Calibration of More Sensitive Detector (graphite furnace equipped AAS)
Concentration (g/L)
![Page 97: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/97.jpg)
Concentrationg/L Response
Standard Deviation Average
Average/Std Dev.
.02 383
.02 178
.02 400 124 320 2.6
.03 451
.03 500
.03 754 163 568 3.5
.05 1448
.05 1178
.05 1220
.05 1089 153 1234 8.1
Evaluate the calibration data The calculated MDL must be less than the spike levelThe spike level should not be greater than 10 times calculated MDL (prefer spike 1-5 times MDL) Optional: pick a spike at level where signal/noise (average/Sd) is 2.5-5.0
Select the Spiking Level
![Page 98: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/98.jpg)
Based on initial calibration curve and review of low level standards chose to spike at .05 mg/L
AliquotInstrument Response
Concg/L
1 1331 0.049972 1052 0.041933 1066 0.042344 1245 0.047515 1069 0.042446 1138 0.044437 1267 0.048128 1325 0.04981
Conc.g/L
Average 0.04582
Std Dev 0.00342MDL 0.010PQL .05
Calculating the MDL
MDL = t(n-1, 0.99) x Sd
–for 8 aliquots: t(n-1, 0.99) = 2.998MDL = 2.998 x .0034 = 0.010
![Page 99: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/99.jpg)
Some Common Mistakes
Miscalculation of MDLuse population (n) standard deviation instead of sample (n - 1) in the statistical calculations
pick wrong t statistic (df = number of samples - 1)use less than 7 aliquots
Spike too high or too lowMDL < spike < 10 times MDL
(preferably spike at 1-5 times MDL)
Select method based on reported MDLs, without considering whether this detection limits is achievable in the true matrix
![Page 100: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/100.jpg)
In a study by Wisconsin DNR (1993), it was found that 23 of 56 labs incorrectly calculated MDL.A 1998 survey found 26% of submitted lab results in Wisconsin were incorrect.only 17% of the laboratories (total of 122) reported data that met the criteria for all analyte.
for this study 2313 MDLs were reported.
Mistakes are Common everywhere
![Page 101: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/101.jpg)
Question the MDL Study Results
What are the problems with the following MDL results?
Mercury MDL study
Zinc MDL study
spike 0.2 .32 .31 .34 .33 .35 .34 .35 s = .0151MDL =.047
spike .05
.0511 .0516 .0511 .0507 .0512 .0505 .0520s =
.00051MDL
= .0016
spike/MDL =31
![Page 102: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/102.jpg)
Question the MDL Study Results
Mercury MDL studyrecovery of the spiked samples is approximately 160% indicating quantification bias in this range
Zinc MDL studyspiked level is 31 times MDL, should be 10 or less
spike 0.2 .32 .31 .34 .33 .35 .34 .35 s = .0151MDL =.047
spike .05
.0511 .0516 .0511 .0507 .0512 .0505 .0520s =
.00051MDL
= .0016
spike/MDL =31
![Page 103: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/103.jpg)
Conclusions
Detection limit language is looseregulations specify required DLs
do they mean IDLs, MDLs, PQLs?
USEPA has defined MDLs that is widely used, but statisticians debate precise formulation.
Procedural implementation is not standardizedlabs routinely take shortcuts (e.g., do not rerun at appropriate concentrations);
miscalculations are common; andmatrix effects seldom considered or reported.
Know what you want and how to communicate it with all parties in the process (lab-client).
![Page 104: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/104.jpg)
Comparability Using More Than One Data Source
![Page 105: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/105.jpg)
Comparability
Comparability is a qualitative expression of the measure of confidence that two or more data sets may contribute to a common analysis.a comparability DQI is a qualitative indicator of the similarity of attributes of data sets.
![Page 106: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/106.jpg)
Common Indicators of Comparability
Field attributesmatrix compositionsample collection methodtime/season of sampling
Qualitative Analytical attributessample preparationanalytical methoddetection limit determination/reporting
Quantitative Analytic attributesspread or variability of datacommonalties in central tendency
![Page 107: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/107.jpg)
Comparability of New and Existing Data
Combining existing dataThe comparability of data sets generated at different times or by different organizations must be evaluated.
The evaluation should establish whether two data sets can be considered equivalent with respect to the measurement of a specific variable or groups of variables.
Gathering new data New data must be collected so that it is comparable to existing data for key characteristics.
![Page 108: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/108.jpg)
Existing Data Example
A Regional government agency and a local Woreda town have both collected data. Are the two data sets comparable?both data were collected during the summer of 2009;some of the data collected was from very similar locations;
several common chemicals were analyzed by both; and
equivalent analytical methods were used.
![Page 109: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/109.jpg)
Existing Data Example (cont.)
![Page 110: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/110.jpg)
Existing Data Example (cont.)
![Page 111: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/111.jpg)
Compare: Sample Design
Are all data equally representative of the population of interest?random implies equal probability of selection;biased implies unequal probability of selection; andcombining data from multiple design types may limit the usability of the data.
![Page 112: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/112.jpg)
Compare: Temporal/Spatial Consistency
Data representative of the same populationdata from similar time frames and locations;data from different times or locations are fine if equivalency across space/time is reasonable to assume;
data from different temporal or spatial zone may be used for trend analyses, if are comparable in other ways.
![Page 113: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/113.jpg)
Compare: Sample Collection Methods
Field methodsType of sampling instrument;sample collection procedure; andfield splits, duplicates, or composites.
Sample handlingFiltering;preservatives or other special requirements; andtime of sample preparation and time of analyses.
![Page 114: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/114.jpg)
Compare: Variables of Interest
Variables for grouping data–location, date, media, etc.
Variables for determining comparability–particle size, total organic compounds, percent moisture, etc.
Variables for analysis–reported concentration, depth to groundwater, etc.
![Page 115: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/115.jpg)
Compare: Units of Measurement
All data sets should have units that are convertible to a common metric or SI unitmg/kg, , ng/g, and mg/g - AcceptablepCi/g and mg/L - Not Acceptable (unless additional information is available for the conversion)
![Page 116: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/116.jpg)
Comparability Summary TableField Attributes
Data Attribute
Data Set #1City: Mota OARD
Data Set #2BoARD Remarks
Sample collection method
Composited topsoil sample
Grab sample
Matrix Soil SoilSample handling unknown Followed approved SOPsSampling event Oct 2009 Sept 2008
![Page 117: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/117.jpg)
Sample preparation methods should be consistentsample handling timesSample preparation (e.g., saturated paste vs. 1:2.5 H2O)
Laboratories sometimes differwas the same laboratory used for all analyses?are there noticeable differences between laboratory operations and techniques?
Compare: Sample Preparation
![Page 118: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/118.jpg)
Compare: Analytical Methods
Analytical methods should be consistentAre the methods documented and how do they perform for the intended elements under investigation?
Are the same methods used for all analyses?Are the same method options used for all analyses and how do they compare to each other?
Are the methods capable of selecting the right analyte or fraction?
Can the required MDL/PQL be achieved consistently?
![Page 119: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/119.jpg)
Compare: Detection Limits
Are the same type of detection limits reported for all data? (e.g., MDL, PQL, etc.)How do the detection limits compare between data sets?How do the detection limits compare to detected values in the data sets?
Are the detection limits acceptable for use in decision making?
![Page 120: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/120.jpg)
Compare: Quality Control
Quality control of data entryresults reported into database in similar manner
Qualification and/or validation of datasimilar QA/QC information available from all laboratories
criteria consistent for data qualification or validation
![Page 121: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/121.jpg)
Comparability Summary TableAnalytical Attributes
Data Attribute
Data Set #1City: White Springs
Data Set #2Federal Agency Remarks
Sample preparation
unknown 1:2.5 H2O
Analytical method unknown Method 34-b
Analytical method option
unknown UV Spectrophotometer
Detection level 680 - 1240 1 -3.4Units g/kg (ppb) mg/kg (ppm) Can be converted to
matchFields of interest that were reported
Missing QC data and sample collection method
All desired fields Locations in different coordinates: conversion possible
Criteria for exclusion of samples
None Rosner's test for outliers No data excluded based on these rules
![Page 122: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/122.jpg)
Statistical Comparability
VarianceMean or medianDistribution
![Page 123: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/123.jpg)
Example of Temporal Differences
3 related sites, samples collected from 1993 - 1998Changes in site conditions not expected over time
![Page 124: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/124.jpg)
Thresholds of interest are 30 mg/kg and 500 mg/kg and there are data sets outside this region. This may affect comparisons of a statistical natureCombining data from across years for joint analysis is probably not appropriate (compare 1996 data with 1998 data, the latter being much greater in magnitude)Cause of differences (bias) should be investigated
Conclusion: Temporal Differences
![Page 125: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/125.jpg)
CompletenessAnalysis and Prevention
![Page 126: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/126.jpg)
Completeness
Completeness % = 100 x Number of valid measurements
Total # of measurements
e.g., 12 field samples were collected, but at the laboratory two were found to be contaminated by a foreign substance and were rejected
Completeness % = 100 x 10 (valid samples) = 83%
12 (total samples)
![Page 127: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/127.jpg)
Causes of Incompleteness
Loss through contaminationInvalidation due to violation of QA protocolsRestriction due to time constraints in obtaining samplesLoss through QC calibration mistakesPhysical loss through sample destructionErrors in field collection techniquesInsufficient physical sample material for analysis
![Page 128: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/128.jpg)
Qualitative Incompleteness
If the missing data are randomly disbursed throughout the study and the variability is less than expected, the DQOs established at the outset of the project may still be achieved.If the missing data are not randomly disbursed but affect either the target area for inference, or affect predominantly one element of interest, then even a high degree of completeness may be inadequate for decision making purposes.
![Page 129: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/129.jpg)
The Consequences of Incompleteness
Simple sampling schemeRandom missing data
loss of statistical power and a general weakening of confidence in conclusions.
Clustered or systematic missing dataloss of statistical power and weaker confidence in conclusions.
loss in assumption of representativeness.
![Page 130: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/130.jpg)
Consequences of Incompleteness (cont.)
Complex or statistical sampling schemeRandom missing data
loss of statistical power and a weakening of confidence in conclusions.
possible severe compromising of results depending on where the missing data occurred.
Clustered or systematic missing datasevere compromising of results.probable abandonment of the study.
![Page 131: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/131.jpg)
Corrective Actions for Incompleteness
Minor incompleteness in simple random sample additional sample collection to achieve DQOsanalysis of effect on false decision errors for DQOs
Minor incompleteness in a complex sampleadditional sample collection to achieve DQOsstatistical imputation (repair schemes) necessary
Minor incompleteness in sample analysesadditional re-analyses from existing samples
Major incompleteness in sample analysesadditional sample collection to achieve DQOsstatistical correlation analysis of surrogates
![Page 132: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/132.jpg)
Corrective Actions for Incompleteness (cont.)
Incompleteness due to failure of QC requirementsstatistical readjustment of the dataexpert opinion from the QA manager
Incompleteness due to violated QA protocolsexpert opinion from the QA manager
Incompleteness due to lack of representativenessexpert opinion on potential impact on DQO decisions
![Page 133: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/133.jpg)
Data Verification, Data Validation, and Data
Integrity
![Page 134: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/134.jpg)
Definition of Verification & ValidationDATA VERIFICATION: Evaluating the completeness, correctness, and conformance or compliance of a specific data set against the method, procedural, or contract requirements (lab staff activity);DATA VALIDATION: Confirming that the particular requirements for a specific intended use are fulfilled by examination and provision of objective evidence (3rd party activity); Sample-specific process extending data evaluation beyond method, procedural, or contractual compliance to decide analytical quality of a specified data set.
![Page 135: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/135.jpg)
DATA INTEGRITY: Assurance that individuals and organizations involved in the collection and analysis of laboratory measurements, have performed their activities in accordance with appropriate ethical standards, and that the resulting data sets are free from deliberate falsification.
Definition of Integrity
![Page 136: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/136.jpg)
Goals of Data Verification
To document steps that were taken to collect or analyze laboratory measurements.
To confirm compliance with applicable technical requirements.
![Page 137: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/137.jpg)
Who is Responsible for Verification?
Verification is part of an organization's management processMultiple parties are involved in verification:
those involved with data should document their actions;peers should observe and confirm performance of tasks;supervisors/managers should review processes and outputs; and
verification can also be external (performed by an independent party) or internal (performed by a specially assembled group or team).
![Page 138: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/138.jpg)
The Use of a Graded Approach
There is no "one size fits all" formula for verificationVerification steps may vary according to project needs;Variables can include:amount and type of documentation;extent of observation by peers;number of levels of management review; androle of third parties.
![Page 139: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/139.jpg)
Typical Verification Activities
Confirm that the practices required by method, procedural, or contract requirements were actually employed;Perform independent double check of calculations;Confirm that field or lab notes are consistent with formal reports;Assess analytical and field uncertainty; andCheck the operation of Laboratory Information Management System (LIMS)
![Page 140: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/140.jpg)
DQIs to Consider in Verification
Completeness: Did the desired number of samples/data points were obtained, along with corresponding meta-data?
Comparability: Can it be confirmed that all samples were analyzed using comparable methods?
Representativeness: Has the planned sampling design been followed appropriately?
![Page 141: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/141.jpg)
Verification Outputs
Documented field report;Verified laboratory data package; andVerified and documented data.
![Page 142: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/142.jpg)
What Verification Does/Doesn't Achieve
Does:document performance; andconfirm compliance with SOPs, methods, contract provisions, or similar requirements.
Doesn't:resolve data quality issues; ordetermine data usability.
![Page 143: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/143.jpg)
Goals of Data Validation
An independent scrutiny of processes for collecting and analyzing laboratory related activities and measurements; andAn evaluation and qualification of resulting data with respect to its intended use.
![Page 144: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/144.jpg)
Who should do Data Validation?
An independent party (individual or organization) reporting to the data user, orIf both field and laboratory validation are pursued, they are likely to be conducted by different parties.
![Page 145: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/145.jpg)
Graded Approach for Validation
There is no "one size fits all" method of validationThe depth of validation depends on the goals of the project and eventual use of the data.Variables can include:percentage of data validated;double checking of calculations;formal "flags" versus narrative evaluations of data; andextent of scrutiny of pre-analytical steps (e.g., sample preparation)
![Page 146: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/146.jpg)
Typical Validation Activities
Review records for consistency and completeness;Evaluate analytical method performance;Inspect data packages and draw conclusions about data quality;Assign or review data qualification codes; andReview data against project requirements.
![Page 147: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/147.jpg)
DQIs to Consider in Validation
Sensitivity: Were the desired detection limits achieved?
Bias: Do recoveries meet performance goals?
Precision: Were performance goals established in the QA Project Plan met?
![Page 148: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/148.jpg)
Validation Outputs
Validation reports for data user or data usability assessor; andDocumented qualified (flagged) data.
![Page 149: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/149.jpg)
What Validation Does/Doesn't Achieve
Does:provide independent confirmation that requirements for a specific intended use were fulfilled; and
evaluate the quality (versus MQOs) of data points/data sets.
Doesn't:resolve data usability issues.
![Page 150: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/150.jpg)
Integrity: a Vital Issue
Every component of the data collection process is subject to manipulation;Lapses in integrity can lead to dire consequences; andPast history has demonstrated that individual and organizational integrity cannot be simply “assumed”.
![Page 151: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/151.jpg)
Consequences of Lack in Integrity
Delays in completion of projects;Cost overruns/rework;Loss of public credibility by affected agencies/institutions;Continuing risks to the environment or natural resources; andPotential civil or criminal penalties for offenders.
![Page 152: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/152.jpg)
Improper Field Activities
Failure to collect required samples or specimens from the proper locations;The tampering of collected samples; andMisrepresentation of field procedures and documentation.
![Page 153: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/153.jpg)
Improper Laboratory Activities
Failure to analyze samples;Manipulation of sample prior to analysis;Failure to conduct required analytical steps;Manipulation of results during or after analysis; andPost-analysis alteration of records.
![Page 154: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/154.jpg)
Falsification is Difficult to Detect
Its purpose is to "disguise" records such as data packages - to hide itself from scrutiny;Some practices are legitimate in certain cases but improper in others;Falsification can be conducted by isolated individuals and on an organization-wide basis; andOnce a type of falsification can be detected, other types can take its place.
![Page 155: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/155.jpg)
Detection Tools
Data validation;Site audits of management and technical systems;Data audits;Review of data tapes/disks;Performance evaluation (PE) and split sampling programs;Review of LIMS audit trail when available; BUT, no single tool can guarantee detection.
![Page 156: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/156.jpg)
Warning Signs of Breakdown in Integrity
Inconsistencies in dates and times;Evidence of "special handling" of QC or PE samples;Indications of instrument manipulation; andEvidence of retroactive editing of data.
![Page 157: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/157.jpg)
Is Lack of Integrity Universal?
No, but serious violations do occur;Be conscious of the possibility;Lab industry is aware of the problem and putting effort into ethics training; andMost soil, plant and water analysis data from laboratories are the result of proper analyses with appropriate QA protocols, only a few are not.
![Page 158: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/158.jpg)
Summary
![Page 159: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/159.jpg)
PARCCS & VVIQuantitative DQIs:
Precision Bias (Accuracy)Sensitivity
Qualitative DQIs:RepresentativenessComparabilityCompleteness
Associated DQIs:VerificationValidationIntegrity
![Page 160: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/160.jpg)
Precision
Precision is the measure of agreement among repeated measurements of the same property under identical or substantially similar conditions.A precision DQI is a quantitative indicator of the random errors or fluctuations in the measurement process.
![Page 161: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/161.jpg)
Bias
Bias is systematic or persistent distortion of a measurement process that causes error in one direction.A bias DQI is a quantitative indicator of the magnitude of systematic error.
![Page 162: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/162.jpg)
Sensitivity
Sensitivity is the capability of a method or instrument to discriminate between measurement responses representing different levels of the variable of interest.A sensitivity DQI describes the capability of measuring a constituent at low levels.
![Page 163: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/163.jpg)
Comparability
Comparability is a qualitative expression of the measure of confidence that two or more data sets may contribute to a common analysis.A comparability DQI is a qualitative indicator of the similarity of attributes of data sets.
![Page 164: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/164.jpg)
Completeness
Completeness is a measure of the amount of valid data obtained from a measurement system, expressed as a percentage of the number of valid measurements that should have been collected.The DQI for completeness is often expressed as a percentage.
![Page 165: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/165.jpg)
Representativeness
Representativeness is the measure of the degree to which data suitably represent a characteristic of a population, parameter variations at a sampling point, a process condition, or an environmental condition.Representativeness DQIs are qualitative and quantitative statements regarding the degree to which data reflect the true characteristics of a well defined population.
![Page 166: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/166.jpg)
Verification
Data verification refers to the procedures needed to ensure that a set of data is a faithful reflection of all the processes and procedures used to generate the data.
![Page 167: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/167.jpg)
Validation
Data validation is an analyte and sample matrix-specific process to determine the analytical quality of a specific data set.
![Page 168: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/168.jpg)
Integrity
Lack of integrity affects all aspects of data interpretation, especially data used for decision making.
![Page 169: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/169.jpg)
What are the Most Important DQIs?
Representativeness: If the samples are not representative of the environmental conditions, how much reliability can be given to the results of the analyses? The samples must be representative of the field, and the analytical work performed must be representative of the sample sent for analysis.
Integrity: If the samples have been tampered with, how much reliability can be given to the results of the analyses? The samples must have come from where they were planned, and the analytical work performed properly.
![Page 170: Data Quality Indicators (DQIs) What are they, and how do they affect me? An US-EPA Approach](https://reader036.fdocuments.us/reader036/viewer/2022062409/568145f9550346895db303ba/html5/thumbnails/170.jpg)
Questions?