Quality System Training Module 4 Data Validation and Usability · Quality System Training Module 4...
13
1 Ambient Air Quality System Training QA Strategy Workgroup QA Strategy Workgroup Ambient Air Quality System Training Quality System Training Module 4 Data Validation and Usability Dennis Mikel EPA - Office of Air Quality Planning and Standards 2008 Conference on Managing Environmental Quality Systems 2 Ambient Air Quality System Training QA Strategy Workgroup QA Strategy Workgroup Ambient Air Quality System Training Some Definitions • Verification —is the process for evaluating the completeness, correctness, and conformance/compliance of a specific data set against the method, procedural, or contractual specifications. It essentially evaluates performance against pre-determined specifications, for example, in an analytical method, or a software or hardware operations system. Validation — is an analyte- and sample-specific process that extends the evaluation of data beyond method, procedure, or contractual compliance to determine the quality of a specific data set relative to the end use. It focuses on the project’s specifications or needs, designed to meet the needs of the decision makers/data users and should note potentially unacceptable departures from the QA Project Plan. The potential effects of the deviation will be evaluated during the data quality assessment.
Transcript of Quality System Training Module 4 Data Validation and Usability · Quality System Training Module 4...
1
Ambient Air Quality System Training QA Strategy WorkgroupQA Strategy Workgroup
Ambient Air Quality System Training
Quality System TrainingModule 4
Data Validation and Usability
Dennis Mikel EPA - Office of Air Quality Planning and Standards
2008 Conference on Managing Environmental Quality Systems
2
Ambient Air Quality System Training QA Strategy WorkgroupQA Strategy Workgroup
Ambient Air Quality System Training
Some Definitions• Verification —is the process for evaluating the
completeness, correctness, and conformance/compliance of a specific data set against the method, procedural, or contractual specifications. It essentially evaluates performance against pre-determined specifications, for example, in an analytical method, or a software or hardware operations system.
Validation — is an analyte- and sample-specific process that extends the evaluation of data beyond method, procedure, or contractual compliance to determine the quality of a specific data set relative to the end use. It focuses on the project’s specifications or needs, designed to meet the needs of the decision makers/data users and should note potentially unacceptable departures from the QA Project Plan. The potential effects of the deviation will be evaluated during the data quality assessment.
2
3
Ambient Air Quality System Training QA Strategy WorkgroupQA Strategy Workgroup
Ambient Air Quality System Training
Process Design• Validation and Verification are part of a
process– It should be performed at many different
points– It should be performed by qualified staff– It is the responsibility of QA and QC– There are 4 levels of validated data
– Verification takes place early in process– Validation takes place later in the process
4
Ambient Air Quality System Training QA Strategy WorkgroupQA Strategy Workgroup
Ambient Air Quality System Training
The Process
3
5
Ambient Air Quality System Training QA Strategy WorkgroupQA Strategy Workgroup
Ambient Air Quality System Training
Levels of Validation• Level 0
–Essentially raw data obtained from the DAS, • Data have been reduced, but are unedited and un-
reviewed, nor adjusted.
• Level 1–Checks are performed by DAS software
• screening programs• Qualitative checks are performed by field staff• Quality control flags are assigned
6
Ambient Air Quality System Training QA Strategy WorkgroupQA Strategy Workgroup
Ambient Air Quality System Training
Levels of Validation
• Level 2 – Comparisons with independent data sets, such as
meteorological and ambient pollution data – Utilize database screening tools, SAS or SQL
• Level 3– Detailed analysis and final screening– Validate so there are no inconsistencies among the related
data – Graphics programs used examine the overall consistency
If you have met data use it to validate your pollution data!!!
4
7
Ambient Air Quality System Training QA Strategy WorkgroupQA Strategy Workgroup
Ambient Air Quality System Training
Step 1: Visual Inspection• Inspections are used to make
sure the instruments are fully operational
• Sometimes the “smell, sound or feel” of an instrument isn’t right – (i.e., is something fishy!!)
• Instrument Diagnostics– Some DAS record this information– Routine Operation and
Maintenance forms must be checked
8
Ambient Air Quality System Training QA Strategy WorkgroupQA Strategy Workgroup
Ambient Air Quality System Training
Step 1: Visual Inspection
5
9
Ambient Air Quality System Training QA Strategy WorkgroupQA Strategy Workgroup
Ambient Air Quality System Training
Step 1: Visual Inspection
10
Ambient Air Quality System Training QA Strategy WorkgroupQA Strategy Workgroup
Ambient Air Quality System Training
Step 2: DAS Data Verification• Data Acquisition Systems have built in
verification techniques that can:– Sense a change in an operation– Can receive a signal from instrument that there is a
problem – Can flag data that is out of “range” or exceeds a
limit
6
11
Ambient Air Quality System Training QA Strategy WorkgroupQA Strategy Workgroup
Ambient Air Quality System Training
Step 2: DAS Data Verification
• A few techniques– Did the value exceed the preset maximum limit?– Did the value exceed the preset minimum limit?– Did the value exceed a defined rate of change?
• Fall time or rise time– Does the data need to have a percentage of valid
readings to be considered valid?– DAS will flag any data if exceeds your limits!– This is performed as data are collected and is
automated, how cool is that!
12
Ambient Air Quality System Training QA Strategy WorkgroupQA Strategy Workgroup
Ambient Air Quality System Training
Step 3: Manual Review
• Manual Review pertains to a review of the data either before or after it goes to the central data base
• Review of a print out of the data– Review of any “flagged” data – Review of the high and lows of the data
7
13
Ambient Air Quality System Training QA Strategy WorkgroupQA Strategy Workgroup
Ambient Air Quality System Training
266.845-1.043.451.5913:00:00 PM275.744-1.843.221.8312:00:00 PM294.242.8-1.753.261.8311:00:00 AM332.740.6-1.563.622.0310:00:00 AM39138.1-0.934.012.129:00:00 AM51-0.932.31.966.852.868:00:00 AM59-1.731.52.375.421.767:00:00 AM61-1.432.21.514.551.476:00:00 AM66-1.731.41.544.791.665:00:00 AM62-1.133.51.94.141.174:00:00 AM510.438.3-32.733.156.323:00:00 AM520.837.81.514.451.282:00:00 AM551.335.21.84.881.341:00:00 AM532.133.82.015.861.5300:00 AM
RH (%) 2m TempO3 (ppb)NO2N0y (ppb)NO (ppb)Time
14
Ambient Air Quality System Training QA Strategy WorkgroupQA Strategy Workgroup
Ambient Air Quality System Training
Step 4: Graphic Analysis • Graphs can be used effectively to:
– Look at trends of data– See how interactions occur
• For instance: NO2/NO interaction with Ozone• NO tritrates in the presence of Ozone to form NO2• If NO generally will not be above baseline unless Ozone is
tritrated out.
8
15
Ambient Air Quality System Training QA Strategy WorkgroupQA Strategy Workgroup
Ambient Air Quality System Training
Step 4: Graphic Analysis
16
Ambient Air Quality System Training QA Strategy WorkgroupQA Strategy Workgroup
Ambient Air Quality System Training
Step 4: Graphic Analysis
9
17
Ambient Air Quality System Training QA Strategy WorkgroupQA Strategy Workgroup
Ambient Air Quality System Training
Step 4: Graphic Analysis
18
Ambient Air Quality System Training QA Strategy WorkgroupQA Strategy Workgroup
Ambient Air Quality System Training
Step 5: Comparison Program • Useful when there are more than one station in a
network– Used extensively in PAMS networks
• Expect Ozone and precursors to go up at one station• Migrate to upwind sites
– Compare meteorological parameters• Expect wind from one direction at multiple sites
– Internet Options• AirNow Tech
10
19
Ambient Air Quality System Training QA Strategy WorkgroupQA Strategy Workgroup
Ambient Air Quality System Training
20
Ambient Air Quality System Training QA Strategy WorkgroupQA Strategy Workgroup
Ambient Air Quality System Training
Step 6: Data Base Screening • Most Central Database program have:
– Screening routines that can allow • User to compare datasets against other data • User to compare datasets against historic data
– Other programs are available • Use of Excel to compare data• Use of SQL or SAS languages to screen for outlier
values
11
21
Ambient Air Quality System Training QA Strategy WorkgroupQA Strategy Workgroup
Ambient Air Quality System Training
Step 6: Data Base Screening • A few techniques
– Does O3 rise when NO is dropping?– Are NO/NO2 high in morning and evening?– Are there any negative values? – Does O3 correspond to high/rising Temp or Solar
Rad?– Did the Station temp exceed 30o C?– Does precipitation correspond to low Ozone, SO2,
CO or NOy?
22
Ambient Air Quality System Training QA Strategy WorkgroupQA Strategy Workgroup
Ambient Air Quality System Training
Step 7: Final Evaluation • This is the last step of the process• Usually it is the time to make final
adjustments or invalidate data• Compare data against the known standards
– Zero/span checks– One point QC checks (i.e., precision checks)– Flow Checks– Multipoint Checks– Audits– Routinely collected instrument diagnostics
12
23
Ambient Air Quality System Training QA Strategy WorkgroupQA Strategy Workgroup
Ambient Air Quality System Training
Step 7: Final Evaluation• Validation tables
– State the MQOs of your system
– Give you a reference to the needs of the data users
– Three different types: • Critical • Operational• Systematic
Validation
24
Ambient Air Quality System Training QA Strategy WorkgroupQA Strategy Workgroup
Ambient Air Quality System Training
SO2 - Span Drift (at 90 ppb) - EPA Burdens Creek
-15.0%
-10.0%
-5.0%
0.0%
5.0%
10.0%
15.0%
1/1/
2007
1/15
/200
7
1/29
/200
7
2/12
/200
7
2/26
/200
7
3/12
/200
7
3/26
/200
7
4/9/
2007
4/23
/200
7
5/7/
2007
5/21
/200
7
6/4/
2007
6/18
/200
7
7/2/
2007
7/16
/200
7
7/30
/200
7
8/13
/200
7
8/27
/200
7
9/10
/200
7
9/24
/200
7
10/8
/200
7
10/2
2/20
07
11/5
/200
7
11/1
9/20
07
12/3
/200
7
12/1
7/20
07
12/3
1/20
07
Pere
cnt
SO2-TEI-43C TLE
Span drift MQO Goal = + 10%
13
25
Ambient Air Quality System Training QA Strategy WorkgroupQA Strategy Workgroup
Ambient Air Quality System Training
Step 7: Final Evaluation
26
Ambient Air Quality System Training QA Strategy WorkgroupQA Strategy Workgroup
Ambient Air Quality System Training
Summary• Verification and Validation are a process!• This process is used to make sure the data
collected are for its intended use!• Utilize all of your tools!!!
– Collected instrument diagnostics– Use of DAS screening techniques– Manually review of data– Use graphic displays – Use central database tools – Use validation MQO tables
Data are ready to go into AQS!!
