Institute for Tribal Environmental Professionals

Why QA? Quality systems Quality assurance activities Quality control activities Objectives for your measurements (QMPs) QC calculations: precision, bias, and

accuracy QAPPs, QMPs, and other monkeys

The purpose of QA is to help you get the data you need to answer the questions you have as cheaply and quickly as possible

QA done right does save you time and money QA makes you:

◦ Plan ahead◦ Ensure everyone agrees to the

goals and equipment used, what their jobs are, the schedule, and who reports what to whom when

QA = quality assurance QC = quality control Quality System includes both QA activities and

QC activities, which often overlap A Quality System = a good management

system, which will produce:◦ follow up on problems with permanent solutions, ◦ complete documentation, ◦ everyone knowing what their responsibilities are,◦ making sure all data and reports are reviewed before

going out to the public or EPA,◦ Data that answers the question you are asking about

your environment

Quality Assurance, which are more management type activities, overseeing:

Quality Control, which are actual checks of instruments, checking battery voltage, etc.

QC is one part of QA In a two-person program, both people will

be doing QA and QC, just as part of doing a good job

Category 1 projects are to compare against the NAAQS standard, and require all 24 elements

Category 2-3 projects would include looking at air pollution transport, air toxics, exploratory studies, or evaluating whether further measurements should be made, and require even fewer of the 24 elements of a QAPP

Category 4 projects are education and outreach and may make a few qualitative measurements and only require 6 elements of a QAPP (see example on course website)

7

Important Decisions

Less Important Decisions

National consensus standard authorized by American National Standards Institute (ANSI)

Developed by American Society for Quality (ASQ)

Consistent with ISO standards (there are ISO standards for everything, tires, toilets,

EPA Guidance for Quality Assurance Project Plans: EPA QA/G-5 based on E4

(just so you know)

Anyone expecting their data to be respected must have some form of quality system (remember this is just good management plus documentation, in a QAPP)

Anyone receiving grant funds from a federal agency must have a quality system

Even 1-person programs can have data that is legally and scientifically defensible by following these guidelines

John W. SmithDirector

Your Tribe

Alexandria WashingtonTribal Environmental

Program Manager EPA as appropriate

Michelle WinstonTribal Air Quality

Specialist

Regina LambTribal Air Quality

Technician/QA Coordinator

Laboratory, if weighing filters

or analyzing samples

Bank balances: the goal of our personal quality system is to not bounce checks

QA are the activities of depositing checks, checking bank balances every week, planning ahead about what bills to hold off on paying

QC is routinely balancing our checkbooks and calculating whether checks written or online payments are going to bounce

Tribal Council Tribal

Environmental Office director

EPA Region

Governmental Entities, Contractors, and Key Individuals.

Roles and Responsibilities. How often will these be done? How will each person do their job? To whom will they report?

Weighing or Analysis Laboratory Audits of equipment (performance audits,

in which they compare their hand-held standard device against your device and make sure they agree)

Oversees monitoring project Prepares or reviews quarterly & annual

reports for submittal to EPA Ensures staff is hired and trained

Operates and checks equipment (checking batteries, flow rate)

Prepares QA reports (this may be half a page, saying all is well, flow rates are stable and within the ranges in the owner’s manual)

Does data verification, validation and assessment (we will explain these terms more later, but they are basically checking your equipment, looking carefully at what you are doing and making sure the data are answering the question you have about your environment)

Independence: EPA guidance (and common sense) dictate that there should always be a second person who reviews the data and is not involved in routine data-gathering.

If only 1 person, independence might be done by working with another dept. (water?) and exchanging an afternoon every two weeks to review each others’ data

May sound complicated, but are basic common sense:

Knowing who is to do what, When and how they are supposed to do it Where they write down what they did How they know if what they did shows that

the equipment is operating as it is supposed to

Who is supposed to fix things and where they write down what they did

Why are we doing this?

Why are we making these measurements?◦Is our air over a standard?◦Obtain baseline data◦To determine the need for additional

monitoring◦Health risk evaluation: Report to

community, EPA, health officials

Precision (wiggle) Bias (jump)Accuracy (total error, or wiggle and jump together, “evened out” over time)

Precision =“wiggle” (variability within many measurements of the same thing)

You are trying to estimate the variability within the population of “all” your measurements of the same thing (concentration)

Two ways to estimate precision for a single instrument◦ If you have enough equipment, side-by-side,

can be two or more devices measuring the same concentration

◦ If you have only one continuous instrument, you must estimate precision by how much the measurement fluctuates over time when it is measuring the same concentration?

Bias=“jump” (consistent difference between your result and the “truth” or a standard)

Bias = how far from “truth” (some handheld standard) you are, in terms of percentage

Bias = your result – “truth” “truth”

Expressed as a percentage, so multiply by 100. 0.03 would be 3%0.11 would be 11%

Terminology and Requirements

Planning: QA Project Plans

Doing: QA Project Plans Standard Operating Procedures QA Annual Report and Work PlansChecking: Management Assessments Technical Assessments Data Quality Assessment Data Validation and Verification

Usually can combine with QAPP; this is up to the grantor (may be US EPA Regional office)

EPA Requirements for Quality Management Plans (QA/R-2)

Purpose: To document type and quality of data for environmental decisions; a blueprint for the requirements for collecting and accessing data

Responsibility: Organization performing activity

Documentation: Turbo-QAPP

EPA Requirements for Quality Assurance Project Plans (QA/R-5)

Guidance for Quality Assurance Project Plans (QA/G-5)

explains how environmental data collection activities are planned, implemented, documented, and assessed (checked)

QA Project Plans are required when environmental data operations occur for:◦Contracts, work assignments, delivery orders◦Grants, cooperative agreements◦Interagency agreements (when negotiated)◦Tribal-EPA agreements providing funding◦Responses to statutory or regulatory

requirements and to consent agreements

Questions to be answered:◦Who is making the decision?◦Why are data being collected?◦What data are needed to make the decision?◦Why does the decision maker need that type

and quality of data?◦How does the decision maker plan to use the

data to make a defensible decision?◦What are the "measures of success" for the

project?

Get only the type, quantity, and quality of data necessary

Purpose: To document routine technical and administrative activities to ensure consistency in the quality of the

productResponsibility: Appropriate technical personnel working with QA Manager

Documentation: Guidance for the Preparation of Standard Operating Procedures

(QA/G-6)

Written documents that clearly describe what you will do

Detail who will do what when things go wrong

If different people do it, but follow the same SOP, it should result in the same answer

You can print out once, and then revise in pen and photocopy if different people do the work

Can be short, even ONE PAGE or less

Plan, do, check, fix, write it down

Measuring Comparing with an

objective, such as +-10% Graphing it Fixing it when needed

Everything must be documented and, when significant, taken into account in the reports

Preparing for the field Sampling in the field Gathering numbers Entering the data into computer

spreadsheet Reporting the data

Field

Lab

Cows can be problems….

Estimate the random “wiggle” error If you have more than one of the same

type of instrument, place side-by-side (measuring the same sample of air within a room or small area)

If you only have one instrument make repeated measurements (same sample of air, quickly in time so the air does not “change”)

Verify that results are within limits (do they agree at all? Or are they perfectly the same?)

Measure anything that affects the result outside of what you are measuring

May make the result greater (contamination)

Or decrease the result

For real-time measurements zero checks display the value with no air

Manual methods using filters must use field blanks—accompany “real” samples

Labs must measure their own blanks to assess whether there is any contamination in the lab

If it is possible that samples get damaged or contaminated during shipping use shipping blanks (trip blanks)

Handled exactly as field samples Some field blanks go everywhere field

samples go With each operator, site, procedure

“if you didn’t write it down, can you prove it happened?”

(1) Data processing (WHAT)(2) Data end use (WHY) (3) Data access (WHO) (4) Data distribution (WHERE)(5) Data storage and retrieval (HOW)

(6) Data disposal (WHEN)

Site Data Sheets ◦ document the site information◦ in site files and in computer

Sampler run data sheets◦ go into field◦ input info into database

Verification data sheets◦ any checks that you make of flow rate, temp,

pressure, battery checks, zero checks

Review the number of each item you will need, and bring backups (use a checklist)

Check field data sheet from previous visit to site

Ensure that there are enough materials for routine, field blanks, and collocated samples

Ensure there are enough field transport containers, ice substitutes, max/min thermometers, preprinted mailing labels, if mailing immediately

Continuously update the checklist in pen◦make photocopy◦put in “to-do” pile to

add the information to the database change the form in the computer if appropriate

At the site draw a map on the field data sheet

Take digital photos-these are GREAT data management tools

Every two weeks or as data are gathered—add “back up today” to calendar

Alternate between two sets of backup USB drives, use CDs for permanent archive data storage

Store archive disks in a relatively fireproof location (another building, garage)

Plan for the time and money it takes to save copies of files

Checking up on yourself

Basically compare◦What is actually being done in the field and the office

◦Against what is stated in the QAPP and SOPs

◦Can be done by you or someone outside your office

Number, frequency, and types of assessments People or organizations doing the assessments Schedule Criteria for assessments (we will compare what we do against the SOP or QAPP) Reporting and responsibility for follow-up (we will revise the SOP in pen for now, and retype it with changes within a week)

CLOSE THE LOOP (fix problems and take action to make sure

they do not happen again)And make sure it is documented

Demo how to download sections at e-CFR Copies available in course website http://itep68.itep.nau.edu/itep_downloads/

51

Anyone expecting their data to be respected must have some form of quality system (remember this is just good management plus documentation, in a QAPP)

Anyone receiving grant funds from a federal agency must have a quality system

Even 1-person programs can have data that is legally and scientifically defensible by following the guidelines

52

53