2011 LLNL Template

The Importance of Having All Data Users Know the Analytical Data Quality Objectives: Lessons LearnedDOE Analytical Services Program 2014 WorkshopDon MacQueenUCRL-VG-137402

This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under contract W-7405-Eng-48.Lawrence Livermore National LaboratoryUCRL-VG-134702#1Once upon a time...There was a site

At which three organizations decided to collect some samples for a radiological assessment

And split them

But the results didnt come out quite as well as they had hoped...Lawrence Livermore National LaboratoryUCRL-VG-134702#1995 Results

Lawrence Livermore National LaboratoryUCRL-VG-134702#1995: 19 Soil SamplesSplit in the field between three organizations

Lawrence Livermore National LaboratoryUCRL-VG-134702#Analyzed for 239+240Pu, 238PuKey questions for 239+240Pu:

Is it above 2.5 pCi/g?i.e., above a derived-from-risk screening level?

Is it above 0.01 pCi/g?i.e., is it above fallout background?More precisely, is it above a value being used to represent an upper limit for fallout background?

How did the Pu reach the site?Lawrence Livermore National LaboratoryUCRL-VG-134702#Method and resultsTotal dissolutionChemical separationAlpha spectroscopy

Lab A and B results were surprisingly differentLab A and C good agreementLawrence Livermore National LaboratoryUCRL-VG-134702#Locations 4 and 13

Lawrence Livermore National LaboratoryUCRL-VG-134702#Locations 3 and 9

Lawrence Livermore National LaboratoryUCRL-VG-134702#Locations 7 and 8

Lawrence Livermore National LaboratoryUCRL-VG-134702#Why are A and B so different?Data are obviously of different quality

Organization A didnt really understand whyNoted that counting times were differentSpeculated about particulate nature of radionuclides in soil

Noted that some Lab B QC showed large variation

Organization A wrote a report, used uncertainty-weighted average of Lab A and B resultsLawrence Livermore National LaboratoryUCRL-VG-134702#1998: Organization DWas not involved in any way in 1995

Reviewed data, released a draft report for public comment

Used only Lab B resultsCounted number of samples above 0.01 pCi/g

Reached a very different conclusion about how the Pu reached the siteLawrence Livermore National LaboratoryUCRL-VG-134702#Alternate InterpretationsBased on Lab B results13 of 19 above 0.01 pCi/g, ignoring uncertainty6 of 19 above 0.01 pCi/g, considering uncertainty13/19 = 68% = throughout the site = air pathway

Based on Lab A results6 of 19 above 0.01 pCi/g, ignoring uncertainty6 of 19 above 0.01 pCi/g, considering uncertainty5 of the 6 located consistent with sludge pathway Lawrence Livermore National LaboratoryUCRL-VG-134702#Locations 4 and 13

Lawrence Livermore National LaboratoryUCRL-VG-134702#Locations 3 and 9

Lawrence Livermore National LaboratoryUCRL-VG-134702#Details from two samples

Lawrence Livermore National LaboratoryUCRL-VG-134702#Counts to exceed 0.01 pCi/gIn order to exceed 0.01 pCi/g (the so-called background)Lab A needed 20 to 30 net countsLab B needed 2 to 4 net counts

Is it valid to use Lab B results to make above/below background inferences?Not alwaysLab B results are much more variable at low levelsExample: 0.0105 0.0189 with MDC = 0.0438 was considered evidence of being above 0.01 pCi/g by Organization DLawrence Livermore National LaboratoryUCRL-VG-134702#Data quality objectivesInstructions to Lab A:MDC should be 0.005 pCi/g or lessThis is a surrogate for good performance at low levels

Instructions to Lab B:Tell us if it is above 2.5 pCi/gThis implies that large variability at low levels is acceptableLawrence Livermore National LaboratoryUCRL-VG-134702#Lab implementation of DQOsLab A5 gram aliquot24 hour counting time

Lab B0.5 gram aliquot16 2/3 hours counting time

This explains why the two labs have such different results on many samples, and why Lab B results are so variable

Aliquot size is the more crucial of these Lawrence Livermore National LaboratoryUCRL-VG-134702#Particulate Sampling

From Pitard, F., 1993 as reproduced in Myers, J., 1997Don MacQueen:Note, demonstrate concept visually here using bag of granola, with raisins as the contaminantLawrence Livermore National LaboratoryUCRL-VG-134702#What went wrongLab Bs results do not measure small differences near background precisely, but this was not well understoodMost especially, by Organization D!Based on their draft report, Organization D did not attempt to understand DQOs, or differences between data setsOrganization A did not understand well enough soon enough to effectively discuss Ds interpretationOrganizations A, B, and C had little or no discussion of DQOsOrganizations B and C did not raise the DQO issue when it would have helpedOrganization As 1995 report would have been better if they had understood the DQOs more thoroughlyLawrence Livermore National LaboratoryUCRL-VG-134702#What about the analytical labs?All of the labs met their clients requirements

It is the clients responsibility to understand and share DQOs

But, Lab B could have helpedLab B was aware that comparisons with 0.01 pCi/g were of interestLab B project manager did make comparisons with 0.01 pCi/g in the analytical report cover letter

Lawrence Livermore National LaboratoryUCRL-VG-134702#First lessonWhen splitting samples, clients should insist on the same data quality objectives. Organization A did not do this.

If the DQOs are not the same, make sure everyone understands the limitations (and document them).Lawrence Livermore National LaboratoryUCRL-VG-134702#

We learned our lesson: 1998Lawrence Livermore National LaboratoryUCRL-VG-134702#SummaryData users were not sufficiently aware of the 1995 data quality objectives

Data users did not understand the limitations of Lab Bs 1995 results (limits that were specified by Organization B and met their needs)

Misuse of Lab Bs results enabled Organization D to introduce a huge red herring into the site assessment processBecause of failure to consider the data quality objectivesLawrence Livermore National LaboratoryUCRL-VG-134702#Lessons learnedThe clients responsibility does not end at the labs front door

Good scientific interpretation of a number requires full knowledge of how the number was generated

Coordinate, Cooperate, COMMUNICATE!

Lawrence Livermore National LaboratoryUCRL-VG-134702#OrganizationA(LLNL)BC

LabA(not LLNL)BC

Number of samples19197

Net countsBackground CountsResult

A210.0007 0.0014

B2.3350.0105 0.0189

A1110.0036 0.0023

B4.3320.0111 0.0115