Tutorial 3-3 Importing Field and Lab Chemistry Data and QA · Tutorial 3-3 Importing Field and Lab...

Tutorial 3-3Importing Field and Lab Chemistry Data and QA

For use with ESDAT version 5 (version 5 software updates available from www.esdat.com.au)

Tutorial 3-3 Importing Field and Lab Chemistry Data and QA www.esdat.com.au

Copyright @ EarthScience Information Systems P a g e | 1

Tutorial 3-3 Importing Field and Lab Chemistry Data and QA

Contents

Summary ....................................................................................................................................2

Key Learning...............................................................................................................................2

Pre-requisites .............................................................................................................................2

Introduction ...............................................................................................................................3

Exercises.....................................................................................................................................4

Exercise One: Importing Field Chemistry ...............................................................................4

Exercise Two: Importing a Laboratory Chemistry Data File ...................................................7

Step 1 – Importing Sample Information.................................................................................8

Identifying and Reconciling QA Samples................................................................................9

Step 2 – Importing Chemistry Results ..................................................................................10

Updating or deleting Lab Chemistry data ............................................................................13

Exercise Three: QA Checking and Reporting in ESdat ..........................................................14

Exercise Four: Unit Conversion ............................................................................................18

Units Correction ...................................................................................................................18



SummaryThis tutorial:

examines importing field data and laboratory chemical data, reviewing QA issues andgeneration of QA reports

is performed in the Sample Hydrogeology and Contam dataset in Access mode (andtherefore can be completed using the demonstration version of ESdat)

applies to both Access and SQL Server versions of ESdat

requires approximately 90 minutes to complete

Key LearningFollowing completion of this tutorial, the user will have developed the following ESdat skills:

importing of field data recorded in hard copy format

importing laboratory analytical data

completing quality control (QA) reviews on data

quality control (QA) reporting

filtering data (note – filtering is covered in greater detail in other tutorials)

conversion of data units

Pre-requisitesIt is assumed the user has previously completed:

Tutorials 3-1 Analysing Chemistry and 3-2 Importing Historical Chemistry should becompleted in conjunction with this tutorial.

For a full list of the Tutorials available for use with ESdat, please see the Tutorial List.

Tutorial 2 – Introducing ESdat

http://www.esdat.net/esdathelp/index.html?tutorials.htm



IntroductionESdat users typically have a range of data types to be managed, including the followingfundamental data types:

Laboratory analytical reports in ESdat format

Field instrument measurements (such as water levels or water quality parameters)

This range of data must be imported into ESdat in a consistent and coherent way tomaintain data integrity within the database.

In addition, ESdat users will generally apply various Quality Assurance (QA) procedures andprocesses (and associated Quality Controls) to provide adequate confidence in the reliabilityand suitability of data for the chosen purpose.

These QA procedures and processes may apply to both field activities (such as soil orgroundwater sampling) and to laboratory analysis.

ESdat features functionality designed specifically to address these requirements.

In this tutorial, a series of exercises have been developed to reflect typical ESdat userscenarios relating to importing field data, laboratory analytical data and performing QAchecks and generating associated reports.

These are considered key skills for efficient use of ESdat.

The exercises are summarised as follows:

Exercise 1

Groundwater parameters measured in the field and recorded in hard copy are imported intoESdat.

Exercise 2

Electronic laboratory analytical data received in the ESdat Electronic Lab Data Format (ELDF)is to be imported into ESdat.

Exercise 3

The data imported in Exercise 2 will undergo Quality Assurance (QA) checking and a QAreport will be produced.

Exercise 4

Groundwater field parameter data received in incorrect units is converted to units to ensureconsistency with other measurements of the same parameter.

(Note – EDLF are specifically formatted Excel based laboratory analytical reports provided byanalytical laboratories to be imported directly into ESdat)

http://www.esdat.net/Lab_Specifications.aspx



ExercisesExercise One: Importing Field Chemistry

Field data can be imported into ESdat using an Import Template.

ESdat provides a number of pre-formatted Excel work sheets, called Import Templates thatare used to compile and upload data into a database using ESdat.

Data can be copied/cut then pasted into the Import Templates, or entered manually. In asubsequent tutorial a form is created to automatically populate an Import Template.

For exercise 1, the Field Chemistry Import Template will be used.

Open ESdat and

Select the Sample Hydrogeology and Contam project

Select Import from the View Type tool bar (the top tool bar, as shown in the figurebelow)

Prior to opening an Import Template, user-required Fields can be selected to be included inthe Import Template. (Fields are specific data types, such as Matrix type or Date/Time thatare arranged in columns within import templates).

Select Setup from the top left of the screen

Navigate to chemistry then sample fields shown, which opens the Default Fields forChemistry Import dialogue box (as shown in the figure below)



Select the fields as shown in the figure below

Click Update to Save your settings and Close the Default Fields for Chemistry Importdialogue box

Return to the Import View

Click the Field button (as shown in the figure below)

This will open a blank Field Chemistry Import Template (a pre-formatted Excel sheet).

Type or copy & paste the following data into the Field Chemistry Import TemplateLocCode Sampled_Date-

TimeSample DepthRange

DissolvedOxygen (mg/L)

Temperature

(C)

pH

BH01A 01/01/2004 3-8 6 21 8.1

BH02 02/01/2004 4-9 7 22 7.9

BH04 05/01/2004 5-8 5.5 23 6.9

BH05 10/01/2004 5-10 6.7 20 7.7

BH06 11/01/2004 2-5 5.4 19 7.1

BH11 12/01/2004 4-7 6.2 22 6.8



Take care to enter data into correct columns. In particular the A from BH01A is a WellCode.Sample Depth will need to be entered into Depth From and Depth To, and Matrix Type willneed to be specified (water).

Copy the SampleCode formula from cell A6 down to all records so the requiredSampleCode is generated for all 6 records (as shown in the figure below)

Go to the Add Ins tab at the top of the page and click on this to open the CustomToolbar (as shown in the figure below)

Click the Upload button to open the Upload Data dialogue box

Click Import

Address any errors by reading the error comment on each purple Cell.

Close the Excel spreadsheet (without saving)

The imported Field Chemistry data can now be viewed within ESdat.

Click the Data Views from the View Type tool bar

Click the Water Chemistry Data Types button to select LChem1_Chemistry

(Note - LChem1_Chemistry is selected by default - hovering cursor over the Water Chemistrybutton will present more options)

Filtering by Method_Type or Method_Name and by Field will present the field chemistrydata.

Go to Filter Panel (click on the grey tab labelled Filter (OFF), as shown below)

Click Method_Type in the list of Fields (about half way down the list to the left)

A list of method types will appear in the values list area (to the right)

Double click on the value Field to create the following filter in the Filter Text Area

Method_Type = ‘Field’ Click Apply Filter



Go to Data Panel (click on the grey tab labelled Data) and view the imported fieldchemistry data

Exercise Two: Importing a Laboratory Chemistry Data File

Laboratory chemistry (analytical reports) and related quality assurance data generated by alaboratory (in Electronic Lab Data Format or EDLF) can be imported directly into ESdatdatabases.

Data can then be manipulated in a variety of ways for analysing within ESdat, such as thepreparation of analytical results tables for comparison with criteria, or to undertakestatistical analysis.

All laboratory data in ESdat are associated with a Lab Report Number.

A Lab Report Number is a unique number associated with a specific laboratory certificate ofanalysis. The Lab Report Number is generated by the analytical laboratory.

In general, for each Lab Report Number, multiple (2-4) .csv files are generated as the ELDF.

In this exercise ELDF laboratory chemistry data (analytical results) will be imported to thedatabase.

Select the Import View Type

Click the orange Laboratory EDD button (as shown in the figure below) to open a filebrowser

Find, select and open the file: Tute2.LABORATORYAA.Sample22.csv

The file will typically be in:

C:\Program Files (x86)\ESDAT\Sample Contam

An ESdat dialog box will open if this data already exists (as shown in the figure below).

Click Yes to proceed



Laboratory data will be displayed in an Import Template that will open, allowing furtherreview before the actual importing.

Note that Chemistry_Samples and Chemistry_Results will be in separate Excel work sheets(as can be seen in the two tabs at the bottom of the Import Template spread sheet).

There are two steps to successfully import the laboratory analytical chemistry data, asfollows:

1. Import the sample information (contained within the Chemistry_Samples sheet)

2. Import the analytical information (contained within the Chemistry_Results sheet)

The two steps are described as part of the continuation of Exercise 2, as described in thefollowing sections.

Step 1 – Importing Sample Information Select the Chemistry_Samples tab at the bottom of the Import Template and review

the data

The entry with the column labelled Field_ID reflects how it is generally provided tothe laboratory on a Chain Of Custody form. In this case it is created with acombination of:

a. Location Code (such as BH01)

b. Sample_Depth_from

c. Sample_Depth_to

Note the entries within the Field ID and LocCode are identical (as shown in the figurebelow).



Before importing the data the location and depth details need to be separated among therespective columns. ESdat has a tool to do this (note - when importing data though LSPECS(if installed) this is done automatically).

Select the Add_Ins tab at the top of the page and click on Split button (as shown inthe figure below)

The Field_ID values will remain unchanged, but the Location (Location Code) and Depthcomponents will update in the correct columns.

Check the data for any errors in this automated process and make corrections ifnecessary

Identifying and Reconciling QA Samples

Field QA samples, such as Field Duplicates (also known as Intra-laboratory Duplicates), FieldBlanks, Trip Blanks, Inter-laboratory Duplicates (also known as triplicates) need to bespecified and reconciled by the user.

For example, sample QA1 is a Field Duplicate of primary sample BH05_1.3-1.4 and QA2 is aField Blank.

This information needs to be entered into the Import Template.

Click on the cell of Sample_Type column for sample ‘QA1’ (Cell I7), which willgenerate a drop down menu

Select Field_D as Sample_Type to define it as Field Duplicate

A prompt to specify (reconcile) the Parent Sample of this Field Duplicate will appear.

Click cell A5 to reconcile QA1 with BH05_1.3-1.4

Click OK to close the Field Duplicate dialog box

Note the LocCode is replaced by BH05 and the SampleCode for BH05_1.3-1.4 will be copiedin Parent_Sample column of QA1.



Click on the cell of Sample_Type column for sample QA2 (Cell I8), which will generatea drop down menu.

Select Field_B as the Sample_Type to define it as Field Blank (note the LocCode isdeleted, as a Field Blank samples have no specific location)

Click the Upload button under Add_Ins menu to upload chemistry samples intoESdat

Make sure Add New and Require Parent Records is selected

If prompted, correct any errors and re-import the data

Step 2 – Importing Chemistry Results Select the Chemistry_Results tab at the bottom of the Import Template and review

the analytical data (as shown in the figure below)

Click the Upload button under Add_Ins menu to upload chemistry results intoESdat

Make sure Add New and Require Parent Records is selected

Click Import

Close the Upload Data dialogue box

Some records will not be imported as they do not have a ChemCode or have invalidChemCode (the cells containing records that have not been imported will the shadedpurple).



ChemCode provides a unique code for each compound/analyte. This is usually a CAS(Chemistry Abstract Service1) code. All chemistry data imported into the database areassigned ChemCodes.

ESdat contains many previously assigned ChemCodes, however some analytes (chemicals)will require the assignment of a ChemCode, which can be completed as follows:

Within Excel, click the Get ChemCode button (located under the Add_Ins menu asshown in the figure below)

This will open a new dialogue box, entitled Associate Chem Codes (as shown in the figurebelow).

In the main (left) frame, the analytes that could not be imported from the Import Templateare listed in the left Original Chem Name column.

Some Original Chem Name values have automatically had a ChemCode assigned byreconciling with a ChemName. Review them. Dimethylbenzene has no matching ChemCodein the right panel.

If a ChemCode has not been automatically assigned, ESdat provides links to an onlinedatabase.

Click on Dimethylbenzene, which will appear in the Lookup box at the bottom left ofthe window (as shown in the figure above)

Also located at the bottom of the window are four (blue) links for searching variousreferences for an appropriate ChemCode.

Start from the first option (using NIST) by clicking on the link

1 http://www.cas.org/



ESdat will automatically search the chemical name in the online database.

Note that Dimethylbenzene is referred to as Xylene in the NIST ChemCode database and isthen automatically selected in the Associate Chem Codes dialogue box, as shown in thefigure overleaf.

Close the NIST window

Click on the grey Assign button to assign the Chemcode (the button located atthe centre right of the Associate Chem Codes dialogue box, as shown above)

If no matching ChemCode is located at any of the provided links, other external sourcesmay be used (at the users discretion), or new ChemCodes can be created by the user.

To create a new ChemCode and add it as new Analyte see Help Portal

http://www.esdat.net/esdathelp/index.html?chemistrypreimportutilities.htm



Click Update to update the ChemCodes in the Import Template, ChemCodes shouldbe assigned to all Results.

Click the Upload button under Add_Ins menu to upload chemistry results intoESdat

Click Import and close the Upload Data window

Close Excel without saving

The laboratory analytical chemistry data can now be viewed in ESdat.

Return to ESdat and select Data Views from the View Type tool bar

Hover over the Chem Tracking View Type button (as shown in the figure below) andselect the ChemQA2_Chemistry_Samples_with_Raw_Results table

Create the following filter within the Filter Panel and click Apply Filter

Lab_Name = 'LABORATORYAA'

View the raw data in the Data Panel

A list of Lab Reports or SDGs (Sample Delivery Groups) can be viewed in the ChemQA3 andChemQA4 tables, respectively. These tables are accessed via the drop down list that appearswhen hovering over the Chem Tracking button, as can be seen in the figure below.

Hover over the Soil/Rock Chemistry Data Type button and click onSChem1_Chemistry table to view Soil/Rock chemistry results

Updating or deleting Lab Chemistry dataShould a laboratory analytical chemistry data need to be reimported, it is recommendedthat the entire analytical report be deleted from ESdat and then be re-imported.

Re-importing laboratory results with the Overwrite option is not recommended (it should beused only in specific cases described in other tutorials), as it may leave some of the previousdata in the database. See here for more information.

http://www.esdat.net/esdathelp/index.html?deleting_lab_reports.htm



Hover over the Chem Tracking View Type button and select ChemQA4_Lab_Reportstable

The laboratory report that has been imported will be listed under the Data panel (the first ofthe three grey tabs located at the upper left of the window).

Select the row for the report to be deleted

Click the edit button at the left of the screen (will turn red when selected)

Delete the relevant row (right mouse click on the far left grey cell of the row)

(Note: this next step can be completed at a later date unless a mistake was made duringimporting data at the start of this tutorial)

Re-import the data as per the steps described in Exercise 2

Exercise Three: QA Checking and Reporting in ESdatESdat is able to automatically check field QA results such as:

o Field duplicates and triplicates (i.e. intra and inter-laboratory duplicates)

o Trip spikes

o Trip blanks (or field blanks)

o Rinsate samples

In addition, ESdat is able to check laboratory QA results including:

o Laboratory blanks

o Laboratory duplicates

o Matrix spikes

o Laboratory control samples

o Surrogates

o Certified reference materials

o Holding time violations

Experience in working with these kinds of QA information is generally required to accuratelyinterpret the QA data. Different jurisdictions have different requirements (or none) andreference should be made to these particular requirements.

Specific descriptions of QA measures pertaining to site contamination assessment aredescribed in this reference.

The most common method of reporting Chemistry QA data in ESDAT is to use theLaboratory QA function.

Go to Data Types button and click Reports (as shown in the figure below)



Click Laboratory QA and the ESdat Chemistry QA Checker dialogue box will open (asshown in the figure overleaf)

The tabs on the right of the panel (Duplicates, Blanks, etc.) contain standard QA measures,such as setting Relative Percent Difference (RPD) limits or determining holding timeviolations.

All the settings on these tabs are configurable in order to allow customisation of the QAprogram.

The tabs at the left of the panel (By SDG, By Date, etc.) contain batch numbers for a set ofsamples. Data for the report can be selected by SDG, Lab Report Number or sampling dates.

This allows an assessment of QA combining multiple laboratory reports, hence potentiallysaving time in QA reporting.

Select the By SGD tab

Select SGD 1266 (the selected cell will turn green)

Click QA Summery and QA Errors (lower left corner of the QA Checker panel).



An ESDAT QA Checker report will be created in in Excel (similar in appearance to the figureabove).

Blue text denotes a link to other pages with specific detail (such as Count of Samples).

To navigate back to the original home screen, click on the blue text labelled Contents.

Click on Field Blanks to view a report on Field Blank results

Return to the table of contents by clicking on Contents (cell A2)

Click on Field and Interlab Duplicates to view a report on intra and inter laboratoryduplicates

Return to the Contents

Click some of the other links to view any QA problems, remembering to return to theContents each time

Close the ESDAT QA Checker Report in Excel

On the ESdat Chemistry QA Checker panel, review the settings for the different tabs again.The option is provided here to customise aspects of both the data and error summaries, andto save these customized settings as a QA Profile.

Review and adjust the QA settings on the right panel



Save any desired QA Settings by typing a name into the QA Profile box at the top ofthe window and click the save button to the right of it

QA profiles can also be managed by clicking QA Profiles at the top left of the window.

Close the ESDAT Chemistry QA checker



Exercise Four: Unit ConversionConsistency of units within the database is critical and ESdat provides a function to allowunit conversion to avoid any discrepancies.

In the current database all Dissolved Oxygen measurements (data) are stored as mg/L.

This exercise demonstrates the unit conversion of some Dissolved Oxygen data received ing/L units.

If required, the new data can be imported as g/L and the unit conversion completed at laterstage.

The table below contains the new data received in g/L units.LocCode Sampled_Date-

TimeSample_Depth Dissolved Oxygen (g/L)

BH12 01/01/2004 5-8 0.0076

Copy the above table of data into the Field Chemistry Import Template found underImport.

Manually change the units to g/L

Import the data (via the Add-Ins tab) and close Excel

To view the data:

Go to Data View panel

Hover the cursor on Chem Tracking Data Type Button and selectChemQA2_Chemistry_Samples_with _Raw_Results

Go to Filter Panel and create a filter with:

Method_Type = 'Field' AND ChemName = 'Field DO'

Return to Data Panel and view the data.

The measurement BH12 Dissolved Oxygen 0.0076 g/L (scroll across to the right if required).

Go to Water Chemistry Data Type Button and select LChem1_Chemistry table.

BH12 will not be present in the data list as the Water Chemistry data view can only presentdata that can be converted to the predefined unit within ESdat (which is mg/L for DissolvedOxygen).

Units Correction From the View Types toolbar in ESdat and click on Check Units (the associated label

will turn red, as shown in the figure below).



The Chemistry Unit conversion Problems dialogue box will open (as shown below)

Note that ESdat has identified that Dissolved Oxygen data has been imported in the unitsg/L as opposed to the predefined to be mg/L.

The conversion factor between these two units in the Conversion column must now bespecified.

Click on the Conversion cell to open another dialogue box

Type 1000 for the conversion (1g/L = 1000mg/L)

The Unit Conversion Problem (as labelled on the grey tab) is resolved and is removedautomatically

Close the dialogue box

Click again on LChem1_Chemistry table under Water Chemistry.

Note BH12 is now displayed with the correct (converted) unit.

Congratulations! – this completes the Importing Field and Lab Chemistry Data and QAtutorial.

For more information, please go to www.escis.com.au

Tutorial 3-3 Importing Field and Lab Chemistry Data and QA · Tutorial 3-3 Importing Field and Lab...

Documents

Transcript of Tutorial 3-3 Importing Field and Lab Chemistry Data and QA · Tutorial 3-3 Importing Field and Lab...