Version 6.2 2018-11-30

Page 1 of 26

CCQM ORGANIC ANALYSIS WORKING GROUP (OAWG)

Practices and Guidelines Document

Contents

INTRODUCTION .............................................................................................................................................. 3

BACKGROUND ................................................................................................................................................ 3

OBJECTIVES OF THE CIPM MRA ................................................................................................................. 4

Working Groups within the CCQM ............................................................................................................... 4

CIPM Comparison Categories ....................................................................................................................... 4

Related CIPM Documents.............................................................................................................................. 5

OAWG PRACTICES ......................................................................................................................................... 5

Terms of Reference ........................................................................................................................................ 5

Organization ................................................................................................................................................... 6

Activities ........................................................................................................................................................ 6

STRATEGIC APPROACH FOR SELECTING OAWG COMPARISONS ...................................................... 7

The Multi Track Strategy of the OAWG ..................................................................................................... 10

PREPARING PROPOSALS FOR OAWG COMPARISONS ......................................................................... 12

Scope of Proposals for Consideration .......................................................................................................... 12

Mechanism for Suggesting OAWG Comparison or Studies ........................................................................ 13

Technical Information Required for Proposed Protocols for Comparisons or Studies ................................ 13

OAWG Comparison or Study Protocol Template ........................................................................................ 14

OAWG Process for Selecting Comparisons or Studies ............................................................................... 14

GUIDANCE FOR PARTICIPANTS IN OAWG KEY COMPARISONS OR PILOT STUDIES .................. 16

Eligibility for Participation .......................................................................................................................... 16

Call for Participation .................................................................................................................................... 16

Required Response from NMI/DIs to the Call for Participation .................................................................. 16

Distribution of Comparison Samples for Measurement by Participants ...................................................... 17

Distribution of Reporting Form for Use by Participants .............................................................................. 17

Contents of the Participant’s Submitted Report ........................................................................................... 17

Metrological Traceability Requirements of Comparison Calibrator Materials ............................................ 17

Submission of Results to the Coordinating Laboratory ............................................................................... 18

Distribution of Preliminary Results to Participants in Study ....................................................................... 18

Presentation and Discussion of Results ........................................................................................................ 18

Process for Preparing the Draft A Reports ................................................................................................... 19

OAWG Report Template ............................................................................................................................. 19

Key Comparison Reference Value (KCRV) for Track A and C Comparisons ............................................ 20

Key Comparison Reference Function (KCRF) for a Track A Model 2 Comparison ................................... 20

Process for Draft B Reports of Key Comparisons ....................................................................................... 20

Process for Drafting, Reviewing and Approving Final Summary Reports for Key Comparisons ............... 22

Version 6.2 2018-11-30

Page 2 of 26

Process for Drafting, Reviewing and Approving Final Summary Reports for Pilot Studies ....................... 22

PUBLICATION OF MEASUREMENT RESULTS FROM OAWG COMPARISON STUDIES .................. 23

ROLES AND RESPONSIBILITIES OF OAWG PARTICIPANTS ............................................................... 24

Responsibilities of the Chair, in conjunction with the vice-Chair ............................................................... 24

The following are the responsibilities internal to the OAWG: .................................................................... 24

The following are the responsibilities external to the OAWG: .................................................................... 24

Responsibilities of a Coordinating Laboratory ............................................................................................ 24

Responsibilities of a Participant Laboratory ................................................................................................ 25

Responsibilities of Participants in the OAWG Meetings and Other Activities............................................ 26

OAWG REPORTS AND DOCUMENTS ........................................................................................................ 26

Version 6.2 2018-11-30

Page 3 of 26

INTRODUCTION

This document is intended to describe the current practices of the CIPM/CCQM Organic Analysis

Working Group (OAWG). This is not intended to be a prescriptive document, but rather a descriptive

document for the agreed upon approaches and processes of the OAWG. It is intended that this

document will be dynamic and be adapted to the needs and changes in practices of the OAWG1.

BACKGROUND

The International Committee for Weights and Measures (CIPM) was established in 1875 by the Metre

Convention to oversee the implementation of the treaty (Convention du Mètre)2. The CIPM meets

regularly to ensure the implementation of the decisions of the General Conference of Weights and

Measures (CGPM) and to manage the activities of the CIPM Consultative Committees (CCs) and of

the International Bureau of Weights and Measures (BIPM).

The objectives and processes that guide the CCs are set out in the CIPM Mutual Recognition

Arrangement3 (MRA). The MRA was developed and first ratified in 1999 in response to a growing

need for a framework through which National Metrology Institutes (NMIs) could demonstrate the

international equivalence of their measurement standards and the calibration and measurement

certificates they issue and to give end users and stakeholders transparent and objective quantitative

information on the relative performance and comparability of national metrology services. The MRA

constitutes the technical basis for wider agreements negotiated for international trade, commerce and

regulatory affairs. General rules of procedure for the CCs, most of which predate the MRA, are

described in the document, CIPM-D-014.

The Consultative Committee for Amount of Substance (CCQM) was established by the CIPM in 1993

and had its first meeting in 1995. The CCQM had its origin in the working group on metrology in

chemistry which was established by the CIPM in 1991. The initial terms of reference for the CCQM

were adopted at its first meeting in 1995. The full designation of the CCQM was subsequently

clarified to be the Consultative Committee for Amount of Substance – Metrology in Chemistry and

Biology.

The organization of international metrology under the Metre Convention is illustrated on the BIPM

website5. The relationships among the CCs, regional metrology organizations (RMOs) and the BIPM

are also described therein6.

1Prior reports and report formats of the OAWG may differ from those described here which represents the OAWG’s practices as of the date of

this document. 2 http://www.bipm.org/en/worldwide-metrology/metre-convention/ 3 http://www.bipm.org/en/cipm-mra/ 4 Document only available for download via ‘search facility’ on BIPM website: http://www.bipm.org/en/about-us/ 5 http://www.bipm.org/en/worldwide-metrology/ 6 http://www.bipm.org/en/committees/

Version 6.2 2018-11-30

Page 4 of 26

OBJECTIVES OF THE CIPM MRA

The stated objectives of the MRA are to:

1. establish the degree of equivalence of national measurement standards maintained by National

Metrology Institutes (NMIs) and officially Designated Institutes (DIs);

2. provide for the mutual recognition of calibration and measurement certificates issued by NMIs

or DIs (NMI/DIs);

3. thereby provide governments and other parties with a secure technical foundation for wider

agreements related to international trade, commerce and regulatory affairs.

The technical objectives specified in the MRA are to be achieved by the CIPM through its CCs and

the Working Groups (WGs) of the CCs. The CCQM achieves these objectives through the comparison

studies and other technical activities of its WGs.

Working Groups within the CCQM

Working groups are responsible for selecting and overseeing the operation of comparison studies that

address chemical measurement-related issues important for international trade, environmental, health

and food safety-related decision making and that provide evidence for the establishment of the

equivalence of measurement results among NMI/DIs.

Currently active within the CCQM are technical working groups on Gas Analysis (GAWG), Organic

Analysis (OAWG), Inorganic Analysis (IAWG), Electrochemical Analysis (EAWG), Cell Analysis

(CAWG), Nucleic Acid Analysis (NAWG), Protein Analysis (PAWG) and Surface Analysis

(SAWG). At the 2018 plenary meeting of the CCQM it was agreed to establish a new working group

for Isotope Ratio Measurement Analysis (IRWG). Administrative working groups include the Key

Comparisons and CMC Quality (KCWG) and the Strategic Planning (SPWG) working groups. Ad

hoc groups are appointed by the CCQM to provide recommendations in specific areas as needed.7

CIPM Comparison Categories

Comparisons are the primary mechanism for establishing the equivalence of measurement capabilities

among NMI/DIs. Individual CCs and their technical WGs identify and conduct international

comparison studies that are relevant to each of their terms of reference. The various types of

comparison studies specified in the MRA are organized at the level of either the CIPM, an RMO or a

CCQM WG.8 These include:

CIPM Key Comparison (KC). A comparison coordinated by a Consultative Committee to test

principal techniques and methods9 in the field. A KC may include comparison of representations of

multiples and sub-multiples of SI base and derived units and comparisons of artifacts. A KC is

intended to provide evidence for the equivalence of the measurements made by NMI/DIs (as evinced

7 See CCQM: http://www.bipm.org/en/committees/cc/ccqm/ 8 See CIPM MRA-D-05 Measurement comparison in the CIPM MRA: http://www.bipm.org/en/cipm-mra/cipm-mra-documents/ 9 OAWG KCs and subsequent comparisons are directed to assessing the degree of equivalence of chemical measurement capabilities used by

NMIs and DIs to provide measurement services; they are not directed to assessing specific techniques and methods for the determination of

chemical measurands.

Version 6.2 2018-11-30

Page 5 of 26

by the degree of equivalence (DoE) of the NMI/DI result for the comparison with the Key Comparison

Reference Value (KCRV)), and to underpin the Calibration and Measurement Claims (CMCs)

declared by those NMI/DIs.

CIPM subsequent KC. A similar, but separate comparison carried out by a CC after completion of a

prior KC for the same measurand. A subsequent KC must include at least one NMI/DI that

participated successfully in the prior study. The results of institutes with an ongoing record of

participation are used to link the results from the subsequent comparison to the results of the original

comparison. In the OAWG area the need, appropriateness and risk of resource expenditure of these

studies will be carefully assessed before execution.

RMO KC. A comparison executed by an RMO. The link to the KCRV is obtained by reference to the

results from those institutes which have also taken part in a related CIPM KC.

RMO Supplementary Comparison. A supplementary RMO comparison to meet specific needs not

covered by other KCs, including comparisons to support confidence in calibration and measurement

certificates. An illustration of the relationship between CIPM KCs and those carried out in RMOs is

provided by the CIPM MRA scheme describing the international equivalence of measurements10.

CIPM Pilot Study. A comparison study to evaluate the suitability of methods and procedures or to

provide an opportunity for NMI/DIs to participate in studies that are new to the NMI/DI and for

participation by expert guest laboratories. Performance by an NMI/DI in a pilot study cannot

normally provide the primary evidence for equivalence of measurement capabilities or to underpin a

CMC claim.

Related CIPM Documents

A variety of general guidance documents have been published by the CIPM and CCQM that detail

the operations of Consultative Committees and their Working Groups. These cover important issues

such as CMC Excel file templates, traceability requirements, general subcontracting requirements etc.

These documents are available at the CIPM website11, and regular participating members of the

OAWG should be broadly familiar with their contents. The section on the website “Guidance on

Comparisons” should be reviewed by all coordinators of comparisons and studies, particularly the

document CIPM MRA-D-05 Measurement Comparisons in the CIPM MRA12.

OAWG PRACTICES

Terms of Reference

The primary focus of OAWG activities is the critical evaluation and benchmarking of NMI/DI

capabilities for the execution of "higher order" measurement procedures for well-defined organic

molecular entities for which the SI traceable amount of substance is to be determined. The group will

also consider, on a selective basis, similar activities for high-priority method-dependent

10 See CIPM overview on international comparisons: http://www.bipm.org/en/cipm-mra/#comparisons 11 See CIPM MRA documents: http://www.bipm.org/en/cipm-mra/cipm-mra-documents/ 12 See CIPM guidance on comparisons: www.bipm.org/utils/common/documents/CIPM-MRA/CIPM-MRA-D-05.pdf

Version 6.2 2018-11-30

Page 6 of 26

analyses/measures. “Organic molecular entities” are taken to exclude gaseous compounds,

organometallic compounds, and large bio-molecules (> 1000 molecular weight).

Organization

The OAWG is led by a Chair who is appointed, along with a vice-Chair, by the president of the

CCQM. A Rapporteur may be appointed by the OAWG Chair for individual OAWG meetings. One

or more OAWG members, also appointed by the Chair of the OAWG, represent the OAWG on the

KCWG and liaise with the OAWG on issues arising from KCWG meetings. Ad hoc subgroups

comprised of nominated OAWG members can be established when needed to develop policy

guidance and proposals for specific measurement issues.

The OAWG meets twice yearly, once in the first half of year, usually during April at the BIPM, and

once in the second half of the year (October/November) at a site of a participating NMI/DI, rotating

among participants of the OAWG.

The OAWG maintains an archive of the WG and meeting documents in a password protected CCQM

website maintained by the BIPM. An official OAWG mailing list is used to ensure that all OAWG

communications of upcoming meetings, studies, other activities, reports, etc. that may be critical for

CMC support or of interest to the participants in the OAWG activities are sent to the appropriate

parties at the CCQM NMI/DIs. Official communications, presentations, discussions and documents

are in English.

Activities

Given the primary focus of the OAWG described in its Terms of Reference and following the

guidance of the official policy documents of the CIPM for the implementation of the MRA and the

CCQM Strategy for the period 2017-2016,13 the OAWG addresses, within the enormous organic

chemical measurement “space”, the following common responsibilities of CCQM WGs:

to carry out Key Comparisons, and where necessary pilot studies, to critically evaluate and

benchmark NMI/DI competences for measurement standards and capabilities in organic

analysis; providing demonstrable evidence of the validity and international equivalence of

NMI/DI measurement services in organic analysis provided to customers and stakeholders;

to identify and carry out interlaboratory work and pilot studies to underpin the development

of reference measurement systems in organic analysis, of the highest possible metrological

order with traceability to the SI, where feasible, or to other internationally agreed units and to

support NMI/DI measurement services developed in response to customer needs;

to act as a forum for the exchange of information about the research and measurement service

delivery programs and other technical activities of the WG members and thereby creating new

opportunities for collaboration;

to provide input into the development of a CCQM strategic plan and develop and maintain a

work plan consistent with the strategic plan adopted by the CCQM;

13 See CCQM Strategy document: https://www.bipm.org/utils/en/pdf/CCQM-strategy-document.pdf

Version 6.2 2018-11-30

Page 7 of 26

to interface with other CCQM WGs and international stakeholder organizations working on

measurement issues related to organic analysis;

to support the RMOs in the critical evaluation of calibration and measurement capabilities of

NMIs for entry into Appendix C of the CIPM MRA.

These activities include:

the development and conduct of comparison studies in support of the CIPM MRA objectives

within the scope described in the Terms of Reference;

collaboratively addressing relevant issues in organic analysis of international interest;

organizing symposia and workshops on topics that support the OAWG’s objectives;

site visits to participating NMI/DIs and support of peer interaction and networking;

provision of input to the CIPM, CCQM and KCWG activities upon request.

The processes by which these activities are conducted are described in detail in the subsequent

sections of this document.

STRATEGIC APPROACH FOR SELECTING OAWG COMPARISONS

Practices of the OAWG are evolving because of the necessity to accommodate the number and

complexity of the organic measurement challenges encountered by CCQM NMI/DIs in delivering

services to their customers. The OAWG priorities established in 2016 include measurement services

in support of food safety, clinical biomarkers, food labeling, environmental and other organic service

areas (e.g., detection of drugs of abuse and contaminants in cosmetics and medicines).

Prior to the adoption of the strategy described in greater detail below, the OAWG chose comparison

studies based principally on the nature of the analytes in specific matrices at specific quantity amounts

and the relevant measurands were identified by the OAWG for each comparison. However the large

number of possible organic analytes, the variety of matrices in which they might be measured and the

wide range of substance amount to potentially be measured meant that a “one measurand at a time”

approach is not sustainable for addressing all organic chemical measurement needs. The

impracticality is clear from the breadth of CMCs that must be supported and the frequency at which

competency is expected to be demonstrated. It was recognized that a more strategic approach was

required to enable the objectives of the CIPM MRA to be met by NMIs in a manner that was fiscally

and physically practical. Discussions of a revised approach began in 2007 and began to be

implemented in 2009.

If core measurement capabilities, rather than the measurement of specific measurands, are assessed

then a strategy can be designed that results in a finite set of comparisons that can cover the OAWG’s

measurement space within a defined time period. This approach provides a mechanism suitable to

produce the evidence needed to support the breadth of CMCs and demonstration of the equivalence

of measurement results that the CIPM MRA requires from the OAWG. This is achieved through the

conduct of a set of core competencies that comprehensively address the measurement challenges

faced by NMI/DIs for the value assignment of primary calibrators and the provision of reference

measurements of organic analytes in complex matrices.

Version 6.2 2018-11-30

Page 8 of 26

In the high-purity organics measurement space this involves defining three sectors corresponding to

X (low polarity, small size), Y (high polarity, small size) and Z (all polarities, large size) that map the

organic purity space up to MW 1000 where:

X : MW < 500, non-polar (NP) (pKOW < -2)

Y : MW < 500, polar (P) (pKOW > -2)

Z : MW > 500, all polarities

Model for OAWG purity comparisons based on different measurement challenges

For calibration solutions and matrix comparisons the core comparison strategy divides the

measurement space into a series of classes of types of matrices. Dealing with different matrix effects

is considered the greatest challenge in this type of organic analysis. Sitting under this the assignment

of the mass fraction of the analyte is deemed to be the next level of measurement challenge. Finally,

management of the polarity of the analyte is considered an additional level of competency as it can

have a significant influence on the type of analytical techniques that can be employed.

Rather than attempting to develop an objective ‘scale’ for matrix type the strategy chosen by the

OAWG groups matrices initially within one of four classes – calibration solution/low interference

liquid, clinical, food or abiotic – and subsequently within a total of eleven sub-categories within these

classes. The four main classes and eleven sub-categories define a range of organic matrix material

types with associated measurement challenges that is considered to be sufficient to support current

and emerging CMC claims. These classifications are used to define a set of comparisons that can

serve to benchmark the range of competencies of an individual NMI/DI.

For food matrices, four sub-categories have been identified. The focus on this area is justified by the

high strategic priority given by NMIs to the demonstration of competency and equivalence for the

assessment of levels of contaminants and nutrients in a range of foodstuffs and primary produce.

Version 6.2 2018-11-30

Page 9 of 26

The full proposed classifications are:

1. Calibration solutions and low interference liquid matrices

organic solvent calibration solutions

aqueous solvent calibration solutions

low interference aqueous (waste waters, beverages, etc.)

2. Clinical materials

serum/plasma

other

3. Food matrices14

> 60% fat [AOAC Food Triangle Categories 1,3]

> 60% protein [AOAC Food Triangle Categories 8,9]

> 60% carbohydrate [AOAC Food Triangle Categories 5,6]

mixed matrix (no component present > 60 %, AOAC Food Triangle Categories 2,4,7)

4. Abiotic

soil, sediment or particulate

other (plastics, etc.)

Within these classifications the next level of measurement challenge is deemed to be mass fraction of

analyte in the matrix. Four ranges of interest are defined by four analyte mass fraction (w) sectors:

I. > 1 g/kg; < 1 kg/kg [ > ‰]

II. > 1 mg/kg; < 1 g/kg [‰ <, > ppm]

III. > 1 μg/kg; < 1 mg/kg [ppm <, > ppb]

IV. < 1 μg/kg [ppb <]

The polarity classifications of analytes are polar (P) (pKow > -2) or non-polar (NP) (pKow < -2).

Comparisons conducted by the CCQM working groups are assigned to three Tracks (A, C and D) that

address different needs as described in detail below. Each comparison type, regardless of the category,

are further classified as either:

Model 1 - the coordinating laboratory prepares a batch of samples which are established by

the coordinating laboratory to be of suitable homogeneity and stability for the purposes of the

comparison and an agreed number of sub-units from the batch are provided to each participant

for value assignment;

Model 2 - the participant value assigns a sample or set of samples which are forwarded to the

coordinating laboratory. The coordinating laboratory analyzes the ensemble of samples

received from all participants under repeatability conditions. The agreement of the participant

assigned values with those obtained by the coordinating laboratory is assessed.

14 The AOAC food triangle is based on the relative levels of fat, protein, and carbohydrate and is divided into nine sectors, with each corner of

the triangle represents either 100 % fat, protein, or carbohydrate content. This matrix characterization scheme does not take into account other

components (water, fiber, etc.)

Version 6.2 2018-11-30

Page 10 of 26

Unless specified otherwise, comparisons undertaken by the OAWG and described in these Guidelines

are assumed to be based on the Model 1 design.15

The Track A OAWG comparisons test the core competencies for measurement services delivered to

customers, and cover the range of recognized measurement capabilities required to deliver reference

measurement services in organic analysis. Under a Track A comparison, successful demonstration of

a competency in the comparison (e.g., mass fraction content assignment, analyte identification,

sample extraction, measurement method) is considered to be broadly applicable to the measurement

of a range of analytes in a variety of matrices, as described in the “How Far The Light Shines”

(HFTLS) associated with the comparison, provided the same general measurement approach is used.

A Track A key comparison with a Model 2 design used to assess the equivalence of the value

assignment of a designated class of CRMs or PT materials produced by NMIs/DIs were formerly

designated by the OAWG as Track B key comparisons. For the remainder of this document

comparisons of this type will be described as a Track A Model 2 comparison.

Track C key comparisons and Track D pilot studies have different, more specific objectives. Track C

comparisons assess emerging areas of global interest and importance and Track D studies are intended

as capability assessment studies for new techniques or new areas for NMIs/DIs. In theory, both of

these comparison types could invoke either a Model 1 or a Model 2 design. However, a Model 1

design in which samples are dispatched from a coordinating laboratory is the more typical practice

for the OAWG.

The Multi Track Strategy of the OAWG

Track A Key Comparisons. General Track A comparisons are intended to demonstrate the core

competencies of an NMI/DI and the equivalence of measurement procedures that support the CMCs

of an NMI/DI and the various other measurement services that the NMI/DI provides. The set of

OAWG Track A comparisons make up a suite of KCs designed to benchmark the core skills and

competencies that are needed to deliver measurement services that address the OAWG Terms of

Reference.

Specific characteristics of Track A comparisons include:

designed as part of a suite of comparisons described in the OAWG 10-year strategy used to

demonstrate the core competencies required by NMIs/DIs to deliver their measurement

services;

cover the required range of competencies through the careful selection of model systems (i.e.

specific measurands) specific for each KC;

are regularly reviewed and it is not envisaged that specific measurands are repeated;

are divided into two categories: 1) Primary Calibration (typically pure organics and standard

solutions) and 2) Accuracy Control (typically matrix materials);

15 A Model 3 comparison type, carried out by the circulation of a single sample or material between participants in the comparison for value

assignment, is also an option for CCQM or RMO comparisons but has never been implemented by the OAWG.

Version 6.2 2018-11-30

Page 11 of 26

have individual scopes that define the area of the organic measurement space that each is

intended to demonstrate competence in;

allow participants to demonstrate and benchmark their core competencies in the measurement

processes they employ for the KC. The extent to which these competencies are demonstrated

in Track A comparisons is taken into account for the support of associated CMCs;

require NMIs/DIs that deliver services within the scope of a Track A KC to participate or risk

that CMCs linked to the measurement space associated with the KC will not be supported;

make it incumbent on an NMI/DI to carefully select the measurement processes they employ

to carry out Track A comparisons to ensure that the NMI measurement services used to deliver

their CMCs are underpinned by the evidence of equivalence the KC provides;

have their Final Report published in the KCDB Appendix B and provide the basis for

equivalence determinations for relevant CMC claims for the KCDB Appendix C;

specify that only one overall value is stated on the result submission for each NMI/DI.

However, participants can report within their submission results for different methods which

are combined to give their final reported comparison result. The results of individual methods

contributing to a combined submission are reported in an Appendix to the Final Report.

oblige an NMI/DI, where the option of participation in a parallel pilot study is available, not

to change the nature of its participation from parallel pilot to KC, or from KC to parallel pilot,

subsequent to the submission of their study results to the coordinating laboratory.

Track A Model 2 Key Comparisons. These Track A Model 2 comparisons are formerly known as

Track B comparisons, and are a direct comparison of CRM services as delivered by NMI/DIs.

Verification of the equivalence of the practical realizations of measurement services provided by

NMI/DIs is one objective of the CIPM MRA, including the value assignments of CRMs and

proficiency testing materials. Such demonstration is necessary to assure the comparability and

equivalence of the characterization measurements and certification procedures used by NMI/DIs for

the provision of these services.

Specific characteristics of Track A Model 2 comparisons include:

study materials that are CRMs or PT samples value assigned by an NMI/DI;

are intended to provide independent validation of the value assignment of materials provided

by NMI/DIs as a dissemination service for the measurement services they provide;

the coordinating NMI/DI is usually a single institute: if discrepancies are identified input from

another NMI/DI may be requested to assure the absence of bias in the repeatability

measurements;

the measurement results are to be published in the KCDB Appendix B and can be used to

support equivalence determinations for relevant CMC claims for the KCDB Appendix C.

Track C Key Comparisons. Track C comparisons are conducted for specific measurands that are

considered of high importance for the global comparability of measurements or for measurands that

represent a measurement challenge that does not sit within the range of core competencies covered

through the Track A comparisons.

Specific characteristics of Track C comparisons include:

Version 6.2 2018-11-30

Page 12 of 26

minimally five NMI/DIs must agree to participate;

a narrower scope of services can be underpinned by successful participation in the KC

compared with those supported by a Track A KC. The scope is however always broader than

simply for the specific measurand;

may be used in support a limited range of core competencies;

measurement results are published in Appendix B of the KCDB and can be used to support

relevant CMC claims for the KCDB Appendix C.

Track D Pilot Studies. Track D pilot studies examine particular measurement methods or techniques

within a new area. Results from pilot studies normally cannot be used to support CMCs. The results

of Track D pilot studies are confidential to the CCQM unless public release is approved by the

unanimous consent of all participants.

There are two types of Track D pilot studies:

1. Stand-alone pilot studies. Studies are considered stand-alone when no KC is being conducted

at the same time and thus the study is unique regarding the measurand, matrix and objectives.

They address particular measurement areas or the application of specific techniques. Recent

examples are the determination of acrylamide in potato chips (CCQM-P109) or the

applicability of qNMR as a primary method for purity determinations (CCQM-P150).

2. Parallel pilot studies to Track A or Track C KCs. A parallel pilot study is a study in which the

same material is being simultaneously used in a KC. The possibility of a parallel pilot study

under Track D depends on the availability of materials in addition to those needed for the KC.

An NMI/DI that is developing services relevant to a particular KC may choose to participate

in a pilot study in parallel to a KC, to benchmark their performance using the KC reference

value from the parallel KC. An NMI/DI can also choose to participate in both a parallel pilot

study and an associated KC in order to assess the performance of their established methods in

the KC and of their capabilities in other methods under development in the pilot study.

PREPARING PROPOSALS FOR OAWG COMPARISONS

Scope of Proposals for Consideration

The scope and design of proposals for comparisons or pilot studies within the OAWG differs

depending upon the Track within which they are categorized:

Track A key comparisons cover the suite of model systems outlined in the current 10-year

OAWG strategy. They are aimed at comprehensively supporting the core measurement

competencies necessary to underpin OAWG-relevant CMCs. The specific measurands for

each comparison are proposed by an OAWG member and selected by the OAWG as being an

appropriate, practical material that aligns with the goals of the Track A strategy.

Track A CRM key comparisons can be proposed by an OAWG member on behalf of an

NMI/DI based on needs for comparisons of value assignments made to CRM and PT

materials.

Version 6.2 2018-11-30

Page 13 of 26

Track C key comparisons can be proposed by an OAWG member on behalf of an NMI/DI for

measurands representing areas of emerging global importance or constituting a specific

technical challenge.

Track D pilot studies can be proposed as standalone studies by an OAWG member on behalf

of an NMI/DI in areas where investigation of a technique or area is felt to be warranted. When

a KC is proposed OAWG members can also request that the coordinating laboratory provide

the option to participate in a parallel pilot study where this is feasible.

Mechanism for Suggesting OAWG Comparison or Studies

Suggestions for Track A Model 2 Key comparisons, Track C Key comparisons and Track D Pilot

studies should be initiated by communication to the Chair of the OAWG of a request for inclusion for

discussion on the agenda of a future OAWG meeting. If sufficient interest and willingness to

participate exists, and if it is agreed that the suggested comparison is consistent with the OAWG

strategy, a formal proposal may be prepared for submission to the OAWG.

Each suggestion should include a brief justification and information on:

the importance of the comparison, e.g. international requirements for trade or regulation in the

area; particular technical challenges; outline of risks if the OAWG does not conduct a

comparison in this area;

measurands, relevant mass fraction ranges and any uncertainty requirements if relevant;

the availability of CRMs and other sources of materials suitable for use as Primary Calibrators

for the individual analytes for the comparison, outlining how they meet the CIPM traceability

requirements.16

Technical Information Required for Proposed Protocols for Comparisons or Studies

Although key comparisons and pilot studies are the products of the entire OAWG, the responsibility

for drafting the protocol is assumed initially by the coordinating laboratory. Preliminary draft

protocols are submitted to the OAWG Chair for distribution to the working group. Reviewing and

commenting on the draft protocol is expected from all interested participants in the OAWG.

Draft protocols for proposed Track A and C key comparisons and Track D pilot studies should

include:

an Introduction including a clearly defined justification of the comparison, e.g. international

requirements for trade or regulation in this area; particular technical challenges;

for Track A KCs, a description of how the proposed measurand fits within the scope of the

OAWG strategy and why the particular model system was selected;

an outline of the potential consequences if the OAWG does not conduct comparisons in this

area in the next five years (applicable for Track A Model 2 studies, Track C and Track D);

description of the measurand(s) and study material(s), including an assessment of the

material(s) homogeneity and stability;

the number of samples to be provided and the mass or volume of material per sample;

16 See CIPM requirements: http://www.bipm.org/cc/CIPM/Allowed/98/CIPM2009_24_TRAC_MRA_REV_13_OCT_2009.pdf

Version 6.2 2018-11-30

Page 14 of 26

any restrictions for exportation or importation of samples that might be encountered, with

special sample handling and safety requirements (when known);

a detailed protocol for dry mass/moisture correction where relevant;

for all KCs a HFTLS statement using the OAWG prescribed format that defines the scope;

the proposed comparison timetable, availability of calibrants for the comparison and whether

these meet the CIPM traceability requirements,16 and any available information on potential

QC materials including matrix CRMs that may be available;

a projected approach to assigning the KCRV and its uncertainty (not required for Track D);

a draft core competency table for the comparison; core competencies potentially supported by

the study, and any that would be if particular measurement processes were employed;

a draft reporting form (including number of results etc.). The specific format of the reporting

form will depend on the procedures and methods employed.

OAWG Comparison Protocol Template

A Key comparison protocol template for Track A or Track C comparisons is available in the

OAWG General Documents area of the OAWG website.17 This outlines the suggested minimum

technical information for proposed protocols to ensure that critical areas are addressed. This

template may be used for Pilot studies; however, the requirements for such studies may not be as

rigorous. Additions to the template are not intended to be precluded by the format and whenever

important additional considerations arise, they can be added to the report.

A corresponding coordinating laboratory checklist is also available in the OAWG General

Documents area of the OAWG website. This checklist is provided to outline and summarize the

general roles and responsibilities of the Coordinating Laboratory for conducting a Key comparison

and/or Pilot study.

OAWG Process for Selecting Comparisons or Studies

Using the strategies described above, the OAWG develops a work program for comparison studies

based on their potential value to the OAWG in achieving the goals described in the OAWG Terms of

Reference and the current OAWG strategy document.13 Only in exceptional circumstances should the

number of comparisons undertaken exceed that planned in the strategy document.

Track A comparisons. Key comparison proposals aligned with the proposed suite of studies that

cover the Track A plan are considered by the OAWG. The primary focus is on the selection of

model systems that represent the measurement challenges outlined in the OAWG strategic plan and

potentially support a range of core competencies mapping the OAWG measurement space.

Track A comparisons are identified and agreed upon in order to be consistent with the requirements

to provide technical support for the NMI CMCs covered under the terms of reference of the OAWG.

The coordinating laboratory for each proposed KC is subject to approval of the OAWG Chair.

Track A Model 2 comparisons, Track C comparisons and Track D pilot studies. Comparisons and

studies proposed by individual NMI/DI or by groups of NMI/DIs are considered by the OAWG.

17 Open access area within OAWG: https://www.bipm.org/en/committees/cc/wg/oawg.html

Version 6.2 2018-11-30

Page 15 of 26

Considerations include the number of NMI/DIs indicating their willingness to participate, the extent

to which the proposed study can contribute to meeting the OAWG strategic goals, the impact on the

practical work load of the OAWG members, and whether the proposed studies might be more

appropriately conducted under the auspices of an RMO.

Track A Model 2 comparisons can be conducted with as few as two unique CRM and PT materials,

similar to a bilateral comparison. However, this category of comparisons is more commonly designed

to determine a Key Comparison Reference Function (KCRF), for which it is advised to have at least

five unique CRM and PT materials that should also span a reasonable mass fraction or concentration

range. For Track C comparisons that will result in a KCRV and Track D studies that will result in a

reference value, participation by a minimum of five laboratories is also recommended.

Decisions regarding proposals for key comparisons and pilot studies are documented in relevant

OAWG meeting minutes.

Approval of protocol. An individual laboratory normally volunteers to take on the coordination role

for a comparison. This is subject to agreement by the OAWG and approval by the OAWG Chair. If

necessary, the coordinating laboratory will revise the comparison protocol to reflect decisions made

by the OAWG. For example, these could include changes to the timetable, reporting form and draft

HFTLS statement. The OAWG Chair distributes the revised protocol to OAWG participants for

further review, comments and eventually final approval.

All proposed KCs must be presented by the OAWG chair to the CCQM annual plenary meeting for

formal approval, rejection or modification. Comparisons and studies approved for inclusion in the

OAWG work program must have a comparison number assigned by the Chair of the KCWG on

request from the coordinating laboratory. The coordinating laboratory completes the form “Request

form for CCQM Study Number (CCQM-F-02)” 18 available under the “CCQM forms” tab of the

“CCQM publications and forms” page of the CCQM website and forwards it to the OAWG Chair for

review and subsequent submission to the KCWG Chair. A unique identifier number is assigned by

the KCWG Chair and communicated to the coordinating laboratory and the OAWG Chair. The

identifier must comply with the nomenclature requirements described in Section 3.1 of the document

CIPM-D-05 “Measurement Comparisons in the CIPM MRA”.19

When the list of participants for the comparison/study has been finalized the coordinating laboratory

completes the form “Key and supplementary comparison (and pilot studies) - registration and

progress form”20 and forwards it to the OAWG Chair who then submits it to the KCDB office in order

to have the comparison formally registered in the KCDB.

18 Download form in CCQM forms sections from: http://www.bipm.org/en/committees/cc/ccqm/publications-cc.html 19 See CIPM guidance on comparisons: www.bipm.org/utils/common/documents/CIPM-MRA/CIPM-MRA-D-05.pdf 20 See CCQM forms sections (download and complete locally): http://www.bipm.org/en/committees/cc/ccqm/publications-cc.html

Version 6.2 2018-11-30

Page 16 of 26

GUIDANCE FOR PARTICIPANTS IN OAWG KEY COMPARISONS OR

PILOT STUDIES

Participation by an NMI/DI in a key comparison or pilot study under the auspices of the OAWG

involves the considerations and steps described below.

Eligibility for Participation

Participation in Key Comparisons organized by the CCQM is only open to laboratories that meet

the requirements of section 6.1 of the CIPM MRA. Section 4.1 of CIPM MRA-D-0519 provides

additional guidance. The laboratory must be listed in Appendix A of the CIPM MRA. When

choosing between participation in a key comparison or a parallel pilot study, an eligible NMI/DI

should also consider the requirements of the CCQM KCWG Guidelines on the Review of CCQM

CMCs21 if the NMI/DI intends their results for the comparison to be used to support CMCs covered

by the comparison HFTLS statement.

An expert laboratory not listed in Appendix A of the CIPM MRA may request or be invited to

participate in a Track D pilot study. Note that there are strict controls placed on “guest laboratory”

participation in CCQM pilot studies. The coordinating laboratory and the candidate guest laboratory

must complete the form CCQM-F-0122 requesting participation, forward it initially to the OAWG

Chair for comment and subsequently to the CCQM President for a final decision on approval to

participate. The CCQM Executive Secretary shall also be advised of the final decision. If study

materials are limited, preference is given to the participation of CCQM NMI/DIs.

Call for Participation

The OAWG Chair issues a call for participation to NMI/DIs registered on the official OAWG

mailing list via an email which includes a protocol with the draft HFTLS statement for the

comparison.

Required Response from NMI/DIs to the Call for Participation

NMI/DIs agreeing to participate in the comparison study reply to the study coordinator via email and

provide the following information:

name, contact details and email address of the person responsible for coordination of the

participation and result submission at the NMI/DI;

name, shipping address, email address, and phone number of the person to receive the samples

at the participating NMI/DI (if different from the person identified above);

any other documentary information that will facilitate shipment and transfer of the materials

through a country’s Customs and Border Authorities;

a statement or selection via the associated forms to participate in the KC or the parallel pilot

study (or both).

21 See https://www.bipm.org/wg/CCQM/KCWG/Allowed/Guidance_Documents/CCQM-KCWG_Guidance_document_for_chemistry_CMCs-

KCWG-2018.pdf 22 See CCQM forms section: http://www.bipm.org/en/committees/cc/ccqm/publications-cc.html

Version 6.2 2018-11-30

Page 17 of 26

Distribution of Comparison Samples for Measurement by Participants

The coordinator notifies the participants by email that samples are being dispatched on a certain date.

The carrier identity may be provided to facilitate the tracking of shipments. Requirements for the

storage of samples upon their receipt should be clearly described in the protocol and may be repeated

in documentation provided with the samples.

Distribution of Reporting Form for Use by Participants

The comparison coordinator emails to each participant an electronic copy of the comparison protocol

and the form to be used by the participant for reporting their results and other required information.

This should occur as part of the sample distribution process.

Contents of the Participant’s Submitted Report

Participant reports will include:

description of their measurement process

approach to calculation of their reported results and uncertainty

description of their uncertainty budget and its components

metrological traceability of calibrants used with information of any in-house assessment

carried out, the methods employed and the evidence for capability in these methods.

other pertinent information as requested by the study coordinator

Metrological Traceability Requirements of Comparison Calibrator Materials

Metrological traceability (as defined by the CIPM23) of the value assigned to the primary calibrant

used in a key comparison can only be established from the NMI/DI’s own primary realization or from

that of another recognized NMI/DI. The OAWG interpretation of this CIPM requirement means that

in both cases evidence for the competency required to provide value assignment of the primary

calibrant must be available (e.g., successful participation in a CCQM Track A purity assignment KC

using the techniques employed to value assign the primary calibrant). Thus, simply using a calibrant

from an NMI/DI is not sufficient. The evidence for compliance with traceability requirements will be

requested from each participant in a comparison for each primary calibrant they employ.

Metrological traceability cannot come directly from a pure material sourced from a provider that is

not an NMI/DI, even if produced as a CRM under ISO 17034 accreditation and cannot be realized by

a result obtained from a material value assigned through participation in a comparison or PT scheme.

An NMI/DI may carry out purity analysis on materials that do not meet the CIPM requirements, such

as commercial standards, in order to use them as calibrants if the quality of the material is fit-for-

purpose. When such an approach is used, the traceable assigned value is that obtained by the NMI/DI

and not as reported by the supplier of the material. The ability to carry out in-house purity assessment

needs to have been previously demonstrated in relevant OAWG comparisons/studies.

23 See CIPM requirements: http://www.bipm.org/cc/CIPM/Allowed/98/CIPM2009_24_TRAC_MRA_REV_13_OCT_2009.pdf

Version 6.2 2018-11-30

Page 18 of 26

Submission of Results to the Coordinating Laboratory

The comparison coordinator should remind all participants no later than two weeks prior of the

approach of the deadline for the submission of results. If a participant needs to request an extension

for the submission of results, they may send an email to the comparison coordinator with a

justification of why they have requested an extension. It is at the discretion of the comparison

coordinator and the OAWG Chair to grant an extension and decide on an appropriate duration for the

delayed submission of results.

Participants in a comparison submit their results electronically to the comparison or study coordinator

using the supplied form. Additional information may be included if judged to be of use to the

coordinator or for subsequent discussion in the OAWG.

If a key comparison participant uses multiple methods to measure analytes in a given key comparison,

the NMI/DI may report all of its results in its submission to the coordinating laboratory. However, a

participant is not allowed to submit more than one result for its key comparison “reported value” as

presented in the table of results, degree of equivalence table, or other plots. Only one key comparison

result is allowed per institute.

Distribution of Preliminary Results to Participants in Study

In order for results from a study to be discussed effectively at an OAWG meeting, preliminary results

must be distributed to participants by the study coordinator with a copy to the OAWG Chair prior to

the OAWG meeting. Ideally the result distribution should occur no later than ten days prior to the

meeting to allow for internal review and discussion. Failure by the study coordinator to distribute

results as specified above may lead to deferral of the discussion to the next OAWG meeting. The

document presenting these preliminary results should be watermarked with “CCQM

CONFIDENTIAL” diagonally across all pages.

Presentation and Discussion of Results

Overview presentation by the coordinating laboratory. The coordinating laboratory will describe

the preparation of the comparison materials, the results of its homogeneity and stability assessment

of the material and provide an overview of the study results. Typically, only limited statistical

analysis of the results is involved at this point in the discussion.

Participant presentations. A brief summary presentation may be requested by the coordinating

laboratory, in consultation with the OAWG Chair, from selected study participants to facilitate

discussion of significant variations in the reported values and to inform further discussion and

recommendations regarding the KCRV determination. In particular, participants who used different

approaches may be asked to present. Study participants who cannot attend the OAWG meeting may

be requested to provide summary presentations to the study coordinator prior to the meeting for

presentation on their behalf.

Participant’s presentations would normally be expected to focus on the approach used for:

analysis and supporting method validation studies,

establishing the metrological traceability of its measurements,

Version 6.2 2018-11-30

Page 19 of 26

measurement uncertainty calculation (including components considered but not included),

control of potential sources of anomalies, e.g. sample handling, method biases, etc.

Participants whose result may be a potential outlier can be asked to review their data for reasons for

the apparent discrepancy and to propose possible explanations for further investigation if necessary.

Data from KC and parallel pilot studies are presented together at this stage in the discussion

process. Copies of the individual presentations are posted in the meeting folder within the OAWG

restricted access of the CCQM website24.

OAWG discussion. Any identified anomalies and other metrological issues are to be discussed. If

these issues cannot be resolved with the information immediately available, participants may be

required to carry out further investigations and discussion may be continued at subsequent OAWG

meetings. Action items required to provide the information necessary to address the unresolved

issues are identified in the report of the meeting.

Process for Preparing the Draft A Reports

The coordinating laboratory will prepare a Draft A report for KCs and its parallel pilot or a draft

summary report for stand-alone pilot studies. This document is confidential to the OAWG and is not

to be distributed beyond the OAWG. All results from a parallel pilot may be included in the Draft A

report; these results will be removed and placed in a separate pilot summary report prior to distributing

the Draft B report.

The Draft A reports should include:

an Introduction, including a description of the measurand(s) and the scope of the comparison;

description of the preparation and characterization of the materials, including homogeneity

and stability assessments;

compilation of the results reported by each participating institution;

discussion of the metrological traceability of each result;

proposed methods of combination of the comparison results, and one or more candidate

estimates for the KCRV and its associated uncertainty;

where necessary, the agreed basis for the exclusion of results from individual participants;

a draft How Far the Light Shines (HFTLS) statement and core competency statements

The Draft A reports are distributed having the watermark “CCQM CONFIDENTIAL” diagonally

across all pages.

OAWG Comparison Report Template

A template for Track A or Track C Key comparison reports is available in the OAWG General

Documents area of the OAWG website.25 This outlines the suggested minimum content and format

of reports to ensure that critical areas are addressed. This template may be used for generating reports

24 CCQM/OAWG working documents (Restricted access area): http://www.bipm.org/wg/CCQM/OAWG/Restricted/welcome.jsp; user name

and password provided by the OAWG Chair 25 Open access area within OAWG: https://www.bipm.org/en/committees/cc/wg/oawg.html

Version 6.2 2018-11-30

Page 20 of 26

for Pilot studies; however, the requirements for such studies may not be as rigorous. Additions to the

template are not intended to be precluded by the format and whenever important additional

considerations arise, they can be added to the report.

Key Comparison Reference Value (KCRV) for Track A and C Comparisons

For each comparison the approach for calculating the KCRV needs to be agreed within the OAWG

in the course of discussions of the results for the particular comparison. CCQM guidance for

estimating a consensus KCRV is provided elsewhere.26 In summary, the following considerations are

essential for the assignment of the KCRV:

the estimator and its associated uncertainty must have broad scientific acceptance and its

application can be defended for the particular data structure of the comparison;

an assessment of results has been made by the OAWG with respect to the metrological

traceability of each result used for the calculation of the KCRV. Results that do not meet the

CIPM traceability requirements are to be excluded from use in the KCRV calculation;

a participant’s value may be excluded from the calculation of the KCRV after review of the

results for other reasons. If, after investigation and determination that there is a sound

metrological reason, the participant may be invited to withdraw their result(s) from the KCRV

calculation. However, it rests within the OAWG and not individual participants to decide

which results shall be accepted for use in the final calculation.

the identification of a data point simply as a statistical “outlier” is not acceptable as the sole

basis for its exclusion.

It will usually require at least one revision of a Draft A report before the KCRV is agreed by the

OAWG. Once the final version of the Draft A report including the proposed KCRV and DoE tables

is reviewed and approved by the participants the report is considered as Draft B. The comparison

coordinator cannot proceed to prepare a Draft B report until the KCRV has been agreed.

Key Comparison Reference Function (KCRF) for a Track A Model 2 Comparison

A KCRF is determined that describes the relationship between the assigned and the measured values

for the CRM and/or PT materials. A generalized linear regression model is the anticipated method of

data analysis for the assessment of the participant degree of equivalence for Track A CRM studies.

Several publications provide guidance for the assignment of a KCRF.27

Process for Draft B Reports of Key Comparisons

After circulation of the final Draft A report and agreement by the OAWG of the KCRV the

coordinating laboratory will prepare a Draft B report. The Draft B report is provided to the OAWG

26 CCQM Guidance note: Estimation of a consensus KCRV and associated Degrees of Equivalence, Version: 10

Date: 2013-04-12 27 Duewer et al, Demonstrating the comparability of certified reference materials, Anal. Bioanal. Chem (2009) 395:155-169; Duewer et al,

Experimental design and data evaluation considerations for comparisons of reference materials, Accred Qual Assur (2012) 17: 567-588; Camara

et al. An international assessment of the metrological equivalence of higher-order measurement services for creatinine in serum, Anal Bioanal

Chem (2012) 403:527-535; Toman et al, A Bayesian approach to the evaluation of comparisons of individually value-assigned reference

materials, Anal Bioanal Chem 2012;403:537-548.

Version 6.2 2018-11-30

Page 21 of 26

chair who will circulate it to the entire OAWG. Only the data from the KC participants is normally

reported in the Draft B report. A separate Draft Final report is prepared for the results of a parallel

pilot study. At this stage, individual KC participant results are no longer considered confidential and

can be used to support CMCs and for presentations and publications. However, the KCRV and the

associated DoE tables must be considered confidential until they are approved by the CCQM WG

chairs and the Final Report published in the KCDB Appendix B.

Graphs and Tables of Equivalence. Comparison results are presented in graphs and tables in the Draft

B report. It is recommended that individual results, xi, are graphed using bars for each data point

corresponding to the combined standard uncertainties, u(xi), reported for the result. The plot should

indicate the KCRV and the range plus or minus the standard uncertainty of the KCRV, u(KCRV).

That is, the plot should show every xi as a “dot” with error bars that span the interval xi - u(xi) to xi +

u(xi) and have horizontal lines at x = KCRV - u(KCRV), KCRV, and KCRV + u(KCRV). The formula

used to calculate u(KCRV) differ somewhat among the different KCRV estimators; these formulae

are detailed in Ref 26.

The preferred format of the plot of the DoE of each participant result, di, is to graph each di with bars

indicating the associated 95 % level of confidence expanded uncertainty, U95(di). That is, the plot

should show every di as a “dot” with error bars that span the interval di - U95(di) to di + U95(di). If the

confidence range associated with the DoE value includes the KCRV (i.e. crosses a horizontal line

corresponding to DoE = 0), the reported value is consistent with the KCRV within the stated

uncertainties with about 95 % confidence. The formula used to calculate the di differ somewhat among

the different KCRV estimators28.

The DoE plot should also include the title/name and CCQM number of the comparison and a

horizontal line at d = 0. When appropriate, a secondary Y-axis can be added along the right-edge of

the graph displaying a percent relative scale for the di ± U95(di); that is, the secondary axis exactly

span the interval 100×dmin/KCRV to 100×dmax/KCRV where dmin and dmax are the minimum and

maximum values of the primary Y-axis.

The DoE table should list the di and their associated U95(di). As an aid to interpretation, the

absolute value of the ratio of the di divided by the 95 % uncertainty, |di/U95(di)|, can also be listed.

When |di/U95(di)| ≤ 1 the participant result is consistent with the KCRV with about 95% confidence.

The core competency tables provided at the submission of their result by each participant are

included in the Draft B report. Any participant reporting a result that is not consistent with the

KCRV will have their core competency table annotated to that affect. Where possible the particular

influence that caused the deviation from the KCRV will be identified and a note included in the core

competency table.

28 CCQM Guidance note: Estimation of a consensus KCRV and associated Degrees of Equivalence, Version: 10

Date: 2013-04-12

Version 6.2 2018-11-30

Page 22 of 26

Process for Drafting, Reviewing and Approving Final Summary Reports for Key Comparisons

After review by the OAWG and incorporation of any editorial changes, the Draft B report will then

be posted to the Open Access OAWG area to assist members of the KCWG with ongoing CMC claim

reviews. At this point, the report content and its associated DoE tables will not be considered

confidential and thus any “Confidential” watermarks will need to be removed.

An identical copy of the Draft B report will simultaneously be circulated by the OAWG chair to the

other CCQM Working Group Chairs for their review and approval. Any comments from the CCQM

WG chairs will be addressed by the comparison coordinator and the OAWG chair. If there are any

major or unresolved issues the OAWG chair will raise these with the CCQM President for final

approval. Once all issues are resolved the comparison coordinator will complete the “Publication of

a Final Report in Metrologia’s Technical Supplement” form29, including publication title, contact

person name(s), abstract, key words, and a complete list of authors, their affiliations and addresses.

The completed form is to be sent to the OAWG chair. The OAWG Chair will then forward the form

to the KCDB office with the Word version of the Final Report and an Excel spreadsheet summary of

the results for posting in the KCDB. Once the Final Report has been posted in the KCDB, the tandem

copy of the Draft B report will be removed from the Open Access OAWG area. A PDF version of the

Final Report will also be published online as a Metrologia Technical Supplement.

Process for Drafting, Reviewing and Approving Final Summary Reports for Pilot Studies

Final reports for pilot studies may use the template for Track A and C key comparisons. For parallel

pilot studies the KCRV can be included as a reference value but DoE calculations are omitted. It is

expected the Final Report would include suitable discussion of the participant results. Final pilot

study reports will also be distributed bearing the watermark “CCQM CONFIDENTIAL” diagonally

across all pages. The participants are to be identified in the report and the report will remain

confidential to the CCQM at all times unless publication (for example as a Metrologia Technical

Supplement) is agreed to by all participants.

The Final Reports for pilot studies will be approved by the OAWG chair following review by the

study coordinator and all of the participants and then posted in the “Finalised Pilot Study Reports”

area of the OAWG website. In the first instance these reports are confidential to the OAWG.

With the approval of all participants, a PDF version of the Final Report may be published as a

Metrologia Technical Supplement.

29 Download form from Guidance on comparisons section (and complete locally) from: http://www.bipm.org/en/cipm-mra/cipm-mra-

documents/

Version 6.2 2018-11-30

Page 23 of 26

PUBLICATION OF MEASUREMENT RESULTS FROM OAWG

COMPARISON STUDIES

As required by the terms of the CIPM MRA and to comply with its commitment to transparency, the

CIPM makes the final reports of all KC studies openly available through the Key Comparison

Database Appendix B. Other publicly available publications are provided by the BIPM.30

Result summaries and Draft A reports of key comparisons and Draft Final reports of pilot studies are

for internal use only by OAWG members and authorized guest laboratory participants in pilot studies.

Restricted documents are maintained in the OAWG working documents (Members’ area), and are

password controlled. The password can be obtained from the Chair of the OAWG. All documents are

stored in folders based on the meeting they were discussed at. The final pilot study reports are stored

in the “Finalised Pilot Study Reports” folder of the OAWG members’ area.

Authorship of reports based on data produced in any OAWG comparisons or studies must adhere to

any CIPM and/or CCQM requirements, and must be agreed to by all participating NMI/DIs to ensure

the appropriateness of listing of authors from their institutions. The CIPM has a guidance document

(CIPM-MRA-G-04.pdf) on authorship31.

Regarding the publication of the Final Report of pilot studies:

the coordinating laboratory has first right to publish it as a Metrologia Technical Supplement.

if the coordinating laboratory does not exercise this right, one or more other participant(s) in

the pilot study may do so in another journal, provided that the coordinating laboratory agrees.

the order of authorship is based on the intellectual contributions to the publication or otherwise

as agreed among all authors.

all contributors to the study must be acknowledged in the publication.

the choice of journal is at the discretion of the publication’s authors.

attribution of results to an NMI/DI in a publication requires the unanimous consent of each

participant. This consent must be documented in writing to the OAWG Chair. If agreement is

not obtained from ALL participants, no attribution other than to the author can be made.

Approved KC Final Reports are published on the publicly accessible BIPM KCDB website32. An

approved abstract of the KC Final Report is published in Metrologia with a link to the full report on

the BIPM website33.

An NMI/DI can publish its own results once the Draft B Report has been prepared but cannot report

the KCRV prior to the study results being made publicly accessible on the KCDB.

Once KC data is publicly available on the KCDB website, the results may be included in publications

with all participants identified and without their prior approval. Content of an external publication

30 http://kcdb.bipm.org/ and http://www.bipm.org/en/metrologia/ 31 See guidance on comparisons: http://www.bipm.org/en/cipm-mra/cipm-mra-documents/ 32 http://kcdb.bipm.org/ 33 http://www.bipm.org/en/metrologia/

Version 6.2 2018-11-30

Page 24 of 26

related to a KC study by an individual NMI/DI should accurately represent its results and relate them

to the overarching KC results publicly available on the BIPM website.

New methods developed by an NMI/DI for delivering services to its customers may be published by

the contributing NMI. Acknowledgment should be made to the CCQM when the study samples used

during the development or that the impetus for the development originated from an OAWG study.

ROLES AND RESPONSIBILITIES OF OAWG PARTICIPANTS

Responsibilities of the Chair, in conjunction with the vice-Chair

The following are the responsibilities internal to the OAWG:

preside at meetings of the WG

draft meeting agendas for the WG

issue calls for participation and protocols for studies as agreed by the OAWG

distribute draft reports and documents for comment and review by OAWG

review all reports and liaise with coordinators to finalize each stage of reporting

establish processes/guidance documents for the WG as required

provide strategic direction for the WG

The following are the responsibilities external to the OAWG:

represent OAWG to the CCQM

present the report of the past year’s activities of the OAWG to the CCQM plenary meeting

liaise with other WG chairs on common areas of interest for potential joint activities

represent the OAWG at CCQM symposia or external events as required

represent the OAWG and provide input into the CCQM Strategic Planning WG

obtain the unique study identifiers for OAWG KCs and pilot studies from the KCWG Chair

distribute OAWG approved Key Comparison Final Reports to the Chairs of other CCQM

WGs for comment and/or approval

review all Key Comparison Final Reports from other CCQM WGs

submit OAWG Key Comparison Final Reports to the KCDB coordinator for publication in

the CIPM MRA Appendix B after approval from all WG Chairs and after the results have

been presented at the CCQM plenary meeting

review RMO proposals for KCs and supplemental studies within the areas defined by the

OAWG Terms of Reference prior to the conduct of any such comparison study

circulate to the OAWG for review RMO approved Final Reports for registered KCs and

supplemental studies within the areas defined by the OAWG Terms of Reference

Responsibilities of a Coordinating Laboratory

The coordinating laboratory is responsible for the preparation of the candidate comparison material

and carries out characterization studies sufficient to establish that the homogeneity and stability of

Version 6.2 2018-11-30

Page 25 of 26

the target measurand(s) in the candidate material in the form to be supplied to participants is

sufficient for the intended purpose and the anticipated timeline of the comparison.

Once the OAWG has reviewed and approved the characterization of the candidate material for use

in the comparison, the coordinating laboratory will:

draft a protocol for the comparison for review and finalization by the OAWG, ensuring all

items specified previously under Technical Information Required for Proposed Protocols

for Comparisons or Studies are adequately addressed therein;

subsequent to the circulation of the Call for Participation in the comparison to the OAWG

by the OAWG Chair, maintain a record of all participants registered to participate in the

comparison and parallel pilot study;

arrange the distribution of the comparison samples and reporting forms to all registered

participants in accordance with the comparison schedule;

compile the results provided by the participants and produce a preliminary summary of the

comparison performance;

chair all discussions of the comparison results at OAWG meetings;

present to an OAWG meeting an outline of the material preparation and characterization

process and a summary of the comparison results, identifying any issues arising from the

comparison for general consideration by the OAWG;

produce the initial Draft A Report from the comparison results and information provided by

participants in the study (a template for Track A or Track C reports is available34)

revise the initial Draft A report as required until the KCRV proposal is agreed by the OAWG

and produce the subsequent Draft B report of the comparison (for review and approval by the

CCQM WG Chairs) and the Final Report for publication in the KCDB Appendix B.

complete the “Publication of a Final Report in Metrologia’s Technical Supplement” form35

and submit to the OAWG chair

Responsibilities of a Participant Laboratory

OAWG members and NMI/DI nominated laboratories agree to participate in comparison studies to

underpin their claimed measurement capabilities by evidence of the equivalence of their measurement

results or their core competencies.

Key comparison and pilot study participants are expected to:

review and contribute to the proposed protocol for the comparison study;

agree to the draft How Far the Light Shines (HFTLS) statement that outlines the scope of the

KC, together with input from the entire WG;

indicate at the time of acceptance by the OAWG of a comparison study the intended nature of

participation in the study (KC or parallel pilot study);

designate at the time of registration the nature of the participation (KC or parallel pilot study)

34 Open access area within OAWG: https://www.bipm.org/en/committees/cc/wg/oawg.html 35 Download form (and complete locally) from: http://www.bipm.org/en/cipm-mra/cipm-mra-documents/

Version 6.2 2018-11-30

Page 26 of 26

provide measurement results to the coordinating laboratory according to the agreed upon

timetable, and in the format specified in the study protocol;

if requested by the comparison coordinator, present approaches, data obtained and uncertainty

estimation of their reported value at a designated OAWG meeting.

Responsibilities of Participants in the OAWG Meetings and Other Activities

It is expected that participants in the OAWG will contribute to the discussion and evaluation of

OAWG comparisons and to the overarching strategy of the WG through:

contributing to the review of the comparison results and uncertainty budgets presented by

individual participants, the suitability of the HFTLS statement of each KC under consideration

and the approach used for the assignment of KCRVs;

reviewing and commenting on draft reports produced by each coordinating laboratory;

providing input regarding the choice of suggested and/or proposed comparison studies with

respect to the value they potentially bring to the NMI/DI that the participant represents;

providing comment and feedback for technical issues and proposals raised by the OAWG and

to the drafting of OAWG documents, proposals and meeting minutes;

contributing to the OAWG strategic planning process and its ongoing implementation.

OAWG REPORTS AND DOCUMENTS

The OAWG archives its documents and presentations. Items to be posted are circulated in a relevant

electronic format (primarily .pdf) via the WG Chair. Documents related to OAWG activities produced

prior to 2004 are archived by year in the restricted access CCQM main site along with other WG

output files. Meeting documents produced since 2004 are archived as OAWG restricted-access

documents in folders organized by meeting date36

Items saved in individual meeting folders include:

Meeting agenda, logistical information and related items

Draft and Final minutes of the meeting

Copies of all presentations made at the meeting

Draft reports of KCs and pilot studies discussed at the meeting

Finalized pilot study summary reports are posted in a separate folder “Finalised Pilot Study Reports”.

The Final KC reports are posted in Appendix B of the KCDB37.

A yearly summary of recent OAWG activities is reported to the CCQM at its annual meeting. These

“Organic Analytical Working Group Report to the CCQM” presentations are included in the relevant

OAWG meeting folder, the OAWG General Documents folder (for reports since 2013) and in the

relevant CCQM meeting folder. Reports prior to 2004 can be obtained from the CCQM Secretariat.

36 CCQM/OAWG working documents (Restricted access area): http://www.bipm.org/wg/CCQM/OAWG/Restricted/welcome.jsp; user name

and password provided by the OAWG Chair 37 http://kcdb.bipm.org/