INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) The International Seed Testing Association GMO PT...

INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org

The International Seed Testing Association GMO PT Program

Design and Analysis

AIEC Fall MeetingOctober 15-16 2014

New Orleans, Louisana


Outline

• Reference material purity requirements

• Rating labs for ability to detect presence

• Rating labs for ability to quantify


Reference material purity requirements

Seed Lot Purity = 99%

+ +

- -+ -


Reference material purity requirements

• To make “sure” that the distributed samples contain the targeted GMO levels– GM seed is required to have purity greater than

99.25% with 95% confidence– Conventional seed is required to have impurity

less than 0.01% with 95% confidence


Binomial Distribution:

1. Success– With probability p

2. Failure– With probability q = 1-p

How Do I Do That Tim?

http://www1.istockphoto.com/file_thumbview_approve/2512836/2/istockphoto_2512836_coin_flip.jpg


Example 1: Probability of X heads out of 10 coin flips

How many head would you expect to see?

Would you be shocked if all 10 coins came up heads?


0 1 2 3 4 5 6 7 8 9 10

0.0

0.05

0.10

0.15

0.20

0.25

Number of Heads

Pro

babi

lity

Coin Tossing: binomial probability of X heads in 10 coin tosses


• If I tell you 1% of the beads are red how many would you red beads would you expect to see?

• Would you be surprised if you took a scoop and no red beads were observed?

Example 2: Probability of X red beads out of 300


0 1 2 3 4 5 6 7 8 9 10

Positive Seeds

0%

5%

10%

15%

20%

Pro

babi

lity


• I have another bag of beads• This bag could have any number of red beads• I let you take 1 and only 1 sample• In the sample you observe no red beads• If you had to say something about the purity

of the new bag what would you say?


Confidence Claim• If the bag had 1% red beads, a scoop of 300

would contain at least 1 red bead 95% of the time

• I took a scoop of 300 beads and observed 0 red

• Thus, I am 95% confident the percent red beads is less than 1%.


Using the same logic in our case• To demonstrate GM seed purity greater than

99.25% with 95% confidence requires 400 seeds tested with 0 negatives

• To demonstrate conventional seed impurity less than 0.01% with 95% confidence requires 30,000 seeds tested with 0 positives


Can we do this cheaper • The cost of testing reference material for PT

program was becoming too large• Needed to find a way to reduce cost and still

make “sure” that the distributed samples contain the targeted GMO levels


Father of Bayesian Methods

Reverend Thomas Bayes1702 –1761, London England

Nonconformist ministerin Tunbridge Kent, England

and a mathematician on the side


• The seed provider has already performed purity tests• By taking into account this information using

Bayesian statistics, the final assessment of seed purity performed by ISTA requires less testing efforts $$$$ savings

Sequential updates:• Prior updated using seed provider’s data• Then update the posterior

1 2 1 2

1 2

1 1 2 1

1 2 1

(chain rule)

( | , ) ( , | ) ( )

( , , ) ( )

( )

( ) ( | ) ( | , )

( | ) ( | , )

f y y f y y f

f y yf

f

f y f y f y y

f y f y y

Prior (given y1) Likelihood (given y1)

: proportionality sign; replaces a constant normalizer


• Results :

1 1 11 1 1 1 1( | ) [(1 ) (1 )] [1 (1 ) (1 )]y m yf y Updated prior :

1 1 11 1 1 1

1 1 1 1 1( | ) (1 )[1 (1 ) ] (1 ) 1 (1 )[1 (1 ) ] (1 )y n ym m m mf y

Updated prior :

• Formulas for individual testing (GM seeds):

• Formulas for group testing (conventional seeds):

2 2 22 1 2 2 2 2( | , ) [(1 ) (1 )] [1 (1 ) (1 )]y m yf y y Likelihood :

2 2 22 2 2 22

2 1 2 2 2 22

( | , ) (1 )[1 (1 ) ] (1 ) 1 (1 )[1 (1 ) ] (1 )y n ym m m mn

f y yy

Likelihood :

l1, l2: false negative rates; d1, d2: false positive rates; m1, m2: # of seeds; y1, y2: # of negative seeds observed

l1, l2: false neg. rates; d1, d2: false pos. rates; n1, n2: # of pools ; m1, m2: # of seeds/pool; y1, y2: # of neg. pools observed

1 2 1 2 1( | , ) ( | ) ( | , )f y y f y f y y

. Purity requirements fulfilled. $$$$ savings


Rating a laboratory in a given proficiency test

• We need to compute objectively, and transparently to the participants, criteria to rate laboratories for each proficiency test on:– Presence/absence – Quantification


Proficiency tests rating system

• A No problem has been detected in this test

• B There are small problems, but no specific look or action is suggested to the participant

• C Problems, ISTA indicates there might be things to consider by the laboratory to explain or

correct things

• BMP Below Minimum of Performance, ISTA indicates that the results were poor and the

laboratory need to find explanations and to improve/correct


One test rating

One test Score Value

Overall rating on 6 tests

Range on 6 tests

A 5 points A 28 – 30 points

B 4 points B 21 – 27 points

C 3 points C 16 – 20 points

BMP 0 points BMP below 16 points

each proficiency test and a run of 6 tests

Example 1:

5A rating and 1 BMP

5*5points + 1*0point =25 points

Overall rating is B

Example 2:

4 B =16 points, no overall BMP


Rating ability to detect presence

• True value is known• 2 types of error can occur:

– Laboratory report as negative a positive sample

– Laboratory report as positive a negative sample

• The number of mistakes is used as a basis for the rating computations


Presence/absenceRating system

Rate Percentage of misclassified samples

A 0% - 5%

B >5% - 10%

C >10% - 20%

BMP >20%

Example with 12 samples in a Test

3 mistakes 3/12=25%


Presence/absenceRating systems

Rate Percentage of misclassified samples

A 0 errors

B 1 error

C 2 errors

BMP More than 2 errors

Example with 12 samples in a Test


Confidence intervals are rather wide:

If 20 test and all correct 95% confident error rate less than 17%

Many more samples (over 300) would be necessary to estimate error rates with 1% precision

=> PT does not check error rates


Rating quantification

• True value is known, and/or can be estimated from all results received

• Too many sample results are too far from truth ->BMP• Average results by spiking level are not accurate ->C• Too many inaccurate sample results ->B• Otherwise rate A


Z-Scores• The rating system for quantification is based

on Z-scores• Before we get to far into the rating system

lets talk really quick about Z-scores


Example: Birth Weights are Normally Distributed


z-scores Consider a value x from a distribution with mean m and standard-deviation s. The formula for converting x into its corresponding z-score is:

x

z

Indicates how far and in what direction x deviates from m , in units of s

z-scores distribution’s mean = 0 z-scores distribution’s std-deviation = 1

When the distribution of reference is normal, the z-scores distribution is also normal and the probability to have a z-score in the interval [-2 ; +2] is ~ 0.95


More than ½ of the sample resultsoutside [½ true level; 2 x true level]

C Yes

Probability to observe amore extreme value of the sum

of absolute spiking levels z-scores whenassuming the laboratory provides

accurate resultsis 0.01

No

No

More than 1/6 (~17%) ofthe sample z-scores outside

[-2; +2] (missing sample values are

counted as outside)

BMP Yes

AB Yes No


mean of the sample results true levelz_score

Reference Intra Lab std _deviation / # of reported samples by spiking level

Consider the following z-scores computed for each Lab andeach spiking-level:

Probability to observe a moreextreme value of the sum of absolute

spiking levels z-scores when assuming thelaboratory provides accurate results

is 0.01

C Yes No

level by spiking provided samplesof#/estimatedeviation_stdLab_Intra

leveltrueresultssampletheofmeanscore_z

Consider the following z-scores computed for each Lab andeach spiking-level:

The theoretical distribution of the sum of these absolute independentz-scores for a given Lab can be defined using simulations.

C rating will characterize Labs that have problems in providing accuratemean results by spiking levels.

We can the define the rejection region at the 0.01 level:# of spiking levels in the

proficiency test

Rejection region at 0.01

level

1 2.552 3.973 5.254 6.435 7.55…

We can then define the rejection region at the 0.01 level:

# of spiking levels in the

proficiency test

Rejection region at 0.01

level

1 2.552 3.973 5.254 6.435 7.55…

abs(sum(z-scores))

De

nsity

0 2 4 6 8

0.0

0.1

0.2

0.3

3 spiking levels: if sum of absolute z-scores > 5.25 C rating

1%


More than 1/6 (~17%) ofthe sample z-scores outside

[-2; +2] (missing sample values are

counted as outside)

AB Yes No

sample result true levelz _ score

Reference Intra Lab std _deviation

Consider the following z-scores computed for each (Lab x spiking-level x sample) combination:

A z-score outside [-2 ; +2] has low probability (~ 0.05) to occur for a laboratory providing accurate sample results.

z-scores

De

nsity

-2 0 2

0.0

0.1

0.2

0.3

0.4


. For each Lab k and each spiking level i, compute the variance of the sample results: 2

ik

. Compute the mean of the variances for a given spiking level i (except the one identified with the Cochran’s test): 2

i

Estimate of the intra-Lab standard deviation

. Carry out the Cochran’s test at the 95% level to see if the Lab

with the highest variance has an outlying spread of replicates.

2i. The intra-Lab standard-deviation estimate is then:


Statistical evaluation of the accuracy of the ISTA GMO PT quantitative rating system

Recent PTs have smaller intra-laboratorystandard-deviations


Statistical evaluation of the accuracy of the ISTA GMO PT quantitative rating system

Leading to more marginal results for the PTs having smaller intra-laboratorystandard-deviations


Re

z-fe

scorrenc

sample

e intra

result - true l

-lab std-devie

evel

( )ation

• Estimated from the data• Characterize the precision of the results• The better the precision, the higher chance to get “bad” ratings!

Particularly true for late PTs:

Change implemented (starting PT18):If the relative reference intra-laboratory standard-deviation is lower than 25%, use 0.25 x true GM content as the reference intra-laboratory standard-deviation.


Conclusion• Reference purity requirements are checked

using Bayesian methods which utilized provider test results

• Labs ability to detect presence/absence rated based on the percent misclassified samples, not intended to estimate error rates

• Labs ability to quantify rated based on z- scores, as labs have gotten more precise some adjustments have needed to be made


Thanks to:• Jean-Louis Laffont• Kirk Remund


Thank you for your attention

www.seedtest.o

rg


Questions ?

INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) The International Seed Testing Association GMO PT...

Documents

Transcript of INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) The International Seed Testing Association GMO PT...