ISTA Proficiency Test Programme · 2005-09-19 · Example: Test round 04-1, Brassica napus Species...
Transcript of ISTA Proficiency Test Programme · 2005-09-19 · Example: Test round 04-1, Brassica napus Species...
ISTA Proficiency Test Programme
StatisticStatistic SeminarSeminarHohenheim, 29.08. – 02.09. 2005
- Martina Rösch -ISTA System Auditor
What is Proficiency Testing?
Interlaboratory ring test with several laboratoriesto assess their technical competence
Testing of samples in accordance with determined methods
Statistical evaluation and comparison of a laboratory‘s results with theresults of all participants
Evaluation of the laboratory‘s testing performance
Detection of systematic errors in the laboratory‘s testing system
Detection of trends in testing performance
Part of the continual improvement process!
The ISTA Proficiency Test Programme -Design
Six crop groups
Representative species tested
Three proficiency test rounds per year with three samples
Germination, Purity, Other Seed Determination, Moisture Content, Tetrazolium
Mandatory for accredited laboratories depending on the individualscope of accreditation
Voluntary for ordinary member laboratories
Accreditationmaintained?
The ISTA Proficiency Test Programme -Procedure
The ISTA Proficiency Test Programme –Reporting
The ISTA Proficiency Test Programme –Computation of Germination Test Results
PrinciplesComputation of results follow the Tattersfield method described in SST, 1979
Calculation is based on the assumption that the germination test resultsfollow a Normal distribution if values are between 2 and 98%
Only results of accredited laboratories are used for determining the meanand standard deviation – All laboratories‘ results are checked againstthese values
The mean percentage of the four replicates is determined to two decimalplaces
Step 11. First estimate of the mean of the basic distribution by finding the
median (-> less disturbed by strangers)
The ISTA Proficiency Test Programme –Computation of Germination Test Results
0
2
4
6
8
10
12
85.5 86 86 .5 87 87.5 88 88 .5 89 89 .5 90 90 .5 91 91.5 92 92 .5 93 93 .5 94 94 .5 95 95.5 96 96 .5 97 97.5 98 98 .5 99 99 .5 100
The ISTA Proficiency Test Programme –Computation of Germination Test Results
Median = 93.5
Step 11. First estimate of the mean of the basic distribution by finding the
median (-> less disturbed by strangers)
The ISTA Proficiency Test Programme –Computation of Germination Test Results
2. First estimate of the variance (s2) by calculating the mean squaredeviation of those values which are above the median
0
2
4
6
8
10
12
85.5 86 86 .5 87 87.5 88 88 .5 89 89 .5 90 90 .5 91 91.5 92 92 .5 93 93 .5 94 94 .5 95 95.5 96 96 .5 97 97.5 98 98 .5 99 99 .5 100
The ISTA Proficiency Test Programme –Computation of Germination Test Results
Median = 93.5
t-value*standard error = 4.75
Values to calculate the 1st estimate of the variance (s2)
Step 11. First estimate of the mean of the basic distribution by finding the
median (-> less disturbed by strangers)
2. First estimate of the variance (s2) by calculating the mean squaredeviation of those values which are above the median
The ISTA Proficiency Test Programme –Computation of Germination Test Results
3. Discarding of outliers ((Median - t-value*standard error) < xi< (Median + t*s))
0
2
4
6
8
10
12
85.5 86 86.5 87 87.5 88 88.5 89 89.5 90 90.5 91 91.5 92 92.5 93 93.5 94 94.5 95 95.5 96 96.5 97 97.5 98 98.5 99 99.5 100
The ISTA Proficiency Test Programme –Computation of Germination Test Results
Median = 93.5
t-value*standard error = 4.75
93.5 - 4.75 = 88.75 93.5 + 4.75 = 98.25
Outliers
Values to calculate the 1st estimate of the variance (s2)
Step 11. First estimate of the mean of the basic distribution by finding the
median (-> less disturbed by strangers)
2. First estimate of the variance (s2) by calculating the mean squaredeviation of those values which are above the median
3. Discarding of outliers ((Median - t-value*standard error) < xi< (Median + t*s))
The ISTA Proficiency Test Programme –Computation of Germination Test Results
Step 21. Determination of the (arithmetic) mean (X) and the mean square
deviation (σ) from all remaining values
0
2
4
6
8
10
12
85.5 86 86.5 87 87.5 88 88.5 89 89.5 90 90.5 91 91.5 92 92.5 93 93.5 94 94.5 95 95.5 96 96.5 97 97.5 98 98.5 99 99.5 100
The ISTA Proficiency Test Programme –Computation of Germination Test Results
Median = 93.5
t-value*standard error = 4.75
93.5 - 4.75 = 88.75 93.5 + 4.75 = 98.25
Outliers
Determination of the mean (X) and standard error (σ)
X = 93.3; σ = 1.87
The ISTA Proficiency Test Programme –Computation of Germination Test Results
2. Calculation of Z-Scores: Zi=(xi-X)/σ -> deviation of the individual result from the mean of the distribution, divided by the standard error
Step 11. First estimate of the mean of the basic distribution by finding the
median (-> less disturbed by strangers)
2. First estimate of the variance (s2) by calculating the mean squaredeviation of those values which are above the median
3. Discarding of outliers ((Median - t-value*standard error) < xi< (Median + t*s))
Step 21. Determination of the (arithmetic) mean (X) and the mean square
deviation (σ) from all remaining values
0
2
4
6
8
10
12
85.5 86 86.5 87 87.5 88 88.5 89 89.5 90 90.5 91 91.5 92 92.5 93 93.5 94 94.5 95 95.5 96 96.5 97 97.5 98 98.5 99 99.5 100
The ISTA Proficiency Test Programme –Computation of Germination Test Results
Median = 93.5
t-value*standard error = 4.75
93.5 - 4.75 = 88.75 93.5 + 4.75 = 98.25
Outliers
Determination of the mean (X) and standard error (σ)
X = 93.3; σ = 1.87Zi=(xi-X)/σ
(90.5 – 93.3)/1.87 = -1.50
EXAMPLEEXAMPLE
-3 -2.5 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2 2.5 3
95%
2.5%2.5%
5% of the results wouldfall outside ±2.0
The ISTA Proficiency Test Programme –Computation of Germination Test Results
Z-Score = -1.5
-2.23
-1.12
Results outside ±2.0 areconsidered as due not to random events butanalytical problems
In order to determine thelaboratory‘s testingcapabilities, the total sumof absolute Z-scores isdetermined
Total Sum of Z-Scores = 4.85
Problem when results are above 98 or below 2% -> Poissondistribution rather than Normal distribution
Selection of seed lots which have a germination capacity below98%
Calculating the arithmetic mean without discarding outliers whenmedian is above 98 or below 2%
The ISTA Proficiency Test Programme –Computation of Germination Test Results
A, B, C and BMP (Below Minimum Performance) for in-round and overallrating
Overall rating will be determined after 6 rounds of participation
Only results of mandatory test rounds are included in the overall rating
The ISTA Proficiency Test Programme –Rating System
Absolute sum of Z-Scores of normal seedlings is determined; thresholdsdefine the in-round rating
The ISTA Proficiency Test Programme –In-Rating System for GER, PUR, MOI, TZ
Score Sum of absolute Z-Scores
A Sum of absolute Z-Scores ≤ 3.5
B 3.5 < Sum of absolute Z-Scores ≤ 5.3
C 5.3 < Sum of absolute Z-Scores ≤ 7.0
BMP Sum of absolute Z-Scores ≥ 7.0
Example: Germination
Example: Germination
Example: Germination
Example: Germination
The ISTA Proficiency Test Programme –Evaluation of Other Seed Determination
PrincipleSeeds of known number and identity are added to the proficiency test sample which has not been purified beforehand
Laboratories are expected to retrieve and identify all seeds at least to genus level
EvaluationOption 1: determine the retrieval and identification rate and determine the
in-round rating based on thresholds (90/80/70)
Option 2: determine the retrieveal and identification rate based on theactual retrieval rate of the added species. Thresholds are then appliedaccordingly.
The ISTA Proficiency Test Programme –Evaluation of Other Seed Determination
Option 1: DifficultiesNumber of seeds per species is different;
Degree of difficulty to retrieve and identify a particular species isdifferent depending on the species to be tested (Sinapis arvensis in Brassica napus versus Sinapis arvensis in Triticum aestivum);
No distinction between retrieval and identification;
Total number of seeds added affects the impact of each seed missed on the percentage of retrieved and identified seeds.
The ISTA Proficiency Test Programme –Evaluation of Other Seed Determination
Option 2: PrinciplesEach species that was added is assigned a value based on the actualretrieval rate of all seeds added.
Total retrievalrate [%]
Assignedvalue
≥90 3
≥85 2
≥80 1
Number of seeds found by a participant is multiplied by the assignedvalue
The ISTA Proficiency Test Programme –Evaluation of Other Seed Determination
Example: Test round 04-1, Brassica napusSpecies Total retrieval
rate [%]Score
Galeopsis tetrahit 88
93
Sinapis alba 83 2
Sinapis arvensis 61 1
Lotus corniculatus 72 1
Raphanus sativus 81 2
Chenopodium album 84 2
Polygonum aviculare 91 3
79
2
Galium aparine 3
Thlaspi arvense 1
The ISTA Proficiency Test Programme –Evaluation of Other Seed Determination
Example laboratory resultsSpecies # of seeds
addedScore # of seeds x
score# of seeds found
2 2
3
2
2
3
1
1
2
1
3
24
9
8
6
9
3
4
8
2
53
Sinapis alba 4
3
4
3
3
0
0
4
2
21
75
8
Sinapis arvensis 3 0
Thlaspi arvense 2 2
C
Total sum 28 46
Percentage 87
Lotus corniculatus 4 0
Raphanus sativus 4 8
Chenopodium album 3 6
Polygonum aviculare 3 9
Seeds found x score
Galeopsis tetrahit 4
Galium aparine 9
In-round rating B
The ISTA Proficiency Test Programme –Evaluation of Other Seed Determination
Example laboratory resultsSpecies # of seeds
addedScore # of seeds x
score# of seeds found
Galeopsis tetrahit 2 2 4 2 4
Galium aparine 3 3 9 0 0
Sinapis alba 4 2 8 0 0
Chenopodium album 3 2 6 3 6
Polygonum aviculare 3 3 9 3 9
Sinapis arvensis 3 1 3 3 3
Lotus corniculatus 4 1 4 4 4
Raphanus sativus 4 2 8 4 8
Thlaspi arvense 2 1 2 2 2
Total sum 28 53 21 36
Percentage 75 68
In-round rating C BMP
Seeds found x score
Score Number given
A 5
B 4
C 3
BMP 0
Overall Score Threshold
A 28-30
B 21-28
C 16-20
BMP below 16
The same rating system is used for germination, purity, other seeddetermination, moisture content determination and tetrazolium
The ISTA Proficiency Test Programme –Overall Rating
02-3
03-1
03-2
03-3
04-1
04-2
A
A
B
BMP
B
C
ACBBMPBB=
In-round rating Accross 6 rounds
A
The ISTA Proficiency Test Programme –Overall Rating
BMP
COverall
Overall
Warning
Suspension
The ISTA Proficiency Test Programme –Consequences
Example: Germination
Confirmation of good performance
Indication of potential deficiencies in testing procedures
Defective equipment
Level of knowledge of staff
Insufficient documented procedures
Testing material
The ISTA Proficiency Test Programme –Impact
Receipt of testreport
Evaluation of testresults
Results o.k.? Stop
Root causeanalysis
Corrective action
Evaluation ofeffectiveness ofcorrective action
Yes
No
Stop
• Are deficiencies systematic?
• Are trends evident?
• Calculation or transcription error?
• Were all items of equipment appropriately maintained and monitoredat the time of testing (Maintenance and monitoring records, etc.)?
• Is staff trained for the particular type of test, have all proceduresbeen adhered to?
• How are the results of repeat tests?
• Was all testing material appropriate?
• Adjusting maintenance of equipment
• Training of staff
• Update and/or amend test procedures
• Carry out ring tests
The ISTA Proficiency Test Programme –Corrective action
The ISTA Proficiency Test Programme –Evaluation of Purity Test Results
Evaluation system for purity results not yet determined due to thefollowing restrictions:
Values for pure seed are close to 100%, inert matter and otherseed are below 2%
Distribution is not Normal
Number of results per test round is small
Thank you for your attention!