The power of replications in difference tests · The power of replications in difference tests Per...
Transcript of The power of replications in difference tests · The power of replications in difference tests Per...
![Page 1: The power of replications in difference tests · The power of replications in difference tests Per Bruun Brockhoff Dept. of Mathematics and Physics The Royal Veterinary and Agricultural](https://reader030.fdocuments.us/reader030/viewer/2022040701/5d596aab88c9937b108bdb64/html5/thumbnails/1.jpg)
The power of replications indifference tests
Per Bruun Brockhoff
Dept. of Mathematics and Physics
The Royal Veterinary and Agricultural University
Thorvaldsensvej 40, DK-1871 Frederiksberg C,Denmark
![Page 2: The power of replications in difference tests · The power of replications in difference tests Per Bruun Brockhoff Dept. of Mathematics and Physics The Royal Veterinary and Agricultural](https://reader030.fdocuments.us/reader030/viewer/2022040701/5d596aab88c9937b108bdb64/html5/thumbnails/2.jpg)
Result
A few reps á day makes the lowpower go away!
![Page 3: The power of replications in difference tests · The power of replications in difference tests Per Bruun Brockhoff Dept. of Mathematics and Physics The Royal Veterinary and Agricultural](https://reader030.fdocuments.us/reader030/viewer/2022040701/5d596aab88c9937b108bdb64/html5/thumbnails/3.jpg)
Settting
w Panel size: n
w Each assessor perform k difference(triangle)test
w The N=nk binomial test is OK(Meyners & Kunert, 1999)
w The power of this test is unknown!
![Page 4: The power of replications in difference tests · The power of replications in difference tests Per Bruun Brockhoff Dept. of Mathematics and Physics The Royal Veterinary and Agricultural](https://reader030.fdocuments.us/reader030/viewer/2022040701/5d596aab88c9937b108bdb64/html5/thumbnails/4.jpg)
Outline
w Introduce the power of the usual binomialtest with replications
w Present and compare different statisticalmodels for replications
w Calculate and compare the power withinthese models
w Give limits for this power
![Page 5: The power of replications in difference tests · The power of replications in difference tests Per Bruun Brockhoff Dept. of Mathematics and Physics The Royal Veterinary and Agricultural](https://reader030.fdocuments.us/reader030/viewer/2022040701/5d596aab88c9937b108bdb64/html5/thumbnails/5.jpg)
The power of the binomial test
w The probability of claiming a differencewhen in fact it is there:n P(X xcritical), where
n X=total number of correct answers
n and assuming some ”alternative situation”:
n some (at least one) on the individual pis arelarger than 1/3.
![Page 6: The power of replications in difference tests · The power of replications in difference tests Per Bruun Brockhoff Dept. of Mathematics and Physics The Royal Veterinary and Agricultural](https://reader030.fdocuments.us/reader030/viewer/2022040701/5d596aab88c9937b108bdb64/html5/thumbnails/6.jpg)
Model types for the alternative
w Beta-binomial (Ennis & Bi, 1998)
w Generalized Linear Mixed Models(Brockhoff, 1997, Hunter et al., 2000)
w Binomial mixture models (Meyners &Kunert, 1999)
![Page 7: The power of replications in difference tests · The power of replications in difference tests Per Bruun Brockhoff Dept. of Mathematics and Physics The Royal Veterinary and Agricultural](https://reader030.fdocuments.us/reader030/viewer/2022040701/5d596aab88c9937b108bdb64/html5/thumbnails/7.jpg)
The Beta-binomial model:
For each assessor:
The individual probabilities are randomly distributed:
![Page 8: The power of replications in difference tests · The power of replications in difference tests Per Bruun Brockhoff Dept. of Mathematics and Physics The Royal Veterinary and Agricultural](https://reader030.fdocuments.us/reader030/viewer/2022040701/5d596aab88c9937b108bdb64/html5/thumbnails/8.jpg)
0.0 0.2 0.4 0.6 0.8 1.0
01
23
45
Beta
Data: n=24, k=12
![Page 9: The power of replications in difference tests · The power of replications in difference tests Per Bruun Brockhoff Dept. of Mathematics and Physics The Royal Veterinary and Agricultural](https://reader030.fdocuments.us/reader030/viewer/2022040701/5d596aab88c9937b108bdb64/html5/thumbnails/9.jpg)
The Generalized Linear MixedModel (GLMM)
For each assessor:
The individual probabilities arerandomly distributed:
![Page 10: The power of replications in difference tests · The power of replications in difference tests Per Bruun Brockhoff Dept. of Mathematics and Physics The Royal Veterinary and Agricultural](https://reader030.fdocuments.us/reader030/viewer/2022040701/5d596aab88c9937b108bdb64/html5/thumbnails/10.jpg)
0.0 0.2 0.4 0.6 0.8 1.0
01
23
45
Beta
Data: n=24, k=12
GLMM
![Page 11: The power of replications in difference tests · The power of replications in difference tests Per Bruun Brockhoff Dept. of Mathematics and Physics The Royal Veterinary and Agricultural](https://reader030.fdocuments.us/reader030/viewer/2022040701/5d596aab88c9937b108bdb64/html5/thumbnails/11.jpg)
The binomial mixture model
For each assessor:
The individual probabilities are randomly distributed:
with probability 1-"
with probability "
![Page 12: The power of replications in difference tests · The power of replications in difference tests Per Bruun Brockhoff Dept. of Mathematics and Physics The Royal Veterinary and Agricultural](https://reader030.fdocuments.us/reader030/viewer/2022040701/5d596aab88c9937b108bdb64/html5/thumbnails/12.jpg)
0.0 0.2 0.4 0.6 0.8 1.0
01
23
45
Beta
Data: n=24, k=12
GLMM
![Page 13: The power of replications in difference tests · The power of replications in difference tests Per Bruun Brockhoff Dept. of Mathematics and Physics The Royal Veterinary and Agricultural](https://reader030.fdocuments.us/reader030/viewer/2022040701/5d596aab88c9937b108bdb64/html5/thumbnails/13.jpg)
The Corrected Beta-binomialmodel:
For each assessor:
The individual probabilities are randomly distributed:
![Page 14: The power of replications in difference tests · The power of replications in difference tests Per Bruun Brockhoff Dept. of Mathematics and Physics The Royal Veterinary and Agricultural](https://reader030.fdocuments.us/reader030/viewer/2022040701/5d596aab88c9937b108bdb64/html5/thumbnails/14.jpg)
0.0 0.2 0.4 0.6 0.8 1.0
01
23
45
Beta
Data: n=24, k=12
GLMM
Cbeta
![Page 15: The power of replications in difference tests · The power of replications in difference tests Per Bruun Brockhoff Dept. of Mathematics and Physics The Royal Veterinary and Agricultural](https://reader030.fdocuments.us/reader030/viewer/2022040701/5d596aab88c9937b108bdb64/html5/thumbnails/15.jpg)
The Corrected Generalized Lin-ear Mixed Model (CGLMM)
For each assessor:
The individual probabilities arerandomly distributed:
![Page 16: The power of replications in difference tests · The power of replications in difference tests Per Bruun Brockhoff Dept. of Mathematics and Physics The Royal Veterinary and Agricultural](https://reader030.fdocuments.us/reader030/viewer/2022040701/5d596aab88c9937b108bdb64/html5/thumbnails/16.jpg)
0.0 0.2 0.4 0.6 0.8 1.0
01
23
45
Beta
Data: n=24, k=12
GLMM
Cbeta
CGLMM
![Page 17: The power of replications in difference tests · The power of replications in difference tests Per Bruun Brockhoff Dept. of Mathematics and Physics The Royal Veterinary and Agricultural](https://reader030.fdocuments.us/reader030/viewer/2022040701/5d596aab88c9937b108bdb64/html5/thumbnails/17.jpg)
Model comparison
0.0 0.2 0.4 0.6 0.8 1.0
02
46
8
GLMM
Beta
CGLMM
Cbeta
![Page 18: The power of replications in difference tests · The power of replications in difference tests Per Bruun Brockhoff Dept. of Mathematics and Physics The Royal Veterinary and Agricultural](https://reader030.fdocuments.us/reader030/viewer/2022040701/5d596aab88c9937b108bdb64/html5/thumbnails/18.jpg)
Data: n=23, k=12
0.0 0.2 0.4 0.6 0.8 1.0
01
23
45
6
0.0 0.2 0.4 0.6 0.8 1.0
01
23
4
GLMM
Beta
CGLMM
Cbeta
![Page 19: The power of replications in difference tests · The power of replications in difference tests Per Bruun Brockhoff Dept. of Mathematics and Physics The Royal Veterinary and Agricultural](https://reader030.fdocuments.us/reader030/viewer/2022040701/5d596aab88c9937b108bdb64/html5/thumbnails/19.jpg)
0.0 0.2 0.4 0.6 0.8 1.0
01
23
4
GLMM
Beta
CGLMM
Cbeta
Model comparison
![Page 20: The power of replications in difference tests · The power of replications in difference tests Per Bruun Brockhoff Dept. of Mathematics and Physics The Royal Veterinary and Agricultural](https://reader030.fdocuments.us/reader030/viewer/2022040701/5d596aab88c9937b108bdb64/html5/thumbnails/20.jpg)
Data: n=6, k=100
0.0 0.2 0.4 0.6 0.8 1.0
01
23
4
CGLMM
Cbeta
Beta
GLMM
![Page 21: The power of replications in difference tests · The power of replications in difference tests Per Bruun Brockhoff Dept. of Mathematics and Physics The Royal Veterinary and Agricultural](https://reader030.fdocuments.us/reader030/viewer/2022040701/5d596aab88c9937b108bdb64/html5/thumbnails/21.jpg)
Model comparison
0.0 0.2 0.4 0.6 0.8 1.0
01
23
4
CGLMM
Cbeta
Beta
GLMM
![Page 22: The power of replications in difference tests · The power of replications in difference tests Per Bruun Brockhoff Dept. of Mathematics and Physics The Royal Veterinary and Agricultural](https://reader030.fdocuments.us/reader030/viewer/2022040701/5d596aab88c9937b108bdb64/html5/thumbnails/22.jpg)
Calculation of power
w Calculation of P(X xcritical)
w Done by Monte Carlo methods:n Simulate X (total number of correct answers)
n Count how often X is larger than or equal to thecritical value
w Easy if software has built in functions:normal, beta, binomial. (e.g. Splus/R/SAS)
![Page 23: The power of replications in difference tests · The power of replications in difference tests Per Bruun Brockhoff Dept. of Mathematics and Physics The Royal Veterinary and Agricultural](https://reader030.fdocuments.us/reader030/viewer/2022040701/5d596aab88c9937b108bdb64/html5/thumbnails/23.jpg)
Power, n=12, k=4
Mixture CGLMM CBeta
![Page 24: The power of replications in difference tests · The power of replications in difference tests Per Bruun Brockhoff Dept. of Mathematics and Physics The Royal Veterinary and Agricultural](https://reader030.fdocuments.us/reader030/viewer/2022040701/5d596aab88c9937b108bdb64/html5/thumbnails/24.jpg)
0.0 0.2 0.4 0.6 0.8 1.0
01
23
4
CGLMM
Cbeta
Beta
GLMM
Extreme data: n=6, k=100
![Page 25: The power of replications in difference tests · The power of replications in difference tests Per Bruun Brockhoff Dept. of Mathematics and Physics The Royal Veterinary and Agricultural](https://reader030.fdocuments.us/reader030/viewer/2022040701/5d596aab88c9937b108bdb64/html5/thumbnails/25.jpg)
The common limit model
w All 3 models converge to the samelimit/extreme situation:
w All individuals are either 100%discriminators or non-discriminators(guessers)
![Page 26: The power of replications in difference tests · The power of replications in difference tests Per Bruun Brockhoff Dept. of Mathematics and Physics The Royal Veterinary and Agricultural](https://reader030.fdocuments.us/reader030/viewer/2022040701/5d596aab88c9937b108bdb64/html5/thumbnails/26.jpg)
The common limit model
For each assessor:
The individual probabilities are randomly distributed:
![Page 27: The power of replications in difference tests · The power of replications in difference tests Per Bruun Brockhoff Dept. of Mathematics and Physics The Royal Veterinary and Agricultural](https://reader030.fdocuments.us/reader030/viewer/2022040701/5d596aab88c9937b108bdb64/html5/thumbnails/27.jpg)
Monte Carlo for the commonlimit model
w Fix the effect sizew Simulate n binomial(1, ")sw For each outcome of 1 set xi =kw For each outcome of 0 simulate a
binomial(k,1/3)w Count the number of times X becomes as
large as the critical value.
![Page 28: The power of replications in difference tests · The power of replications in difference tests Per Bruun Brockhoff Dept. of Mathematics and Physics The Royal Veterinary and Agricultural](https://reader030.fdocuments.us/reader030/viewer/2022040701/5d596aab88c9937b108bdb64/html5/thumbnails/28.jpg)
Limits of power for triangle test
97%48
88%
92%
38
39
91%90%81%70%40%12
.
k=5k=4k=3k=2k=1n:Level 5%, 37.5% effect
![Page 29: The power of replications in difference tests · The power of replications in difference tests Per Bruun Brockhoff Dept. of Mathematics and Physics The Royal Veterinary and Agricultural](https://reader030.fdocuments.us/reader030/viewer/2022040701/5d596aab88c9937b108bdb64/html5/thumbnails/29.jpg)
Summary
w There is no big difference between the different”complicated models” to handle replications
w The loss of power by substituting assessors byreplications is remarkable small
w Given the panel size, a few replications increase thepower considerably
w Tables of limit power is given for some situations –a simple Monte Carlo method for other.
![Page 30: The power of replications in difference tests · The power of replications in difference tests Per Bruun Brockhoff Dept. of Mathematics and Physics The Royal Veterinary and Agricultural](https://reader030.fdocuments.us/reader030/viewer/2022040701/5d596aab88c9937b108bdb64/html5/thumbnails/30.jpg)
Some additional insights
w Computationally, the mixture model is theeasiest to handle:n The EM-algorithm easy to implement AND
gives the option of ”fuzzy clustering” of theindividual assessors
w More powerfull test than the independentbinomial exist!