L2-1 Assessment of Model Performance - Distance · Assessment of model performance • Likelihood...
Transcript of L2-1 Assessment of Model Performance - Distance · Assessment of model performance • Likelihood...
![Page 1: L2-1 Assessment of Model Performance - Distance · Assessment of model performance • Likelihood • AIC • Absolute measures of model fit Chi-squared test Qq plots Kolmogorov-Smirnov](https://reader034.fdocuments.us/reader034/viewer/2022051810/60199f0b49e1c131e435033f/html5/thumbnails/1.jpg)
Assessment of modelperformance
• Likelihood
• AIC
• Absolute measures of model fitChi-squared testQq plotsKolmogorov-Smirnov and Cramér-von Mises tests
![Page 2: L2-1 Assessment of Model Performance - Distance · Assessment of model performance • Likelihood • AIC • Absolute measures of model fit Chi-squared test Qq plots Kolmogorov-Smirnov](https://reader034.fdocuments.us/reader034/viewer/2022051810/60199f0b49e1c131e435033f/html5/thumbnails/2.jpg)
Likelihood
xi = distance of ith detected animal from the line.
)(...)()( 21 ni
n
1=i
xfxfxf)xf(=L
f(x) = probability density function of x
f(x) dx = Pr (animal was between x and x+dx from the line, given it was detected between 0and w) for small dx
When distances are exact, the likelihood is given by
We fit f(x) by finding the values for the parameters of f(x) (or equivalently g(x)) thatmaximize L (or loge(L) ).
![Page 3: L2-1 Assessment of Model Performance - Distance · Assessment of model performance • Likelihood • AIC • Absolute measures of model fit Chi-squared test Qq plots Kolmogorov-Smirnov](https://reader034.fdocuments.us/reader034/viewer/2022051810/60199f0b49e1c131e435033f/html5/thumbnails/3.jpg)
Akaike’s Information Criterion
AIC = -2loge(L) + 2q
L is the maximized likelihood (evaluated at the maximum likelihood estimates of themodel parameters)
and q is the number of parameters in the model.
• Models need not be special cases of one another
• Select the model with smallest AIC
• Gives a relative measure of fit
![Page 4: L2-1 Assessment of Model Performance - Distance · Assessment of model performance • Likelihood • AIC • Absolute measures of model fit Chi-squared test Qq plots Kolmogorov-Smirnov](https://reader034.fdocuments.us/reader034/viewer/2022051810/60199f0b49e1c131e435033f/html5/thumbnails/4.jpg)
Limitations of AIC
Cannot be used to select between models when:
• sample size n differs
• truncation distance w differs
• data are grouped, and cutpoints differ
• data are grouped in one analysis and ungrouped in the other
![Page 5: L2-1 Assessment of Model Performance - Distance · Assessment of model performance • Likelihood • AIC • Absolute measures of model fit Chi-squared test Qq plots Kolmogorov-Smirnov](https://reader034.fdocuments.us/reader034/viewer/2022051810/60199f0b49e1c131e435033f/html5/thumbnails/5.jpg)
Goodness-of-Fit
• Chi-squared test for grouped (interval) data; if data are exact, we must specifyinterval cutpoints for this test
• Q-Q plots and related tests for exact data
![Page 6: L2-1 Assessment of Model Performance - Distance · Assessment of model performance • Likelihood • AIC • Absolute measures of model fit Chi-squared test Qq plots Kolmogorov-Smirnov](https://reader034.fdocuments.us/reader034/viewer/2022051810/60199f0b49e1c131e435033f/html5/thumbnails/6.jpg)
Define u distance intervals, with ni detections in interval i, i = 1, ..., u.
Then
where
Chi-squared tests
ˆ
ˆ
i
2ii
u
1=i
2
n
)n-n(=
n=n i
i
and is the proportion of the area under the estimated pdf, , that lies ininterval i.
If the model is ‘correct’:
q = no. of parameters
i (x)f
2
1-q-u
2 ~
![Page 7: L2-1 Assessment of Model Performance - Distance · Assessment of model performance • Likelihood • AIC • Absolute measures of model fit Chi-squared test Qq plots Kolmogorov-Smirnov](https://reader034.fdocuments.us/reader034/viewer/2022051810/60199f0b49e1c131e435033f/html5/thumbnails/7.jpg)
Chaffinch line transect data
0.0
0.2
0.4
0.6
0.8
1.0
1.2
0 10 20 30 40 50 60 70 80 90 100
Perpendicular distance in meters
![Page 8: L2-1 Assessment of Model Performance - Distance · Assessment of model performance • Likelihood • AIC • Absolute measures of model fit Chi-squared test Qq plots Kolmogorov-Smirnov](https://reader034.fdocuments.us/reader034/viewer/2022051810/60199f0b49e1c131e435033f/html5/thumbnails/8.jpg)
χ2 goodness-of-fit testCell Cut Observed Expected Chi-square
i Points Values Values Values-----------------------------------------------------------------1 0.000 12.5 16 15.32 0.0302 12.5 22.5 11 11.63 0.0343 22.5 32.5 11 10.62 0.0134 32.5 42.5 8 9.33 0.1895 42.5 52.5 9 7.87 0.1646 52.5 62.5 7 6.37 0.0627 62.5 77.5 3 6.96 2.2538 77.5 95.0 8 4.91 1.953
-----------------------------------------------------------------Total Chi-square value = 4.6970 Degrees of Freedom = 6.00
Probability of a greater chi-square value, P = 0.58322
The program has limited capability for pooling. The user shouldjudge the necessity for pooling and if necessary, do pooling by hand.
![Page 9: L2-1 Assessment of Model Performance - Distance · Assessment of model performance • Likelihood • AIC • Absolute measures of model fit Chi-squared test Qq plots Kolmogorov-Smirnov](https://reader034.fdocuments.us/reader034/viewer/2022051810/60199f0b49e1c131e435033f/html5/thumbnails/9.jpg)
Q-Q Plots and Related Tests
0 1 2 3
3
6
9
x
f(x)
Half area under histogram
5/6ths area under histogram
![Page 10: L2-1 Assessment of Model Performance - Distance · Assessment of model performance • Likelihood • AIC • Absolute measures of model fit Chi-squared test Qq plots Kolmogorov-Smirnov](https://reader034.fdocuments.us/reader034/viewer/2022051810/60199f0b49e1c131e435033f/html5/thumbnails/10.jpg)
10
1/2 5/6
1
0
5/6
1/2
Observed fraction of data xi
Exp
ecte
d(f
itte
d)
frac
tio
no
fd
ata
<=
x i
Fitt
edC
um
ula
tive
Dis
trib
uti
on
Fun
ctio
n(C
DF)
Empirical Distribution Function (EDF)
![Page 11: L2-1 Assessment of Model Performance - Distance · Assessment of model performance • Likelihood • AIC • Absolute measures of model fit Chi-squared test Qq plots Kolmogorov-Smirnov](https://reader034.fdocuments.us/reader034/viewer/2022051810/60199f0b49e1c131e435033f/html5/thumbnails/11.jpg)
Example: Rounding to zero
0
12 3
3
6
9
x
f(x)
Half area under histogram
Rounding!
5/6ths area under histogram
![Page 12: L2-1 Assessment of Model Performance - Distance · Assessment of model performance • Likelihood • AIC • Absolute measures of model fit Chi-squared test Qq plots Kolmogorov-Smirnov](https://reader034.fdocuments.us/reader034/viewer/2022051810/60199f0b49e1c131e435033f/html5/thumbnails/12.jpg)
10
1/2 5/6
1
0
5/6
1/2
Observed fraction of data xi
Exp
ecte
d(f
itte
d)
frac
tio
no
fd
ata
<=
x i
Fitt
edC
um
ula
tive
Dis
trib
uti
on
Fun
ctio
n(C
DF)
Empirical Distribution Function (EDF)
Bad fit!
![Page 13: L2-1 Assessment of Model Performance - Distance · Assessment of model performance • Likelihood • AIC • Absolute measures of model fit Chi-squared test Qq plots Kolmogorov-Smirnov](https://reader034.fdocuments.us/reader034/viewer/2022051810/60199f0b49e1c131e435033f/html5/thumbnails/13.jpg)
Kolmogorov-Smirnov test
101/2 5/6
1
0
5/6
1/2
CD
F
EDF
Uses biggest difference
![Page 14: L2-1 Assessment of Model Performance - Distance · Assessment of model performance • Likelihood • AIC • Absolute measures of model fit Chi-squared test Qq plots Kolmogorov-Smirnov](https://reader034.fdocuments.us/reader034/viewer/2022051810/60199f0b49e1c131e435033f/html5/thumbnails/14.jpg)
Cramér-von Mises test
101/2 5/6
1
0
5/6
1/2
CD
F
EDF
Uses sum of all squared differences
(Can use Weighted sum)
![Page 15: L2-1 Assessment of Model Performance - Distance · Assessment of model performance • Likelihood • AIC • Absolute measures of model fit Chi-squared test Qq plots Kolmogorov-Smirnov](https://reader034.fdocuments.us/reader034/viewer/2022051810/60199f0b49e1c131e435033f/html5/thumbnails/15.jpg)
Chaffinch line transect Q-Q plot
0
0.2
0.4
0.6
0.8
1
0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
Empirical distribution function
![Page 16: L2-1 Assessment of Model Performance - Distance · Assessment of model performance • Likelihood • AIC • Absolute measures of model fit Chi-squared test Qq plots Kolmogorov-Smirnov](https://reader034.fdocuments.us/reader034/viewer/2022051810/60199f0b49e1c131e435033f/html5/thumbnails/16.jpg)
K-S test and Cramer-von Mises test
Kolmogorov-Smirnov test-----------------------
D_n = 0.0573 p = 0.9703
Cramer-von Mises family tests-----------------------------
W-sq (uniform weighting) = 0.0368 0.900 < p <= 1.000Relevant critical values:
W-sq crit(alpha=0.900) = 0.0000
C-sq (cosine weighting) = 0.0257 0.900 < p <= 1.000Relevant critical values:
C-sq crit(alpha=0.900) = 0.0000
![Page 17: L2-1 Assessment of Model Performance - Distance · Assessment of model performance • Likelihood • AIC • Absolute measures of model fit Chi-squared test Qq plots Kolmogorov-Smirnov](https://reader034.fdocuments.us/reader034/viewer/2022051810/60199f0b49e1c131e435033f/html5/thumbnails/17.jpg)
Q-Q Plot Summary
• Q-Q plots show goodness-of-fit at “high resolution” – without requiring grouping intointervals
• Kolmogorov-Smirnov test and Cramér-von Mises test are goodness-of-fit tests that do notrequire grouping
• Cramér-von Mises test can be weighted, to give higher weight to x near zero