WP 4.1 Report of the Retrospective Working Group It is a simple but sometimes forgotten truth that...
-
Upload
esther-parrish -
Category
Documents
-
view
217 -
download
0
description
Transcript of WP 4.1 Report of the Retrospective Working Group It is a simple but sometimes forgotten truth that...
WP 4.1 Report of the Retrospective Working Group
It is a simple but sometimes forgotten truth that the greatest enemy to present joy and high hopes is the cultivation of retrospective bitterness.
Robert G. Menzies (Australian Prime Minister 1930-1941&1949-1966)
Retro WG Participants
Special thanks to ICES WGMG
Name Affiliation Name Affiliation Name Affiliation Chris Legault (chair) NEFSC Larry Alade NEFSC Paul Nitschke NEFSC Bob Mohn DFO Andy Applegate NEFMC Loretta Obrien NEFSC Larry Jacobson NEFSC Jessica Blaylock NEFSC Bill Overholtz NEFSC Alan Seaver NEFSC Liz Brooks NEFSC Michael Palmer NEFSC
Steve Cadrin NEFSC Paul Rago NEFSC Laurel Col NEFSC Anne Richards NEFSC Steve Corriea MADMF Fred Serchuk NEFSC Deborah Hart NEFSC Gary Shepherd NEFSC Lisa Hendrickson NEFSC Kathy Sosebee NEFSC Joe Idoine NEFSC Jiashen Tang NEFSC Chad Keith NEFSC Mark Terceiro NEFSC Jennifer Martin NEFSC Michele Traver NEFSC Ralph Mayo NEFSC Jim Weinberg NEFSC Tim Miller NEFSC Susan Wigley NEFSC Joshua Moser NEFSC
Retro
• What is it?• How is it measured?• Why is it important?• What causes it?• Can we fix it?
Retro Defined
• The retrospective problem is a systematic inconsistency among a series of estimates of population size, or related assessment variables, based on increasing periods of data. (Mohn 1999)
• Historical vs Within Model• Not just VPA
0
10
20
30
40
50
1970 1975 1980 1985 1990 1995 2000 2005
SSB
(tho
usan
d m
t)
Don’t Always Know It When You See It
Retrospective pattern in GB haddock SSB trendin tons
year
SS
B
50000
100000
150000
1970 1980 1990 2000
Retrospective pattern in GB haddock SSB as percent difference from terminal year
year
per
cent
diff
eren
ce fr
om te
rmin
al y
r
-20
-15
-10
-5
0
1970 1980 1990 2000
A Strong Retrospective
Terminal Year - 2005 Terminal Year - 2004 Terminal Year - 2003
Terminal Year - 2002 Terminal Year - 2001 Terminal Year - 2000
0.00
40.00
80.00
120.00
160.00
200.00
240.00
280.00
320.00
360.00
400.00
440.00
480.00
520.00
560.00
600.00
Freq
uenc
y
0.200000 0.300000 0.400000 0.500000 0.600000 0.700000 0.800000 0.900000Fishing Mortality
Average Fishing Mortality2000
0.0
0.5
1.0
1.5
2.0
1970 1975 1980 1985 1990 1995 2000 2005
F(4+
)
Mohn rho
0 10 20
0.400
0.450
0.500
0.550
F 3-5
25 years in assessment7 terms in rho calculation
Woods Hole rho
0 10 20
0.400
0.450
0.500
0.550
F 3-5
25 years in assessment300 (=24+23+22+...+1) terms in rho calculation
rho Increases with Bias
0 0.5 1.0
0
0.5
1
rho B
Disturbance
Why are retros important?
• Projections too high• Catch advice high• Causes F > Fref• Feedback loop
0
5
10
15
20
25
30
2000 2001 2002 2003 2004 2005 2006 20070
5
10
15
20
25
30
2000 2001 2002 2003 2004 2005 2006 20070
5
10
15
20
25
30
2000 2001 2002 2003 2004 2005 2006 2007
Causes of Retros
• Change in Catch (missing or extra)• Change in M• Change in Survey q
• Closed Areas (sessile)
Missing CatchD1006
0
1,000,000
2,000,000
3,000,000
4,000,000
5,000,000
6,000,000
7,000,000
1960 1970 1980 1990 2000 2010 2020
SSB
D1006
0
0.2
0.4
0.6
0.8
1960 1970 1980 1990 2000 2010 2020
F
D0610
0
1,000,000
2,000,000
3,000,000
4,000,000
5,000,000
6,000,000
7,000,000
1960 1970 1980 1990 2000 2010 2020
SSB
D0610
0
0.2
0.4
0.6
0.8
1960 1970 1980 1990 2000 2010 2020
F
Change in MM0204vpa02
0
1,000,000
2,000,000
3,000,000
4,000,000
5,000,000
6,000,000
7,000,000
1960 1970 1980 1990 2000 2010 2020
SSB
M0204vpa04
02,000,0004,000,0006,000,0008,000,000
10,000,00012,000,00014,000,00016,000,000
1960 1970 1980 1990 2000 2010 2020
SSB
M0402vpa02
0
1,000,000
2,000,000
3,000,000
4,000,000
5,000,000
6,000,000
7,000,000
1960 1970 1980 1990 2000 2010 2020
SSB
M0402vpa04
0
2,000,000
4,000,000
6,000,000
8,000,000
10,000,000
12,000,000
1960 1970 1980 1990 2000 2010 2020
SSB
Change in Survey qq12
0
2000000
4000000
6000000
8000000
10000000
12000000
14000000
1960 1970 1980 1990 2000 2010 2020
SSB
q12
0
0.2
0.4
0.6
0.8
1
1960 1970 1980 1990 2000 2010 2020
F
q21
0
1,000,000
2,000,000
3,000,000
4,000,000
5,000,000
6,000,000
7,000,000
1960 1970 1980 1990 2000 2010 2020
SSB
q21
0
0.2
0.4
0.6
0.8
1
1960 1970 1980 1990 2000 2010 2020
F
Closed Area
0 10 20 300
250
500
750
Biom 5+
Year
Retro by Chance Unlikely
1998 1999
2000 2001
2002 2003
2004 2005
True Average F
0.0000
0.3001
0.6002
0.9003
1.2004
1.5005
1.8006
Fis
hing
Mor
talit
y
1983 1986 1989 1992 1995 1998 2001 2004Year
Fishing Mortality RetrospectiveVPA Iteration # 65
Can a Retro be Fixed?
• Yes– Can remove retro pattern in many ways
• No– Do not necessarily get closer to the truth
• Need to known timing and source to get it right– Timing can sometimes be identified– Source cannot be identified
Local Influence Surface
• Cadigan and Farrell (2002, 2005)• Use rho – no good• Use other metric, can sometimes identify
timing
10 20 301
2
3
4
5
6 NPeel 10 F3-5
Year
Age -2.7
-1.3
0.16
1.6
3.1
LIS with rho f(npeels)
Source of Retro 1990
-1.5
-1.0
-0.5
0.0
0.5
1.0
1.5
2.0
1980 1985 1990 1995 2000 2005
Sum
of G
radi
ent o
ver A
ges
npeel4npeel8npeel12npeel16
Source of Retro 2000
-1.5
-1.0
-0.5
0.0
0.5
1.0
1.5
2.0
1980 1985 1990 1995 2000 2005
Sum
of G
radi
ent o
ver A
ges
npeel4npeel8npeel12npeel16
Split Surveys
• GB yt “solution” to retro• Removes retro, but not necessarily to truth
Change Catch (seen previously)D1006
0
1,000,000
2,000,000
3,000,000
4,000,000
5,000,000
6,000,000
7,000,000
1960 1970 1980 1990 2000 2010 2020
SSB
D1006
0
0.2
0.4
0.6
0.8
1960 1970 1980 1990 2000 2010 2020
F
D0610
0
1,000,000
2,000,000
3,000,000
4,000,000
5,000,000
6,000,000
7,000,000
1960 1970 1980 1990 2000 2010 2020
SSB
D0610
0
0.2
0.4
0.6
0.8
1960 1970 1980 1990 2000 2010 2020
F
Change Catch Split SurveysD1006
0
1,000,000
2,000,000
3,000,000
4,000,000
5,000,000
6,000,000
7,000,000
1960 1970 1980 1990 2000 2010 2020
SSB
D1006
0
0.2
0.4
0.6
0.8
1960 1970 1980 1990 2000 2010 2020
F
D0610
0
1,000,000
2,000,000
3,000,000
4,000,000
5,000,000
6,000,000
7,000,000
1960 1970 1980 1990 2000 2010 2020
SSB
D0610
0
0.2
0.4
0.6
0.8
1960 1970 1980 1990 2000 2010 2020
F
M Change (seen previously)M0204vpa02
0
1,000,000
2,000,000
3,000,000
4,000,000
5,000,000
6,000,000
7,000,000
1960 1970 1980 1990 2000 2010 2020
SSB
M0204vpa04
02,000,0004,000,0006,000,0008,000,000
10,000,00012,000,00014,000,00016,000,000
1960 1970 1980 1990 2000 2010 2020
SSB
M0402vpa02
0
1,000,000
2,000,000
3,000,000
4,000,000
5,000,000
6,000,000
7,000,000
1960 1970 1980 1990 2000 2010 2020
SSB
M0402vpa04
0
2,000,000
4,000,000
6,000,000
8,000,000
10,000,000
12,000,000
1960 1970 1980 1990 2000 2010 2020
SSB
M Change Split SurveysM0204vpa02
0
1,000,000
2,000,000
3,000,000
4,000,000
5,000,000
6,000,000
7,000,000
1960 1970 1980 1990 2000 2010 2020
SSB
M0204vpa04
02,000,0004,000,0006,000,0008,000,000
10,000,00012,000,00014,000,00016,000,000
1960 1970 1980 1990 2000 2010 2020
SSB
M0402vpa02
0
1,000,000
2,000,000
3,000,000
4,000,000
5,000,000
6,000,000
7,000,000
1960 1970 1980 1990 2000 2010 2020
SSB
M0402vpa04
0
2,000,000
4,000,000
6,000,000
8,000,000
10,000,000
12,000,000
1960 1970 1980 1990 2000 2010 2020
SSB
Change q (seen previously)q12
0
2000000
4000000
6000000
8000000
10000000
12000000
14000000
1960 1970 1980 1990 2000 2010 2020
SSB
q12
0
0.2
0.4
0.6
0.8
1
1960 1970 1980 1990 2000 2010 2020
F
q21
0
1,000,000
2,000,000
3,000,000
4,000,000
5,000,000
6,000,000
7,000,000
1960 1970 1980 1990 2000 2010 2020
SSB
q21
0
0.2
0.4
0.6
0.8
1
1960 1970 1980 1990 2000 2010 2020
F
Change q Split Surveysq12
0
1,000,000
2,000,000
3,000,000
4,000,000
5,000,000
6,000,000
7,000,000
1960 1970 1980 1990 2000 2010 2020
SSB
q12
0
0.2
0.4
0.6
0.8
1
1960 1970 1980 1990 2000 2010 2020
F
q21
0
1,000,000
2,000,000
3,000,000
4,000,000
5,000,000
6,000,000
7,000,000
1960 1970 1980 1990 2000 2010 2020
SSB
q21
0
0.2
0.4
0.6
0.8
1
1960 1970 1980 1990 2000 2010 2020
F
Source: Change q, Fix: Change M q12M0203
0
2000000
4000000
6000000
8000000
10000000
12000000
1960 1970 1980 1990 2000 2010 2020
SSB
q12M0203
0
0.2
0.4
0.6
0.8
1
1960 1970 1980 1990 2000 2010 2020
F
q12M02035
0
2000000
4000000
6000000
8000000
10000000
12000000
1960 1970 1980 1990 2000 2010 2020
SSB
q12M02035
0
0.2
0.4
0.6
0.8
1
1960 1970 1980 1990 2000 2010 2020F
q12M0204
0
2000000
4000000
6000000
8000000
10000000
12000000
14000000
1960 1970 1980 1990 2000 2010 2020
SSB
q12M0204
0
0.2
0.4
0.6
0.8
1
1960 1970 1980 1990 2000 2010 2020
F
Retro Fix Resids
0.00
0.25
0.50
-0.25
-0.50
Res
idua
l
1971 1974 1977 1980 1983 1986 1989 1992 1995 1998 2001 2004 2007 2010Years
ResidualsAGE5 5 0.00
0.25
0.50
-0.25
-0.50
Res
idua
l
1971 1974 1977 1980 1983 1986 1989 1992 1995 1998 2001 2004 2007 2010Years
ResidualsAGE5 5
0.00
0.25
0.50
-0.25
-0.50
Res
idua
l
1971 1974 1977 1980 1983 1986 1989 1992 1995 1998 2001 2004 2007 2010Years
ResidualsAGE5 5
Split Survey Series
Original
Alternative “Fixes”
• Adjust NAA in Projections– Assumes retro will persist– Requires MSE to determine how to adjust
• Provide Alternative States of Nature Advice
Conclusions1. Retrospective pattern is an indication something is inconsistent (data
and/or model).2. Lack of a retrospective pattern does not mean all is well. Based on
simulations, data or model inconsistency does not always produce a retrospective pattern. Retrospective patterning is just one diagnostic to be considered when conducting stock assessments.
3. Simulated retrospective patterns can be caused by time trending changes in biological characteristics, catch, survey catchability, or spatial concentration of the population. Multiple sources may occur in assessments.
4. The source(s) of the retrospective pattern can be anywhere in the time series. Some methods were presented to identify when the change took place (moving window, q surface, mean square residual LIS).
5. The true source(s) of a retrospective pattern have not been identified using current methods. Knowledge of events in the fishery or biological information may help identify probable sources.
Conclusions (cont)6. Interventions (correlated errors) are more likely to cause
retrospective patterns than random noise.7. Splitting surveys, changing M, or changing catch may reduce the
retrospective pattern, but do not necessarily produce an assessment closer to the truth, although the other diagnostics for the new assessment may be fine.
8. The retrospective statistic, rho, may be a useful measure of the amount of retrospective pattern. A strong retrospective pattern can be defined by the degree of overlap between confidence intervals from different terminal years.
9. Local influence surface analysis using rho is not useful for diagnosing the timing or source of retrospective patterns.
10. In many stocks, strong retrospective patterns typically persist.
Recommendations1. Always check for the presence of a retrospective pattern.2. If a model shows a retrospective pattern, then consider alternative models
or model assumptions.3. Develop objective and consistent criteria for the acceptance of
assessments with retrospective patterns. 4. A strong retrospective pattern is grounds to reject the assessment model
as an indication of stock status or the basis for management advice. 5. When a moderate retrospective pattern is encountered: (not an
exhaustive list)a. Consider alternative states of nature approach to advice.b. Investigate the performance of alternative methods for retrospective adjustments through management strategy evaluations.
6. Use biological and fishery hypotheses and auxilliary information as a basis for adjustments for retrospective patterns.
7. Consider use of survey swept area numbers instead of mean catch per tow in assessment models.
8. The presence and implications of a retrospective pattern as a source of uncertainty in the assessment should be clearly communicated to managers.