Rome, 8-11 July 2008 QUALITY 2008 An application to the Italian IIP Revision analysis to detect...
-
Upload
eileen-dean -
Category
Documents
-
view
213 -
download
0
Transcript of Rome, 8-11 July 2008 QUALITY 2008 An application to the Italian IIP Revision analysis to detect...
Rome, 8-11 July 2008
QUALITY 2008
An application to the Italian IIP
Revision analysis to detectpossible weakness in the
estimation procedures
A. Ciammola, T. Gambuti and A. Mancini
ISTAT
European Conference on Quality in Official Statistics
2/21Rome, 8-11 July 2008
QUALITY 2008
Introduction
Why revision analysis?
A case study
Next steps
Outline
3/21Rome, 8-11 July 2008
QUALITY 2008 Quality and some of its dimensions
AccuracyCloseness of the
estimate to the true (but unknown) value of the
variable to be measured
TimelinessSpan between the
reference period and the publication period
RevisionReliability measure
ReliabilityCloseness between
preliminary estimate and subsequent estimates
U
4/21Rome, 8-11 July 2008
QUALITY 2008 Revision analysis
Real-time databases collection of vintages computation of revisions
Revisions Rt = Lt – Pt Rt = (Lt – Pt) / Lt
Revision measures Size (MAR, RMAR, …) Bias (MR, T-test) Efficiency (News or Noise?, MSR, …)
5/21Rome, 8-11 July 2008
QUALITY 2008 Useful references
OECD / Eurostat Guidelines on Revisions Policy and Analysis
http://www.oecd.org Themes related to revision policy and analysis
Recommendations for revisions policy and analysis Guidelines for establishing a real-time database Recommended statistical measures Pre-programmed software for performing revisions analysis A framework for revisions policy of key economic indicators Comprehensive framework of reasons for revisions and their timing Guidelines on how to decompose total revision into different reasons
for revisions Guidelines on how to use the results from revision analyses to improve
compilation methods Case studies on the relationship between timeliness of release and
size of revisions
6/21Rome, 8-11 July 2008
QUALITY 2008
For users
Objective Availability of all the relevant information for using appropriately the estimates of short-term indicators at different stages of the revision process
provision of information about
past revisions
schedule future revisions
(statistical and definitional)
real-time databases gathering all the vintages
analysis of size, bias and efficiency
of revisions
Why do we measure revisions?
7/21Rome, 8-11 July 2008
QUALITY 2008
For producers
Underlying issues
Targets
Why do we measure revisions?
Bias in the revision process
Inefficiency in compilation of preliminary estimates
Reduction of (the size of) “avoidable” revisions
Detection of the source for bias /
inefficiency
8/21Rome, 8-11 July 2008
QUALITY 2008 A case study
Italian Index of Industrial Production (IIP)
1. Sources and timing of revisions
2. Revision analysis
3. Identification of specific sources for bias
4. Some evidences
9/21Rome, 8-11 July 2008
QUALITY 2008 1. Sources and timing of revisions
Y(t-3) Y(t-2) Y(t-1)Current Year Y(t) – Reference month
J F M A M J J A S O N D
M LR CE
A LR CE PC
M LR CE
J LR CE
J LR CE
A LR CE
S LR CE
O LR CE LR CE
N LR CE
D LR CE
J LR CE
F LR CE
First estimate Second estimate Six-month revision Annual revision
LR Late respondents CE Correction of errors PC Productivity coefficients
10/21Rome, 8-11 July 2008
QUALITY 2008
IIP - Revisions on year-on-year growth rates (raw indices)
Legend *
h=1 – after one month h=12 – after 12 months
MAR – Mean Absolute Revision RMAR – Relative MAR
MR – Mean Revision SD – Standard Deviation
Period: Jan-03 / Dec-07 h=1 h=12
# of revisions 60 48
MAR 0.142 0.246
RMAR 0.053 0.087
MR 0.075 0.083
SD of MR(HAC) 0.021 0.056
T-value 3.564 1.489
Significance of MR Yes * No *
2. Revision analysis
11/21Rome, 8-11 July 2008
QUALITY 2008
Jun Dec-03 Jun Dec-04 Jun Dec-05 Jun Dec-06 Jun Dec-07
-0.5
-0.4
-0.3
-0.2
-0.1
0
0.1
0.2
0.3
0.4
0.5
2. Revision analysisIIP - Revisions after one month on year-on-year growth rates (raw indices)
12/21Rome, 8-11 July 2008
QUALITY 2008 2. Revision analysis
Why this systematic component? Late respondents? Correction of errors? Productivity coefficients?
(In revisions after 1 month, only July 2004, January 2005, January 2006 and January 2007 are affected)
Which sectors? All sectors? Some specific sector?
How to proceed? Simulation exercise Top-down approach Quality indicators
13/21Rome, 8-11 July 2008
QUALITY 2008 3. Identification of specific sources for bias
Simulation exercise
Removal of the effect of the productivity coefficients
Isolate sources of revisions external to the survey
Fulfil the condition necessary to compute the average contribution of each components to the IIP revisions
PS: revisions computed on y-o-y growth rates
14/21Rome, 8-11 July 2008
QUALITY 2008
IIP
Migs CND CDU CAP INT ENE
Divisions DINT,1 … DINT,j … DINT,7
Groups GINT,1 … GINT,k … GINT,20
Classes CINT,1 … CINT,m … … ... ... CINT,n ... …
Diagram describing the top-down approach
3. Identification of specific sources for bias
15/21Rome, 8-11 July 2008
QUALITY 2008
Quality indicators
Revision measures Contribution of each component to the mean revision
of the higher component
Response rates
3. Identification of specific sources for bias
16/21Rome, 8-11 July 2008
QUALITY 2008
MIGS - Revisions after one month on raw Y-o-Y growth rates
Period: Jan-03 / Dec-07 CND CDU CAP INT ENE
Weights % 22.9 6.1 23.8 35.5 11.7
MAR 0.272 0.415 0.378 0.223 0.149
RMAR 0.084 0.081 0.088 0.073 0.040
MR 0.092 0.072 0.042 0.143 -.003
Contribution to MR ° 0.019 0.006 0.010 0.047 -.003
SD of MR(HAC) 0.047 0.103 0.071 0.030 0.040
T-value 1.962 0.694 0.589 4.724 -.079
Significance of MR No * No * No * Yes * No *
Legend CND – Consumer non durables
CDU – Consumer durables CAP – Capital goods
INT – Intermediate Goods ENE – Energy
° Period Jan-04 / Dec-07 *
4. Some evidences
17/21Rome, 8-11 July 2008
QUALITY 2008
Dec-03 Dec-04 Dec-05 Dec-06 Dec-07-1.5
-1
-0.5
0
0.5
1
1.5Capital goods
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec-1.5
-1
-0.5
0
0.5
1
1.5Capital goods by month
Dec-03 Dec-04 Dec-05 Dec-06 Dec-07-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
1.2Intermediate goods
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
1.2Intermediate goods by month
Revisions after one month on raw Y-o-Y growth rates
4. Some evidences
18/21Rome, 8-11 July 2008
QUALITY 2008
Average weighted response rates
Year Estimate IIP CND CDU CAP INT ENE
2004First 91.5 93.9 94.3 90.3 88.3 97.4
Second 95.0 95.7 96.1 93.4 93.8 99.6
2005First 90.2 90.6 93.5 88.1 87.9 98.7
Second 93.3 93.1 95.4 91.3 92.4 100.0
2006First 88.7 89.0 90.4 87.4 86.4 97.3
Second 91.7 91.4 92.5 90.1 90.1 99.9
2007First 83.7 84.7 82.4 80.8 80.6 97.6
Second 87.6 88.4 86.0 85.6 84.9 99.0
4. Some evidences
19/21Rome, 8-11 July 2008
QUALITY 2008
Revisions after one month on raw Y-o-Y growth rates
Period: Jan-04 / Dec-07 S SC,INT SC,IIP
Weights % 32.3 / 11.5 67.7 88.5
MAR 0.362 0.263 0.159
RMAR 0.100 0.082 0.055
MR 0.263 0.071 0.056
Contribution to MR 0.088 / 0.030 0.047 0.049
T-value 3.985 1.407 1.766
P-value 0.000 0.166 0.084
Legend
S – Selected subset of INT (19 classes)
SC,INT – Complement of S in INT (S U SC,INT = INT)
SC,IIP – Complement of S in IIP (S U SC,IIP = IIP)
4. Some evidences
20/21Rome, 8-11 July 2008
QUALITY 2008 Next steps
Checking the stability of results over time
Experimenting possible countermeasures to biased revisions Treatment of late respondents with different
estimators Assessment of their effects on revisions
ISTAT web page on revisions Real-time database for several short-term indicators Revision analysis
21/21Rome, 8-11 July 2008
QUALITY 2008
THANK YOU!