A Review of Benchmarking Methods G Brown, N Parkin, and N Stuttard, ONS.
-
Upload
tyrone-reynolds -
Category
Documents
-
view
214 -
download
0
Transcript of A Review of Benchmarking Methods G Brown, N Parkin, and N Stuttard, ONS.
![Page 1: A Review of Benchmarking Methods G Brown, N Parkin, and N Stuttard, ONS.](https://reader036.fdocuments.us/reader036/viewer/2022081520/5697bf871a28abf838c888ae/html5/thumbnails/1.jpg)
A Review of Benchmarking Methods
G Brown, N Parkin, and N Stuttard, ONS
![Page 2: A Review of Benchmarking Methods G Brown, N Parkin, and N Stuttard, ONS.](https://reader036.fdocuments.us/reader036/viewer/2022081520/5697bf871a28abf838c888ae/html5/thumbnails/2.jpg)
Overview
• Introduction• What is benchmarking?• What we did and why• Some methods for benchmarking• Some quality measures• Comparison of methods• Summary
2
![Page 3: A Review of Benchmarking Methods G Brown, N Parkin, and N Stuttard, ONS.](https://reader036.fdocuments.us/reader036/viewer/2022081520/5697bf871a28abf838c888ae/html5/thumbnails/3.jpg)
Introduction
• Purpose: to recommend a method for benchmarking to ONS and wider GSS
• Benchmarking combines two time series of same phenomenon, measured at different frequencies
• Result: benchmarked series is higher quality• Work funded from Quality Improvement Fund
3
![Page 4: A Review of Benchmarking Methods G Brown, N Parkin, and N Stuttard, ONS.](https://reader036.fdocuments.us/reader036/viewer/2022081520/5697bf871a28abf838c888ae/html5/thumbnails/4.jpg)
What we did and why
• Identified appropriate benchmarking methods• Tested using several hundred ONS time series• Used range of quality measures to rank
methods• Made judgment to combine results from
different quality measures• Recommended a benchmarking method• Update of ONS computer systems prompted
examination of methods
4
![Page 5: A Review of Benchmarking Methods G Brown, N Parkin, and N Stuttard, ONS.](https://reader036.fdocuments.us/reader036/viewer/2022081520/5697bf871a28abf838c888ae/html5/thumbnails/5.jpg)
Benchmarking
• Want good estimates of levels and growth• Have two series measuring same
phenomenon• Different frequencies• Higher frequency more timely, accurate
growthso Indicator series
• Lower frequency delayed, more accurate levelso Benchmark series
5
![Page 6: A Review of Benchmarking Methods G Brown, N Parkin, and N Stuttard, ONS.](https://reader036.fdocuments.us/reader036/viewer/2022081520/5697bf871a28abf838c888ae/html5/thumbnails/6.jpg)
Benchmarking
• Resulting high frequency serieso Benchmarked series
• Has good estimates of growth combined with good estimates of level
6
![Page 7: A Review of Benchmarking Methods G Brown, N Parkin, and N Stuttard, ONS.](https://reader036.fdocuments.us/reader036/viewer/2022081520/5697bf871a28abf838c888ae/html5/thumbnails/7.jpg)
Benchmarking
• Two types of relation between indicator and benchmark:o Point in timeo Average
7
![Page 8: A Review of Benchmarking Methods G Brown, N Parkin, and N Stuttard, ONS.](https://reader036.fdocuments.us/reader036/viewer/2022081520/5697bf871a28abf838c888ae/html5/thumbnails/8.jpg)
Benchmarking, point in time
• Example: unemployment monthly and quarterly
• Benchmarks apply to the third month in each quarter
• Third monthly estimate in each quarter is forced to equal benchmark
8
![Page 9: A Review of Benchmarking Methods G Brown, N Parkin, and N Stuttard, ONS.](https://reader036.fdocuments.us/reader036/viewer/2022081520/5697bf871a28abf838c888ae/html5/thumbnails/9.jpg)
9
![Page 10: A Review of Benchmarking Methods G Brown, N Parkin, and N Stuttard, ONS.](https://reader036.fdocuments.us/reader036/viewer/2022081520/5697bf871a28abf838c888ae/html5/thumbnails/10.jpg)
10
![Page 11: A Review of Benchmarking Methods G Brown, N Parkin, and N Stuttard, ONS.](https://reader036.fdocuments.us/reader036/viewer/2022081520/5697bf871a28abf838c888ae/html5/thumbnails/11.jpg)
Benchmarking, average
• Example: turnover monthly and quarterly• Benchmarks apply to each month in each
quarter• Average turnover of three months in each
quarter is forced to equal benchmark
11
![Page 12: A Review of Benchmarking Methods G Brown, N Parkin, and N Stuttard, ONS.](https://reader036.fdocuments.us/reader036/viewer/2022081520/5697bf871a28abf838c888ae/html5/thumbnails/12.jpg)
12
![Page 13: A Review of Benchmarking Methods G Brown, N Parkin, and N Stuttard, ONS.](https://reader036.fdocuments.us/reader036/viewer/2022081520/5697bf871a28abf838c888ae/html5/thumbnails/13.jpg)
13
![Page 14: A Review of Benchmarking Methods G Brown, N Parkin, and N Stuttard, ONS.](https://reader036.fdocuments.us/reader036/viewer/2022081520/5697bf871a28abf838c888ae/html5/thumbnails/14.jpg)
Non-negativity
• Most indicator series must be non-negative
• In those cases the benchmarked series must be non-negative too
• Process of benchmarking can produce negative benchmarked series
14
![Page 15: A Review of Benchmarking Methods G Brown, N Parkin, and N Stuttard, ONS.](https://reader036.fdocuments.us/reader036/viewer/2022081520/5697bf871a28abf838c888ae/html5/thumbnails/15.jpg)
15
![Page 16: A Review of Benchmarking Methods G Brown, N Parkin, and N Stuttard, ONS.](https://reader036.fdocuments.us/reader036/viewer/2022081520/5697bf871a28abf838c888ae/html5/thumbnails/16.jpg)
What we did and why
• Identified appropriate benchmarking methods• Tested using several hundred ONS time series• Used range of quality measures to rank
methods• Made judgment to combine results from
different quality measures• Recommended a benchmarking method• Update of ONS computer systems prompted
examination of methods
16
![Page 17: A Review of Benchmarking Methods G Brown, N Parkin, and N Stuttard, ONS.](https://reader036.fdocuments.us/reader036/viewer/2022081520/5697bf871a28abf838c888ae/html5/thumbnails/17.jpg)
Benchmarking methods
• Methods suggested by ONS, variants with different splineso proc Expand (in SAS)
o INTER
o Kruger
• Denton
• Cholette-Dagum
• Constrained versions of the above for non-negativity
17
![Page 18: A Review of Benchmarking Methods G Brown, N Parkin, and N Stuttard, ONS.](https://reader036.fdocuments.us/reader036/viewer/2022081520/5697bf871a28abf838c888ae/html5/thumbnails/18.jpg)
Benchmarking methods
• Methods suggested by ONS, variants with different splineso proc Expand (in SAS)
o INTER
o Kruger
• Denton
• Cholette-Dagum
• Constrained versions of the above for non-negativity
18
![Page 19: A Review of Benchmarking Methods G Brown, N Parkin, and N Stuttard, ONS.](https://reader036.fdocuments.us/reader036/viewer/2022081520/5697bf871a28abf838c888ae/html5/thumbnails/19.jpg)
Benchmarking methods
• Methods suggested by ONS, variants with different splineso proc Expand (in SAS)
o INTER
o Kruger
• Denton
• Cholette-Dagum
• Constrained versions of the above for non-negativity
19
![Page 20: A Review of Benchmarking Methods G Brown, N Parkin, and N Stuttard, ONS.](https://reader036.fdocuments.us/reader036/viewer/2022081520/5697bf871a28abf838c888ae/html5/thumbnails/20.jpg)
Benchmarking methods
• Methods suggested by ONS, variants with different splineso proc Expand (in SAS)
o INTER
o Kruger
• Denton
• Cholette-Dagum
• Constrained versions of the above for non-negativity
20
![Page 21: A Review of Benchmarking Methods G Brown, N Parkin, and N Stuttard, ONS.](https://reader036.fdocuments.us/reader036/viewer/2022081520/5697bf871a28abf838c888ae/html5/thumbnails/21.jpg)
ONS methods (and variants)
• Summary: fits smooth curve through knots
1. Aggregate indicator series
2. Calculate ratio of aggregated to benchmark
3. Augment with fore/backcasts using X-12-ARIMA
4. Interpolate to frequency of indicator
5. Multiply indicator by interpolated series
6. Iterate 1 to 5
• Variants use different ways to interpolate21
![Page 22: A Review of Benchmarking Methods G Brown, N Parkin, and N Stuttard, ONS.](https://reader036.fdocuments.us/reader036/viewer/2022081520/5697bf871a28abf838c888ae/html5/thumbnails/22.jpg)
Interpolation
• Three types of cubic spline
1.Proc Expand (point in time/average)
2. INTER (average)
3.Kruger (point in time)
• Progressively less prone to produce negative values
22
![Page 23: A Review of Benchmarking Methods G Brown, N Parkin, and N Stuttard, ONS.](https://reader036.fdocuments.us/reader036/viewer/2022081520/5697bf871a28abf838c888ae/html5/thumbnails/23.jpg)
Denton type
• Summary: try to preserve movements in indicator
• Minimise a penalty function of differences or relative differences between indicator and benchmark
• Minimisation using either special methods or off-the-shelf methods for quadratic minimisation
• Denton usually set up to minimise first differences or proportionate first differences
23
![Page 24: A Review of Benchmarking Methods G Brown, N Parkin, and N Stuttard, ONS.](https://reader036.fdocuments.us/reader036/viewer/2022081520/5697bf871a28abf838c888ae/html5/thumbnails/24.jpg)
Denton and Cholette-Dagum
• For indicator points with no benchmark:• Denton carries forward the most recent
difference between benchmark and indicator• Cholette-Dagum assumes the difference
decays to zero in a defined way• Flexible in the way this is modelled• We assume:
o Decay is geometrico Rate of decay fixed in advance for all series
24
![Page 25: A Review of Benchmarking Methods G Brown, N Parkin, and N Stuttard, ONS.](https://reader036.fdocuments.us/reader036/viewer/2022081520/5697bf871a28abf838c888ae/html5/thumbnails/25.jpg)
25
![Page 26: A Review of Benchmarking Methods G Brown, N Parkin, and N Stuttard, ONS.](https://reader036.fdocuments.us/reader036/viewer/2022081520/5697bf871a28abf838c888ae/html5/thumbnails/26.jpg)
Non-negativity
• ONS suggestion:o Benchmark on log scaleo Exponentiateo Distribute residual differences
• Optimisation approach for Denton type:o Set up basic method as a matrix problemo Add constraints as part of matrix setupo Solve using off-the-shelf optimiser in SAS
26
![Page 27: A Review of Benchmarking Methods G Brown, N Parkin, and N Stuttard, ONS.](https://reader036.fdocuments.us/reader036/viewer/2022081520/5697bf871a28abf838c888ae/html5/thumbnails/27.jpg)
What we did and why
• Identified appropriate benchmarking methods• Tested using several hundred ONS time series• Used range of quality measures to rank
methods• Made judgment to combine results from
different quality measures• Recommended a benchmarking method• Update of ONS computer systems prompted
examination of methods
27
![Page 28: A Review of Benchmarking Methods G Brown, N Parkin, and N Stuttard, ONS.](https://reader036.fdocuments.us/reader036/viewer/2022081520/5697bf871a28abf838c888ae/html5/thumbnails/28.jpg)
Time series used for testing
• Mixture of: o Monthly to quarterlyo Quarterly to annualo Average and point in time
• Different lengths• Included some awkward series (to test non-
negativity)
28
![Page 29: A Review of Benchmarking Methods G Brown, N Parkin, and N Stuttard, ONS.](https://reader036.fdocuments.us/reader036/viewer/2022081520/5697bf871a28abf838c888ae/html5/thumbnails/29.jpg)
What we did and why
• Identified appropriate benchmarking methods• Tested using several hundred ONS time series• Used range of quality measures to rank
methods• Made judgment to combine results from
different quality measures• Recommended a benchmarking method• Update of ONS computer systems prompted
examination of methods
29
![Page 30: A Review of Benchmarking Methods G Brown, N Parkin, and N Stuttard, ONS.](https://reader036.fdocuments.us/reader036/viewer/2022081520/5697bf871a28abf838c888ae/html5/thumbnails/30.jpg)
How the methods were compared
1. Failures
2. Verification of benchmarking constraint
3. Preserving change
4. Revisions
5. Smoothness
6. Closeness
30
![Page 31: A Review of Benchmarking Methods G Brown, N Parkin, and N Stuttard, ONS.](https://reader036.fdocuments.us/reader036/viewer/2022081520/5697bf871a28abf838c888ae/html5/thumbnails/31.jpg)
How the methods were compared
1. Failures – program fails to benchmark
2. Verification of benchmarking constraint
3. Preserving change
4. Revisions
5. Smoothness
6. Closeness
31
![Page 32: A Review of Benchmarking Methods G Brown, N Parkin, and N Stuttard, ONS.](https://reader036.fdocuments.us/reader036/viewer/2022081520/5697bf871a28abf838c888ae/html5/thumbnails/32.jpg)
How the methods were compared
1. Failures
2. Verification of benchmarking constraint - benchmarked not equal to benchmark
3. Preserving change
4. Revisions
5. Smoothness
6. Closeness
32
![Page 33: A Review of Benchmarking Methods G Brown, N Parkin, and N Stuttard, ONS.](https://reader036.fdocuments.us/reader036/viewer/2022081520/5697bf871a28abf838c888ae/html5/thumbnails/33.jpg)
How the methods were compared
1. Failures
2. Verification of benchmarking constraint
3. Preserving change – size and direction
4. Revisions
5. Smoothness
6. Closeness
33
![Page 34: A Review of Benchmarking Methods G Brown, N Parkin, and N Stuttard, ONS.](https://reader036.fdocuments.us/reader036/viewer/2022081520/5697bf871a28abf838c888ae/html5/thumbnails/34.jpg)
How the methods were compared
1. Failures
2. Verification of benchmarking constraint
3. Preserving change
4. Revisions – size & bias when perturbing or adding benchmark
5. Smoothness
6. Closeness
34
![Page 35: A Review of Benchmarking Methods G Brown, N Parkin, and N Stuttard, ONS.](https://reader036.fdocuments.us/reader036/viewer/2022081520/5697bf871a28abf838c888ae/html5/thumbnails/35.jpg)
How the methods were compared
1. Failures
2. Verification of benchmarking constraint
3. Preserving change
4. Revisions
5. Smoothness – relative variance of indicator and benchmarked
6. Closeness
35
![Page 36: A Review of Benchmarking Methods G Brown, N Parkin, and N Stuttard, ONS.](https://reader036.fdocuments.us/reader036/viewer/2022081520/5697bf871a28abf838c888ae/html5/thumbnails/36.jpg)
How the methods were compared
1. Failures
2. Verification of benchmarking constraint
3. Preserving change
4. Revisions
5. Smoothness
6. Closeness – between indicator and benchmarked
36
![Page 37: A Review of Benchmarking Methods G Brown, N Parkin, and N Stuttard, ONS.](https://reader036.fdocuments.us/reader036/viewer/2022081520/5697bf871a28abf838c888ae/html5/thumbnails/37.jpg)
How the methods were compared
• For each one of preserving change, revisions, smoothness and closeness, calculate:o For each method, for each time series, for
different lengths of the serieso Rank methods for each series and lengtho Average the ranks over all serieso Plot and compare average ranks by length
37
![Page 38: A Review of Benchmarking Methods G Brown, N Parkin, and N Stuttard, ONS.](https://reader036.fdocuments.us/reader036/viewer/2022081520/5697bf871a28abf838c888ae/html5/thumbnails/38.jpg)
38
![Page 39: A Review of Benchmarking Methods G Brown, N Parkin, and N Stuttard, ONS.](https://reader036.fdocuments.us/reader036/viewer/2022081520/5697bf871a28abf838c888ae/html5/thumbnails/39.jpg)
39
![Page 40: A Review of Benchmarking Methods G Brown, N Parkin, and N Stuttard, ONS.](https://reader036.fdocuments.us/reader036/viewer/2022081520/5697bf871a28abf838c888ae/html5/thumbnails/40.jpg)
Recommended method
• Around 100 plots compared• Judgment made on overall best performing
method• Based on good performance and lack of bad
performance• Recommended method:
Cholette-Dagum (0.8)
40
![Page 41: A Review of Benchmarking Methods G Brown, N Parkin, and N Stuttard, ONS.](https://reader036.fdocuments.us/reader036/viewer/2022081520/5697bf871a28abf838c888ae/html5/thumbnails/41.jpg)
Summary
• Aim: recommend method for benchmarking to ONS and wider GSS
• Update of ONS computer systems prompted examination of methods
• Used several quality measures to rank methods
• Made judgment to combine results from different quality measures
• Recommended: Cholette-Dagum (0.8)
41
![Page 42: A Review of Benchmarking Methods G Brown, N Parkin, and N Stuttard, ONS.](https://reader036.fdocuments.us/reader036/viewer/2022081520/5697bf871a28abf838c888ae/html5/thumbnails/42.jpg)
Any questions?
42