Outlier Treatment in HCSO
Present and future
Outline
• Outlier detection – types, editing, estimation
• Description of the current method
• Alternatives
• Future work
• Introduction of a new tool: R and Rstudio
UNECE Statistical Data Editing 2014 2
Outlier detection and treatmentPurpose of outlier detection
Identify errors
Estimation
Editing
• Representative outliers• Non Representative outliers
• Decreasing weights• Changing the values• Using robust estimations
Source: MEMOBUST
UNECE Statistical Data Editing 2014 3
Monthly Survey of Manufacturing
• Take-all part• Survey part:
• less than 50 employees (and more than 5, because the smallest businesses are not in the scope of the survey).
• The sampling frame is based on the Register of Enterprises (~10 thousand units)
• The sampling ratio is about 15%• Stratified sample (a lot of NACE categories, categories
of the number of employees, and two territorial strata: the capital and everything else). (Telegdi 2004.)
UNECE Statistical Data Editing 2014 4
Monthly Survey of Manufacturing: data
Distribution of some variables• Skewed distribution• Visible outliers
UNECE Statistical Data Editing 2014 5
Current method of outlier detection
• The aim of the outlier treatment is improving the estimation. (Csereháti 2004.)
• Steps of the method:1) Computing the outlier indicators
2) Manual outlier detection by the methodologist/expert
3) Transfer of the result to the subject matter statistician
4) Discussion of the result by the subject matter statistician (possible modifications), resembles to the process of selective editing
UNECE Statistical Data Editing 2014 6
Outlier indicators
• LNSQRT: main indicator
• Grubbs crit. value
• Standardized value of the variables
• SQUARED: identifying highest values
• MEANX is the ratio of the observed value of
the unit and the weighted mean of the
stratum without this unit value.
• VALOUT indicator shows the difference
between the estimation of the total with and
without the given value in a given stratum.
𝐿𝑁𝑆𝑄𝑅𝑇 𝑗𝑖=𝐿𝑛𝑌 𝑗𝑖 ∙√ 𝑆𝑇𝐴𝑁𝐷𝐴𝑅𝐷 𝑗𝑖
𝐺𝑐𝑟𝑖𝑡 , 𝑗
𝑆𝑄𝑈𝐴𝑅𝐸𝐷 𝑗𝑖=𝑌 𝑗𝑖∙ (𝑆𝑇𝐴𝑁𝐷𝐴𝑅𝐷 𝑗𝑖
𝐺𝑐𝑟𝑖𝑡 , 𝑗)2
𝑀𝐸𝐴𝑁𝑋 𝑗𝑖=𝑌 𝑗𝑖
𝑀𝐸𝐴𝑁 𝑗 ∙𝑁 𝑗−𝑌 𝑗𝑖(𝑁 𝑗−1 )
𝑉𝐴𝐿𝑂𝑈𝑇 𝑗𝑖=𝑁 𝑗−𝑛 𝑗
𝑛 𝑗−1 (𝑌 𝑗𝑖−𝑀𝐸𝐴𝑁 𝑗 ∙𝑁 𝑗
𝑛 𝑗)
𝑃𝑉𝐴𝐿𝑂𝑈𝑇 𝑗𝑖=𝑉𝐴𝐿𝑂𝑈𝑇 𝑗𝑖
𝑁 𝑗2 ∙𝑀𝐸𝐴𝑁 𝑗
∙𝑛 𝑗
UNECE Statistical Data Editing 2014 7
The main indicator: LNSQRT
UNECE Statistical Data Editing 2014 8
Outlier treatment
• Weight trimming: weights of the outliers are changed to 1• Number of outliers: avg. 2% of the cases• Change in the estimates:
• Mean: -15% (in avarage)• Variance: serious decrease
UNECE Statistical Data Editing 2014 9
Alternative methods
• One dimensional methods• Median absolute deviation• Custom indicator: share in total• Quantile
Disadvantage: applying to many variables • Multidimensional method:
• Mahalanobis distance based outlier detection
UNECE Statistical Data Editing 2014 10
Share in total, a custom indicator
• To consider the individual value and the size of the stratum in the same formula
• inspired by the current indicators• The possible outlier:
• shares a considerably great amount of the total• In a big stratum
• The indicator computed for each stratum
UNECE Statistical Data Editing 2014 11
Results
• Quantile method• Threshold 99%• The method can identify almost the same
outliers as the current one.• Easy to implement
• MAD• Problem of the k (threshold)• Too many cases were selected
UNECE Statistical Data Editing 2014 12
Results (2)
• Share in total• Threshold value: 0.5• Smaller number of outliers
• Mahalanobis distance• We used the robust Mahalanobis distance• 3 key variables (Total revenue etc.)
• These are not involved in the current method• avoiding missing values
• Similar results (2/3 of the current outliers are detected)
UNECE Statistical Data Editing 2014 13
UNECE Statistical Data Editing 2014 14
Future plans
• Development of methodology:– More analysis of the effect on estimates– Winsorization
• Development of the process– Automation and reproducibility– More informative report on the process, to help
better understand and analyse the process steps
UNECE Statistical Data Editing 2014 15
Experimental tools
• Outlier treatment is separated from other steps of data
process, belongs to the methodology
• Possible new tool: R (with Rstudio)
• Advantage: ease of development
• Ready-to-use functions for outlier detection
• Disadvantage: need of „expert” user, not a usual tool
UNECE Statistical Data Editing 2014 16
Thank you for your attention!
UNECE Statistical Data Editing 2014 17
Top Related