Validating and Cleaning Data in EG - SAS Group Presentatio… · Title: Validating and Cleaning...
22
Copyright © 2013, SAS Institute Inc. All rights reserved. VALIDATING AND CLEANING DATA IN ENTERPRISE GUIDE ___________________________________________ Judy Orr Lawrence – SAS Training Specialist Health Users Group (HUG)
Transcript of Validating and Cleaning Data in EG - SAS Group Presentatio… · Title: Validating and Cleaning...
Copyr i g ht © 2013, SAS Ins t i tu t e Inc . A l l r ights reser ve d .
VALIDATING AND CLEANING DATA
IN ENTERPRISE GUIDE
___________________________________________
Judy Orr Lawrence – SAS Training Specialist
Health Users Group (HUG)
Copyr i g ht © 2013, SAS Ins t i tu t e Inc . A l l r ights reser ve d .
AGENDA
Why Validate Data?
Validating Structure, Content, Quality, Quantity
Validating Tasks
Cleaning Techniques
Copyr i g ht © 2013, SAS Ins t i tu t e Inc . A l l r ights reser ve d .
WHY VALIDATE?
Accurate conclusions and predictions
Confidence in the results
Can withstand tests over time
RELIABLE DATA
Copyr i g ht © 2013, SAS Ins t i tu t e Inc . A l l r ights reser ve d .
VALIDATING DATA
Structure and Content • Do variables have the correct name, type, length?
• Should data be sorted or indexed?
Quality • Was source data entered accurately?
• Are there duplications, typo errors, missing data?
Quantity • Are numeric amounts within an acceptable range?
Copyr i g ht © 2013, SAS Ins t i tu t e Inc . A l l r ights reser ve d .
VALIDATING TASKS
Data Set Attributes
Characterize Data
One-Way Frequency
Summary Statistics
Distribution Analysis
Query Builder
Copyr i g ht © 2013, SAS Ins t i tu t e Inc . A l l r ights reser ve d .
CLEANING
TECHNIQUES
Sort Task
Query Builder Recode / Compute
Format Task
Interactive
Copyr i g ht © 2013, SAS Ins t i tu t e Inc . A l l r ights reser ve d .
VALIDATING
STRUCTURE
DATA SET ATTRIBUTES TASK
Copyr i g ht © 2013, SAS Ins t i tu t e Inc . A l l r ights reser ve d .
VALIDATING
CONTENT
CHARACTERIZE DATA TASK
Accuracy,
Duplicates,
Missing
Graphs quickly
identify errors
Are Date ranges valid?
Copyr i g ht © 2013, SAS Ins t i tu t e Inc . A l l r ights reser ve d .
VALIDATING
QUALITY / ACCURACY
FREQUENCY TASK
Are data values
accurate and
not missing?
Copyr i g ht © 2013, SAS Ins t i tu t e Inc . A l l r ights reser ve d .
VALIDATING
UNIQUE VALUES
FREQUENCY TASK
Are unique
variables truly
unique?
Copyr i g ht © 2013, SAS Ins t i tu t e Inc . A l l r ights reser ve d .
VALIDATING
DATES
QUERY BUILDER - FILTER
Are date values within
an acceptable range?
Copyr i g ht © 2013, SAS Ins t i tu t e Inc . A l l r ights reser ve d .
VALIDATING
NUMERIC RANGES
SUMMARY STATISTICS / DISTRIBUTION ANALYSIS TASK
Are numeric variables within
an acceptable range?
Are there missing values?
Copyr i g ht © 2013, SAS Ins t i tu t e Inc . A l l r ights reser ve d .
CLEANING
ELIMINATING
DUPLICATES
SORT TASK
Eliminate Duplicates
based on specific
variables
Eliminate Duplicates
based on entire row
Copyr i g ht © 2013, SAS Ins t i tu t e Inc . A l l r ights reser ve d .
CLEANING
CHANGE
VARIABLE TYPE
QUERY BUILDER
PUT Function
NewVar = PUT(OldVar, Numeric_Format)
Numeric to Character
Character to Numeric
INPUT Function
NewVar = INPUT(OldVar, Numeric_INFormat)
Create a new column
with corrected values
Copyr i g ht © 2013, SAS Ins t i tu t e Inc . A l l r ights reser ve d .
CLEANING
RECODING ERRORS QUERY BUILDER
Create a new column
with corrected values
Copyr i g ht © 2013, SAS Ins t i tu t e Inc . A l l r ights reser ve d .
CLEANING
RECODING ERRORS FORMAT TASK
Apply a Format to
existing values
Copyr i g ht © 2013, SAS Ins t i tu t e Inc . A l l r ights reser ve d .
CLEANING
INTERACTIVE CORRECTIONS
Unprotect the source data set and type
corrected values into the table
Copyr i g ht © 2013, SAS Ins t i tu t e Inc . A l l r ights reser ve d .
DEMO
Copyr i g ht © 2013, SAS Ins t i tu t e Inc . A l l r ights reser ve d .
DEMO
Copyr i g ht © 2013, SAS Ins t i tu t e Inc . A l l r ights reser ve d .
SUMMARY
Copyr i g ht © 2013, SAS Ins t i tu t e Inc . A l l r ights reser ve d .
TRAINING & BOOKS HTTP://SUPPORT.SAS.COM/TRAINING/CANADA/
Copyr i g ht © 2013, SAS Ins t i tu t e Inc . A l l r ights reser ve d . sas.com
THANK YOU
___________________________
