Do you have valid, consistent and accurate data? Consider ...€¦ · Do you have valid, consistent...
Transcript of Do you have valid, consistent and accurate data? Consider ...€¦ · Do you have valid, consistent...
![Page 1: Do you have valid, consistent and accurate data? Consider ...€¦ · Do you have valid, consistent and accurate data? Consider a data quality solution. PROCs functions Bill Fehlner,](https://reader034.fdocuments.us/reader034/viewer/2022052010/602100596775547aac53c3bb/html5/thumbnails/1.jpg)
1
Do you have valid, consistent and accurate data?Consider a data quality solution.
functionsPROCs
Bill Fehlner, Education, SAS416 307-4513 [email protected]
BlueFusionSDK
SASJob
dfPowerStudio
odbcSAS
libraryData base
VaultSAS/ACCESS
Data base
![Page 2: Do you have valid, consistent and accurate data? Consider ...€¦ · Do you have valid, consistent and accurate data? Consider a data quality solution. PROCs functions Bill Fehlner,](https://reader034.fdocuments.us/reader034/viewer/2022052010/602100596775547aac53c3bb/html5/thumbnails/2.jpg)
2
Agenda
Why worry about data quality?
Data samples before and after cleaning.
Interactive Data Cleaning
Data Cleaning with programs
Where to learn more
![Page 3: Do you have valid, consistent and accurate data? Consider ...€¦ · Do you have valid, consistent and accurate data? Consider a data quality solution. PROCs functions Bill Fehlner,](https://reader034.fdocuments.us/reader034/viewer/2022052010/602100596775547aac53c3bb/html5/thumbnails/3.jpg)
3
Why Worry about Data Quality?META Group – Ten to twenty percent of the data
used to build data warehouses is corrupt or incomplete.
Data Warehousing Institute – estimates that data quality problems cost US corporations more than $600 billion per year (2002).
![Page 4: Do you have valid, consistent and accurate data? Consider ...€¦ · Do you have valid, consistent and accurate data? Consider a data quality solution. PROCs functions Bill Fehlner,](https://reader034.fdocuments.us/reader034/viewer/2022052010/602100596775547aac53c3bb/html5/thumbnails/4.jpg)
4
Primary Sources of Data Quality Problems
• Source: The Data Warehousing Institute, Data Quality and the Bottom Line, 2002
![Page 5: Do you have valid, consistent and accurate data? Consider ...€¦ · Do you have valid, consistent and accurate data? Consider a data quality solution. PROCs functions Bill Fehlner,](https://reader034.fdocuments.us/reader034/viewer/2022052010/602100596775547aac53c3bb/html5/thumbnails/5.jpg)
5
Data Quality Characteristics
•Data quality affects several attributes associated with data:
– Accuracy – Is it realistic or believable?
– Consistency – Is it consistently defined andmaintained?
– Validity – Is the data valid, based on business or industry rules and standards?
![Page 6: Do you have valid, consistent and accurate data? Consider ...€¦ · Do you have valid, consistent and accurate data? Consider a data quality solution. PROCs functions Bill Fehlner,](https://reader034.fdocuments.us/reader034/viewer/2022052010/602100596775547aac53c3bb/html5/thumbnails/6.jpg)
6
Agenda
Why worry about data quality?
Data samples before and after cleaning.
Interactive Data Cleaning
Data Cleaning with programs
Where to learn more
![Page 7: Do you have valid, consistent and accurate data? Consider ...€¦ · Do you have valid, consistent and accurate data? Consider a data quality solution. PROCs functions Bill Fehlner,](https://reader034.fdocuments.us/reader034/viewer/2022052010/602100596775547aac53c3bb/html5/thumbnails/7.jpg)
7
Definitely Inconsistent, and not that Accurate
![Page 8: Do you have valid, consistent and accurate data? Consider ...€¦ · Do you have valid, consistent and accurate data? Consider a data quality solution. PROCs functions Bill Fehlner,](https://reader034.fdocuments.us/reader034/viewer/2022052010/602100596775547aac53c3bb/html5/thumbnails/8.jpg)
8
The issues here
Inconsistent use of name prefixes
Inconsistent capitalization
Use of nicknames for given names
Misspelling of last names
Occasional use of middle name
![Page 9: Do you have valid, consistent and accurate data? Consider ...€¦ · Do you have valid, consistent and accurate data? Consider a data quality solution. PROCs functions Bill Fehlner,](https://reader034.fdocuments.us/reader034/viewer/2022052010/602100596775547aac53c3bb/html5/thumbnails/9.jpg)
9
Accurate and Consistent
![Page 10: Do you have valid, consistent and accurate data? Consider ...€¦ · Do you have valid, consistent and accurate data? Consider a data quality solution. PROCs functions Bill Fehlner,](https://reader034.fdocuments.us/reader034/viewer/2022052010/602100596775547aac53c3bb/html5/thumbnails/10.jpg)
10
When name processing is not enough
![Page 11: Do you have valid, consistent and accurate data? Consider ...€¦ · Do you have valid, consistent and accurate data? Consider a data quality solution. PROCs functions Bill Fehlner,](https://reader034.fdocuments.us/reader034/viewer/2022052010/602100596775547aac53c3bb/html5/thumbnails/11.jpg)
11
The issues here
Inconsistent use of name prefixes
Inconsistent capitalization
Use of nicknames for given names
Misspelling of last names
Occasional use of middle name
Some names apparently the same have
different addresses
![Page 12: Do you have valid, consistent and accurate data? Consider ...€¦ · Do you have valid, consistent and accurate data? Consider a data quality solution. PROCs functions Bill Fehlner,](https://reader034.fdocuments.us/reader034/viewer/2022052010/602100596775547aac53c3bb/html5/thumbnails/12.jpg)
12
More issues with addresses
Inconsistent use of punctuation
Inconsistent reference to direction
Inconsistent use of street extensions
Misspelling of street names
![Page 13: Do you have valid, consistent and accurate data? Consider ...€¦ · Do you have valid, consistent and accurate data? Consider a data quality solution. PROCs functions Bill Fehlner,](https://reader034.fdocuments.us/reader034/viewer/2022052010/602100596775547aac53c3bb/html5/thumbnails/13.jpg)
13
Joint processing of name and address
![Page 14: Do you have valid, consistent and accurate data? Consider ...€¦ · Do you have valid, consistent and accurate data? Consider a data quality solution. PROCs functions Bill Fehlner,](https://reader034.fdocuments.us/reader034/viewer/2022052010/602100596775547aac53c3bb/html5/thumbnails/14.jpg)
14
Common Processes in a Data Quality Initiative
Consistency Analysis
Standardization Schemes
Gender Analysis
Entity Analysis
Data Parsing
Data Casing
![Page 15: Do you have valid, consistent and accurate data? Consider ...€¦ · Do you have valid, consistent and accurate data? Consider a data quality solution. PROCs functions Bill Fehlner,](https://reader034.fdocuments.us/reader034/viewer/2022052010/602100596775547aac53c3bb/html5/thumbnails/15.jpg)
15
Tasks simplified by a Data Quality Initiative
Matching rows in multiple tables.
De-duplication of rows in one table.
Address Verification.
![Page 16: Do you have valid, consistent and accurate data? Consider ...€¦ · Do you have valid, consistent and accurate data? Consider a data quality solution. PROCs functions Bill Fehlner,](https://reader034.fdocuments.us/reader034/viewer/2022052010/602100596775547aac53c3bb/html5/thumbnails/16.jpg)
16
Agenda
Why worry about data quality?
Data samples before and after cleaning.
Interactive Data Cleaning
Data Cleaning with programs
Where to learn more
![Page 17: Do you have valid, consistent and accurate data? Consider ...€¦ · Do you have valid, consistent and accurate data? Consider a data quality solution. PROCs functions Bill Fehlner,](https://reader034.fdocuments.us/reader034/viewer/2022052010/602100596775547aac53c3bb/html5/thumbnails/17.jpg)
17
DataFlux Functionality
functionsPROCs
BlueFusionSDK
SASJob
dfPowerStudio
odbcSAS
libraryData base
VaultSAS/ACCESS
Data base
![Page 18: Do you have valid, consistent and accurate data? Consider ...€¦ · Do you have valid, consistent and accurate data? Consider a data quality solution. PROCs functions Bill Fehlner,](https://reader034.fdocuments.us/reader034/viewer/2022052010/602100596775547aac53c3bb/html5/thumbnails/18.jpg)
18
dfPower Studio’s Vault
![Page 19: Do you have valid, consistent and accurate data? Consider ...€¦ · Do you have valid, consistent and accurate data? Consider a data quality solution. PROCs functions Bill Fehlner,](https://reader034.fdocuments.us/reader034/viewer/2022052010/602100596775547aac53c3bb/html5/thumbnails/19.jpg)
19
dfPower Studio’s Vault
![Page 20: Do you have valid, consistent and accurate data? Consider ...€¦ · Do you have valid, consistent and accurate data? Consider a data quality solution. PROCs functions Bill Fehlner,](https://reader034.fdocuments.us/reader034/viewer/2022052010/602100596775547aac53c3bb/html5/thumbnails/20.jpg)
20
dfPower Base - Analysis
![Page 21: Do you have valid, consistent and accurate data? Consider ...€¦ · Do you have valid, consistent and accurate data? Consider a data quality solution. PROCs functions Bill Fehlner,](https://reader034.fdocuments.us/reader034/viewer/2022052010/602100596775547aac53c3bb/html5/thumbnails/21.jpg)
21
dfPower Base - Analysis
![Page 22: Do you have valid, consistent and accurate data? Consider ...€¦ · Do you have valid, consistent and accurate data? Consider a data quality solution. PROCs functions Bill Fehlner,](https://reader034.fdocuments.us/reader034/viewer/2022052010/602100596775547aac53c3bb/html5/thumbnails/22.jpg)
22
dfPower Base - Analysis
![Page 23: Do you have valid, consistent and accurate data? Consider ...€¦ · Do you have valid, consistent and accurate data? Consider a data quality solution. PROCs functions Bill Fehlner,](https://reader034.fdocuments.us/reader034/viewer/2022052010/602100596775547aac53c3bb/html5/thumbnails/23.jpg)
23
dfPower Base – Analysis Report
![Page 24: Do you have valid, consistent and accurate data? Consider ...€¦ · Do you have valid, consistent and accurate data? Consider a data quality solution. PROCs functions Bill Fehlner,](https://reader034.fdocuments.us/reader034/viewer/2022052010/602100596775547aac53c3bb/html5/thumbnails/24.jpg)
24
dfPower Base – Analysis Report
![Page 25: Do you have valid, consistent and accurate data? Consider ...€¦ · Do you have valid, consistent and accurate data? Consider a data quality solution. PROCs functions Bill Fehlner,](https://reader034.fdocuments.us/reader034/viewer/2022052010/602100596775547aac53c3bb/html5/thumbnails/25.jpg)
25
dfPower Studio’s Vault
![Page 26: Do you have valid, consistent and accurate data? Consider ...€¦ · Do you have valid, consistent and accurate data? Consider a data quality solution. PROCs functions Bill Fehlner,](https://reader034.fdocuments.us/reader034/viewer/2022052010/602100596775547aac53c3bb/html5/thumbnails/26.jpg)
26
Agenda
Why worry about data quality?
Data samples before and after cleaning.
Interactive Data Cleaning
Data Cleaning with programs
Where to learn more
![Page 27: Do you have valid, consistent and accurate data? Consider ...€¦ · Do you have valid, consistent and accurate data? Consider a data quality solution. PROCs functions Bill Fehlner,](https://reader034.fdocuments.us/reader034/viewer/2022052010/602100596775547aac53c3bb/html5/thumbnails/27.jpg)
27
SAS – Proc Scheme
![Page 28: Do you have valid, consistent and accurate data? Consider ...€¦ · Do you have valid, consistent and accurate data? Consider a data quality solution. PROCs functions Bill Fehlner,](https://reader034.fdocuments.us/reader034/viewer/2022052010/602100596775547aac53c3bb/html5/thumbnails/28.jpg)
28
SAS – DQ functions
![Page 29: Do you have valid, consistent and accurate data? Consider ...€¦ · Do you have valid, consistent and accurate data? Consider a data quality solution. PROCs functions Bill Fehlner,](https://reader034.fdocuments.us/reader034/viewer/2022052010/602100596775547aac53c3bb/html5/thumbnails/29.jpg)
29
SAS – Proc Matchprocess name only
![Page 30: Do you have valid, consistent and accurate data? Consider ...€¦ · Do you have valid, consistent and accurate data? Consider a data quality solution. PROCs functions Bill Fehlner,](https://reader034.fdocuments.us/reader034/viewer/2022052010/602100596775547aac53c3bb/html5/thumbnails/30.jpg)
30
Match Codes – name only
![Page 31: Do you have valid, consistent and accurate data? Consider ...€¦ · Do you have valid, consistent and accurate data? Consider a data quality solution. PROCs functions Bill Fehlner,](https://reader034.fdocuments.us/reader034/viewer/2022052010/602100596775547aac53c3bb/html5/thumbnails/31.jpg)
31
SAS – Proc Matchprocess name and address
![Page 32: Do you have valid, consistent and accurate data? Consider ...€¦ · Do you have valid, consistent and accurate data? Consider a data quality solution. PROCs functions Bill Fehlner,](https://reader034.fdocuments.us/reader034/viewer/2022052010/602100596775547aac53c3bb/html5/thumbnails/32.jpg)
32
Match Codes – name and address
![Page 33: Do you have valid, consistent and accurate data? Consider ...€¦ · Do you have valid, consistent and accurate data? Consider a data quality solution. PROCs functions Bill Fehlner,](https://reader034.fdocuments.us/reader034/viewer/2022052010/602100596775547aac53c3bb/html5/thumbnails/33.jpg)
33
DataFlux Functionality
functionsPROCs
BlueFusionSDK
SASJob
dfPowerStudio
odbcSAS
libraryData base
VaultSAS/ACCESS
Data base
![Page 34: Do you have valid, consistent and accurate data? Consider ...€¦ · Do you have valid, consistent and accurate data? Consider a data quality solution. PROCs functions Bill Fehlner,](https://reader034.fdocuments.us/reader034/viewer/2022052010/602100596775547aac53c3bb/html5/thumbnails/34.jpg)
34
How to Learn More
• Instructor based training:
– http://support.sas.com /training/Canada
– “SAS Data Quality-Cleanse”, a two-day course, starting on April 22 in Toronto
![Page 35: Do you have valid, consistent and accurate data? Consider ...€¦ · Do you have valid, consistent and accurate data? Consider a data quality solution. PROCs functions Bill Fehlner,](https://reader034.fdocuments.us/reader034/viewer/2022052010/602100596775547aac53c3bb/html5/thumbnails/35.jpg)
35
How to Learn More
• Technical Support
– http://support.sas.com/rnd/warehousing/quality