Warren Stevens-Baytopp Director YoungBlood Consultants Session Code BIN303.
SQL Server Integration Services Deep Dive Warren Stevens-Baytopp BI Architect - GijimaAst...
-
Upload
bertram-phillips -
Category
Documents
-
view
220 -
download
2
Transcript of SQL Server Integration Services Deep Dive Warren Stevens-Baytopp BI Architect - GijimaAst...
![Page 1: SQL Server Integration Services Deep Dive Warren Stevens-Baytopp BI Architect - GijimaAst Warren.stevens-baytopp@gijima.com.](https://reader031.fdocuments.us/reader031/viewer/2022032309/56649d215503460f949f5c3c/html5/thumbnails/1.jpg)
SQL Server Integration Services Deep Dive
Warren Stevens-BaytoppBI Architect - GijimaAst
![Page 2: SQL Server Integration Services Deep Dive Warren Stevens-Baytopp BI Architect - GijimaAst Warren.stevens-baytopp@gijima.com.](https://reader031.fdocuments.us/reader031/viewer/2022032309/56649d215503460f949f5c3c/html5/thumbnails/2.jpg)
• Merge data from heterogeneous data stores: Text files Mainframes Spreadsheets Multiple RDBMS
• Refresh data in data warehouses and data marts
• Cleanse data before loading to remove errors
• High-speed load of data into online transaction processing (OLTP) and online analytical processing (OLAP) databases
• Send status notifications on success/failure
• Build BI into a data transformation process without the need for redundant staging environments
• Automate data-administrative functions
Integration Services Why ETL Matters
![Page 3: SQL Server Integration Services Deep Dive Warren Stevens-Baytopp BI Architect - GijimaAst Warren.stevens-baytopp@gijima.com.](https://reader031.fdocuments.us/reader031/viewer/2022032309/56649d215503460f949f5c3c/html5/thumbnails/3.jpg)
Data sources can be diverse, including custom or scripted adapters Transformation components shape and modify data in many ways. Data is routed by rules or error conditions for cleansing and conforming. Flows can be as complex as your business rules, but highly concurrent. And finally data can be loaded in parallel to many varied destinations.
SSIS Overview
![Page 4: SQL Server Integration Services Deep Dive Warren Stevens-Baytopp BI Architect - GijimaAst Warren.stevens-baytopp@gijima.com.](https://reader031.fdocuments.us/reader031/viewer/2022032309/56649d215503460f949f5c3c/html5/thumbnails/4.jpg)
So Where to Now…
• Data Sources– Excel
Common problem - not all data coming through correctly
• By Default Excel will determine the column types based on a “Majority Type” rule.
• Overcome this by forcing a mixed type in the Data connector
![Page 5: SQL Server Integration Services Deep Dive Warren Stevens-Baytopp BI Architect - GijimaAst Warren.stevens-baytopp@gijima.com.](https://reader031.fdocuments.us/reader031/viewer/2022032309/56649d215503460f949f5c3c/html5/thumbnails/5.jpg)
Excel Demo
![Page 6: SQL Server Integration Services Deep Dive Warren Stevens-Baytopp BI Architect - GijimaAst Warren.stevens-baytopp@gijima.com.](https://reader031.fdocuments.us/reader031/viewer/2022032309/56649d215503460f949f5c3c/html5/thumbnails/6.jpg)
Data Sources Continued
• Data Sources– Verifying Connectivity / Availability
ETL Tasks run through some of the steps and then fail on connectivity issues.
• Why Would you want to check for this?• Use scripting task.
![Page 7: SQL Server Integration Services Deep Dive Warren Stevens-Baytopp BI Architect - GijimaAst Warren.stevens-baytopp@gijima.com.](https://reader031.fdocuments.us/reader031/viewer/2022032309/56649d215503460f949f5c3c/html5/thumbnails/7.jpg)
Scripting Demo
![Page 8: SQL Server Integration Services Deep Dive Warren Stevens-Baytopp BI Architect - GijimaAst Warren.stevens-baytopp@gijima.com.](https://reader031.fdocuments.us/reader031/viewer/2022032309/56649d215503460f949f5c3c/html5/thumbnails/8.jpg)
Scripting Demo Lets Test it
![Page 9: SQL Server Integration Services Deep Dive Warren Stevens-Baytopp BI Architect - GijimaAst Warren.stevens-baytopp@gijima.com.](https://reader031.fdocuments.us/reader031/viewer/2022032309/56649d215503460f949f5c3c/html5/thumbnails/9.jpg)
Scripting Demo
![Page 10: SQL Server Integration Services Deep Dive Warren Stevens-Baytopp BI Architect - GijimaAst Warren.stevens-baytopp@gijima.com.](https://reader031.fdocuments.us/reader031/viewer/2022032309/56649d215503460f949f5c3c/html5/thumbnails/10.jpg)
Data Sources Continued
• Data Sources– OLE DB Provider
Selecting Table or View dropdown as a source.
• So what is the problem with this?• Replace with what
– Select * from [TABLENAME] – not much better or is it?
– Select [field list] from [TABLENAME] – resource usage
![Page 11: SQL Server Integration Services Deep Dive Warren Stevens-Baytopp BI Architect - GijimaAst Warren.stevens-baytopp@gijima.com.](https://reader031.fdocuments.us/reader031/viewer/2022032309/56649d215503460f949f5c3c/html5/thumbnails/11.jpg)
• If a table is selected – SSIS issues an OPENROWSET
• If a SQL statement is used– SSIS issues sp_executesql.
OLE DB Provider Demo
![Page 12: SQL Server Integration Services Deep Dive Warren Stevens-Baytopp BI Architect - GijimaAst Warren.stevens-baytopp@gijima.com.](https://reader031.fdocuments.us/reader031/viewer/2022032309/56649d215503460f949f5c3c/html5/thumbnails/12.jpg)
Sourcing Data
• Common Requirement– Get all Data from one table that does not
exist in anotherGet all rows from a staging table where the business key is not in the dimension table• Conventional T-SQL
![Page 13: SQL Server Integration Services Deep Dive Warren Stevens-Baytopp BI Architect - GijimaAst Warren.stevens-baytopp@gijima.com.](https://reader031.fdocuments.us/reader031/viewer/2022032309/56649d215503460f949f5c3c/html5/thumbnails/13.jpg)
INSERT INTO DIM_DATE SELECT s.* FROM STG_DATE s LEFT OUTER JOIN DIM_DATE d ON s.DateID = d.DateID WHERE d.DateID IS NULL
INSERT INTO DIM_DATE SELECT s.* FROM STG_DATE s WHERE DateID NOT IN (SELECT DISTINCT DateID
FROM DIM_DATE d)
Sourcing Data Conventional T-SQL
![Page 14: SQL Server Integration Services Deep Dive Warren Stevens-Baytopp BI Architect - GijimaAst Warren.stevens-baytopp@gijima.com.](https://reader031.fdocuments.us/reader031/viewer/2022032309/56649d215503460f949f5c3c/html5/thumbnails/14.jpg)
Sourcing Data
• Common Requirement– Get all Data from one table that does not
exist in anotherGet all rows from a staging table where the business key is not in the dimension table• Conventional T-SQL• Using SSIS
![Page 15: SQL Server Integration Services Deep Dive Warren Stevens-Baytopp BI Architect - GijimaAst Warren.stevens-baytopp@gijima.com.](https://reader031.fdocuments.us/reader031/viewer/2022032309/56649d215503460f949f5c3c/html5/thumbnails/15.jpg)
Sourcing Data Using SSIS
• Merge Join– Same as first T-SQL Statement– Requires a Sort and Conditional Split
• Lookup– Using the SSIS functionality.– Less Coding– Uses the error output as the valid
records.
• Speed Comparisons
![Page 16: SQL Server Integration Services Deep Dive Warren Stevens-Baytopp BI Architect - GijimaAst Warren.stevens-baytopp@gijima.com.](https://reader031.fdocuments.us/reader031/viewer/2022032309/56649d215503460f949f5c3c/html5/thumbnails/16.jpg)
Lookups
• Exact MatchingWant data that matches a specific field.
– Normal usage of Lookup
• Range ComparisonsWant data that falls between 2 values
– The Caching SQL Statement – Mapping of Parameters
![Page 17: SQL Server Integration Services Deep Dive Warren Stevens-Baytopp BI Architect - GijimaAst Warren.stevens-baytopp@gijima.com.](https://reader031.fdocuments.us/reader031/viewer/2022032309/56649d215503460f949f5c3c/html5/thumbnails/17.jpg)
Lookups – Range Comparisons
![Page 18: SQL Server Integration Services Deep Dive Warren Stevens-Baytopp BI Architect - GijimaAst Warren.stevens-baytopp@gijima.com.](https://reader031.fdocuments.us/reader031/viewer/2022032309/56649d215503460f949f5c3c/html5/thumbnails/18.jpg)
Date and Time Handling
• Date formatting– Construct a date string in the format “YYYYMMDD
HHMISS”
– Get the month name
– Get the formatted month and year in the form - mmm (yyyy)
– Create a file using a date in the form yyyy-mm-dd
– Create a file using a date in the form yyyy-mm-dd for yesterdays date
– A simple yyyymmdd formatted – two ways of doing this
![Page 19: SQL Server Integration Services Deep Dive Warren Stevens-Baytopp BI Architect - GijimaAst Warren.stevens-baytopp@gijima.com.](https://reader031.fdocuments.us/reader031/viewer/2022032309/56649d215503460f949f5c3c/html5/thumbnails/19.jpg)
Date Formatting YYYYMMDD HHMISS
(DT_STR,4,1252)DATEPART("yyyy",OldDate) + RIGHT("0" + (DT_STR,2,1252)DATEPART("mm",OldDate),2) + RIGHT("0" + (DT_STR,2,1252)DATEPART("dd",OldDate),2) + " " +RIGHT("0" + (DT_STR,2,1252)DATEPART("hh",OldDate),2) + RIGHT("0" + (DT_STR,2,1252)DATEPART("mi",OldDate),2) + RIGHT("0" + (DT_STR,2,1252)DATEPART("ss",OldDate),2)
![Page 20: SQL Server Integration Services Deep Dive Warren Stevens-Baytopp BI Architect - GijimaAst Warren.stevens-baytopp@gijima.com.](https://reader031.fdocuments.us/reader031/viewer/2022032309/56649d215503460f949f5c3c/html5/thumbnails/20.jpg)
Date Formatting Month Name
(MONTH(NewDate) == 1 ? "January" : MONTH(NewDate) == 2 ? "February" : MONTH(NewDate) == 3 ? "March" : MONTH(NewDate) == 4 ? "April" : MONTH(NewDate) == 5 ? "May" : MONTH(NewDate) == 6 ? "June" : MONTH(NewDate) == 7 ? "July" : MONTH(NewDate) == 8 ? "August" : MONTH(NewDate) == 9 ? "September" : MONTH(NewDate) == 10 ? "October" : MONTH(NewDate) == 11 ? "November" : MONTH(NewDate) == 12 ? "December" : "InvalidMonth")
![Page 21: SQL Server Integration Services Deep Dive Warren Stevens-Baytopp BI Architect - GijimaAst Warren.stevens-baytopp@gijima.com.](https://reader031.fdocuments.us/reader031/viewer/2022032309/56649d215503460f949f5c3c/html5/thumbnails/21.jpg)
Date Formatting mmm (yyyy)
(MONTH(OldDate) == 1 ? "Jan" : MONTH(OldDate) == 2 ? "Feb" : MONTH(OldDate) == 3 ? "Mar" : MONTH(OldDate) == 4 ? "Apr" : MONTH(OldDate) == 5 ? "May" : MONTH(OldDate) == 6 ? "Jun“ : MONTH(OldDate) == 7 ? "Jul" : MONTH(OldDate) == 8 ? "Aug" : MONTH(OldDate) == 9 ? "Sep" : MONTH(OldDate) == 10 ? "Oct" : MONTH(OldDate) == 11 ? "Nov" : MONTH(OldDate) == 12 ? "Dec" : "ERR") + " (" + DT_WSTR,4)YEAR(OldDate) + ")"
![Page 22: SQL Server Integration Services Deep Dive Warren Stevens-Baytopp BI Architect - GijimaAst Warren.stevens-baytopp@gijima.com.](https://reader031.fdocuments.us/reader031/viewer/2022032309/56649d215503460f949f5c3c/html5/thumbnails/22.jpg)
Date Formatting Text file with YYYY-MM-DD
"C:\\Temp\\ErrorCodes\\" +(DT_WSTR,4)YEAR(NewDate) + "-" +RIGHT("0" + (DT_WSTR,2)MONTH(NewDate), 2) + "-" + RIGHT("0" + (DT_WSTR,2)DAY( NewDate), 2) + ".txt"
![Page 23: SQL Server Integration Services Deep Dive Warren Stevens-Baytopp BI Architect - GijimaAst Warren.stevens-baytopp@gijima.com.](https://reader031.fdocuments.us/reader031/viewer/2022032309/56649d215503460f949f5c3c/html5/thumbnails/23.jpg)
Date Formatting Same but for Yesterday
"C:\\Temp\\ErrorCodes\\" +(DT_WSTR,4)YEAR(DATEADD("dd", -1, OldDate)) + "-" +RIGHT("0" + (DT_WSTR,2)MONTH(DATEADD("dd", -1, OldDate)), 2) + "-”+ RIGHT("0" + (DT_WSTR,2)DAY(DATEADD("dd", -1, OldDate)), 2) + ".txt“
dateadd function:In each portion of the check
![Page 24: SQL Server Integration Services Deep Dive Warren Stevens-Baytopp BI Architect - GijimaAst Warren.stevens-baytopp@gijima.com.](https://reader031.fdocuments.us/reader031/viewer/2022032309/56649d215503460f949f5c3c/html5/thumbnails/24.jpg)
Date Formatting A Simple YYYYMMDD
(DT_WSTR,4)YEAR(OldDate)
+ RIGHT("0" + (DT_WSTR,2)MONTH(OldDate), 2)
+ RIGHT("0" + (DT_WSTR,2)DAY(OldDate), 2)
OR
(DT_WSTR,8) ((YEAR(OldDate) * 10000) + (MONTH(OldDate) * 100) + DAY(OldDate))
![Page 25: SQL Server Integration Services Deep Dive Warren Stevens-Baytopp BI Architect - GijimaAst Warren.stevens-baytopp@gijima.com.](https://reader031.fdocuments.us/reader031/viewer/2022032309/56649d215503460f949f5c3c/html5/thumbnails/25.jpg)
Some Performance Tuning Tips
• Only Select the Columns that you need• Use a SQL Server Destination instead of an OLE DB Destination• If using an OLE DB Destination – use the table or view with fast
load data access mode.• Use standardized naming conventions• Where possible, filter your data in the Source Adapter rather than using a
Conditional Split transform component
• LOOKUP components will generally work quicker than MERGE JOIN components where the 2 can be used for the same task
• Use caching in your LOOKUP components where possible. It makes them quicker. Just watch that you are not grabbing too many resources.
• Use Sequence containers to organize package structure into logical units of work.
![Page 26: SQL Server Integration Services Deep Dive Warren Stevens-Baytopp BI Architect - GijimaAst Warren.stevens-baytopp@gijima.com.](https://reader031.fdocuments.us/reader031/viewer/2022032309/56649d215503460f949f5c3c/html5/thumbnails/26.jpg)
For More Information…
• Visit TechNet at www.microsoft.com/technet/prodtechnol/sql/2005/technologies/ssisvcs.mspx
• Learn more about SSIS on MSDN athttp://msdn2.microsoft.com/en-us/sql/aa336312.aspx
• Great information available at www.sqlis.com• http://download.microsoft.com for Whitepapers and
downloads of custom transformations• Jamie Thomson’s Blog
http://blogs.conchango.com/jamiethomson/default.aspx • Donald Farmer’s Blog
http://sqljunkies.com/WebLog/donald_farmer/default.aspx
![Page 27: SQL Server Integration Services Deep Dive Warren Stevens-Baytopp BI Architect - GijimaAst Warren.stevens-baytopp@gijima.com.](https://reader031.fdocuments.us/reader031/viewer/2022032309/56649d215503460f949f5c3c/html5/thumbnails/27.jpg)
Thank you to our Partners for their support of TechDays
2007