Performance-Tuning for Extract/Transform/Load (ETL) Operations Presented by : Ken Stuber.
-
Upload
adan-beckett -
Category
Documents
-
view
215 -
download
2
Transcript of Performance-Tuning for Extract/Transform/Load (ETL) Operations Presented by : Ken Stuber.
![Page 1: Performance-Tuning for Extract/Transform/Load (ETL) Operations Presented by : Ken Stuber.](https://reader030.fdocuments.us/reader030/viewer/2022013100/5518a215550346881f8b4904/html5/thumbnails/1.jpg)
Performance-Tuning for Extract/Transform/Load (ETL) Operations
Presented by : Ken Stuber
![Page 2: Performance-Tuning for Extract/Transform/Load (ETL) Operations Presented by : Ken Stuber.](https://reader030.fdocuments.us/reader030/viewer/2022013100/5518a215550346881f8b4904/html5/thumbnails/2.jpg)
About Ken
• Data Architect at Jewelers Mutual Insurance• Has worked with SQL Server since 1997• Has supported Data Warehousing on SQL platform for
7+ years• Missed MCM by one lab exam question. Grrr…. Will try
again with whatever new advanced cert MS comes up with…
• @KenStuber• http://dropcleanbuffers.blogspot.com/• [email protected]• http://www.linkedin.com/pub/ken-stuber/9/b7b/2a0/
![Page 3: Performance-Tuning for Extract/Transform/Load (ETL) Operations Presented by : Ken Stuber.](https://reader030.fdocuments.us/reader030/viewer/2022013100/5518a215550346881f8b4904/html5/thumbnails/3.jpg)
First… know your goals
• BASELINE• SLAs• Refresh Frequency• Process Prioritization
![Page 4: Performance-Tuning for Extract/Transform/Load (ETL) Operations Presented by : Ken Stuber.](https://reader030.fdocuments.us/reader030/viewer/2022013100/5518a215550346881f8b4904/html5/thumbnails/4.jpg)
Then … Design for Performance
• Data Grain• Source Data Local• Re-startable
• Parallel processing• Merge or Rebuild Dims?
![Page 5: Performance-Tuning for Extract/Transform/Load (ETL) Operations Presented by : Ken Stuber.](https://reader030.fdocuments.us/reader030/viewer/2022013100/5518a215550346881f8b4904/html5/thumbnails/5.jpg)
Database and Server Settings
• Recovery Model• Isolation• Max Memory• Power Option• MAXDOP
![Page 6: Performance-Tuning for Extract/Transform/Load (ETL) Operations Presented by : Ken Stuber.](https://reader030.fdocuments.us/reader030/viewer/2022013100/5518a215550346881f8b4904/html5/thumbnails/6.jpg)
Getting to your Source Data
If Local…
• Backup & Restore• SAN Snap or Clone• AA Readable 2nd
• Snapshot of Mirror• Replication• Log Shipping• CDC?
• Performance Trade-off of local vs Remote
• Always TEST!
![Page 7: Performance-Tuning for Extract/Transform/Load (ETL) Operations Presented by : Ken Stuber.](https://reader030.fdocuments.us/reader030/viewer/2022013100/5518a215550346881f8b4904/html5/thumbnails/7.jpg)
Bulk Loading Data
• Reduce IO w/ Minimal Logging• Methods• sys.dm_io_virtual_file_stats• TABLOCK vs Trace Flag 610
![Page 8: Performance-Tuning for Extract/Transform/Load (ETL) Operations Presented by : Ken Stuber.](https://reader030.fdocuments.us/reader030/viewer/2022013100/5518a215550346881f8b4904/html5/thumbnails/8.jpg)
![Page 9: Performance-Tuning for Extract/Transform/Load (ETL) Operations Presented by : Ken Stuber.](https://reader030.fdocuments.us/reader030/viewer/2022013100/5518a215550346881f8b4904/html5/thumbnails/9.jpg)
Functions
CPU killer
UDFvs
CLRvs
In-Line
![Page 10: Performance-Tuning for Extract/Transform/Load (ETL) Operations Presented by : Ken Stuber.](https://reader030.fdocuments.us/reader030/viewer/2022013100/5518a215550346881f8b4904/html5/thumbnails/10.jpg)
Indexing
• Source• Staging• Dest.
![Page 11: Performance-Tuning for Extract/Transform/Load (ETL) Operations Presented by : Ken Stuber.](https://reader030.fdocuments.us/reader030/viewer/2022013100/5518a215550346881f8b4904/html5/thumbnails/11.jpg)
Statistics
• Bad Stats = Catastrophic Performance• Update after every build• FULLSCAN if you can afford it
![Page 12: Performance-Tuning for Extract/Transform/Load (ETL) Operations Presented by : Ken Stuber.](https://reader030.fdocuments.us/reader030/viewer/2022013100/5518a215550346881f8b4904/html5/thumbnails/12.jpg)
Compression?
I/O Reductionat cost of CPU
![Page 13: Performance-Tuning for Extract/Transform/Load (ETL) Operations Presented by : Ken Stuber.](https://reader030.fdocuments.us/reader030/viewer/2022013100/5518a215550346881f8b4904/html5/thumbnails/13.jpg)
Partitioning?
• Can Eliminate the L. of E.T.L.• Increases Complexity• Can Benefit Query performance too
![Page 14: Performance-Tuning for Extract/Transform/Load (ETL) Operations Presented by : Ken Stuber.](https://reader030.fdocuments.us/reader030/viewer/2022013100/5518a215550346881f8b4904/html5/thumbnails/14.jpg)
Other Advanced Features
• Hekaton?
• Updatable Columnstore Indexes?
![Page 15: Performance-Tuning for Extract/Transform/Load (ETL) Operations Presented by : Ken Stuber.](https://reader030.fdocuments.us/reader030/viewer/2022013100/5518a215550346881f8b4904/html5/thumbnails/15.jpg)
Hardware Considerations
• More RAM!!!• IO or CPU bound?• Know your IO
Subsystem• SSDs• Multiple files• TempDB local
![Page 16: Performance-Tuning for Extract/Transform/Load (ETL) Operations Presented by : Ken Stuber.](https://reader030.fdocuments.us/reader030/viewer/2022013100/5518a215550346881f8b4904/html5/thumbnails/16.jpg)
Tuning existing ETL
![Page 17: Performance-Tuning for Extract/Transform/Load (ETL) Operations Presented by : Ken Stuber.](https://reader030.fdocuments.us/reader030/viewer/2022013100/5518a215550346881f8b4904/html5/thumbnails/17.jpg)
Resources
• Microsoft CAT Team’s Data Loading Performance Guide
• SQL Magazine: How to find your most expensive queries
• SAN Performance Tuning with SQLIO• Columnstore Insert/Update Performance