Data Warehouse in the Cloud Marketing or Reality? SQL_2013_DWH_IN_THE... · Data Warehouse in the...
Transcript of Data Warehouse in the Cloud Marketing or Reality? SQL_2013_DWH_IN_THE... · Data Warehouse in the...
![Page 1: Data Warehouse in the Cloud Marketing or Reality? SQL_2013_DWH_IN_THE... · Data Warehouse in the Cloud – Marketing or Reality? Alexei Khalyako Sr. Program Manager ... –Iaas SQL](https://reader033.fdocuments.us/reader033/viewer/2022042223/5ec98a04b83f5f77ec2d4603/html5/thumbnails/1.jpg)
![Page 2: Data Warehouse in the Cloud Marketing or Reality? SQL_2013_DWH_IN_THE... · Data Warehouse in the Cloud – Marketing or Reality? Alexei Khalyako Sr. Program Manager ... –Iaas SQL](https://reader033.fdocuments.us/reader033/viewer/2022042223/5ec98a04b83f5f77ec2d4603/html5/thumbnails/2.jpg)
Data Warehouse in the Cloud –
Marketing or Reality?
Alexei Khalyako
Sr. Program Manager
Windows Azure Customer Advisory Team
![Page 3: Data Warehouse in the Cloud Marketing or Reality? SQL_2013_DWH_IN_THE... · Data Warehouse in the Cloud – Marketing or Reality? Alexei Khalyako Sr. Program Manager ... –Iaas SQL](https://reader033.fdocuments.us/reader033/viewer/2022042223/5ec98a04b83f5f77ec2d4603/html5/thumbnails/3.jpg)
Data Warehouse we used to know
• High-End workload
• High-End hardware
• Special know-how
![Page 4: Data Warehouse in the Cloud Marketing or Reality? SQL_2013_DWH_IN_THE... · Data Warehouse in the Cloud – Marketing or Reality? Alexei Khalyako Sr. Program Manager ... –Iaas SQL](https://reader033.fdocuments.us/reader033/viewer/2022042223/5ec98a04b83f5f77ec2d4603/html5/thumbnails/4.jpg)
*BeyeNetwork Big Data research
Reality is
• Thousands of departmental level DW
• Relatively low perfSLA
![Page 5: Data Warehouse in the Cloud Marketing or Reality? SQL_2013_DWH_IN_THE... · Data Warehouse in the Cloud – Marketing or Reality? Alexei Khalyako Sr. Program Manager ... –Iaas SQL](https://reader033.fdocuments.us/reader033/viewer/2022042223/5ec98a04b83f5f77ec2d4603/html5/thumbnails/5.jpg)
New BI demands
• Utilize external data sources
• Non Structured Data
• Origin is in the Cloud
*BeyeNetwork Big Data research
![Page 6: Data Warehouse in the Cloud Marketing or Reality? SQL_2013_DWH_IN_THE... · Data Warehouse in the Cloud – Marketing or Reality? Alexei Khalyako Sr. Program Manager ... –Iaas SQL](https://reader033.fdocuments.us/reader033/viewer/2022042223/5ec98a04b83f5f77ec2d4603/html5/thumbnails/6.jpg)
New opportunity
• Platform is there
– Iaas SQL VM
– Paas SQL Azure DB
• “Closer” to data
• Less administrative overhead
• Lower initial and TCO cost
![Page 7: Data Warehouse in the Cloud Marketing or Reality? SQL_2013_DWH_IN_THE... · Data Warehouse in the Cloud – Marketing or Reality? Alexei Khalyako Sr. Program Manager ... –Iaas SQL](https://reader033.fdocuments.us/reader033/viewer/2022042223/5ec98a04b83f5f77ec2d4603/html5/thumbnails/7.jpg)
SQL Server Data Warehousing in Windows Azure Virtual Machines
• Inspired by the Fast Track Reference Architecture guide
• Based on the High Memory images
• Up to 1TB
• MSDN: SQL Server Data Warehousing in Windows Azure Virtual Machines
![Page 8: Data Warehouse in the Cloud Marketing or Reality? SQL_2013_DWH_IN_THE... · Data Warehouse in the Cloud – Marketing or Reality? Alexei Khalyako Sr. Program Manager ... –Iaas SQL](https://reader033.fdocuments.us/reader033/viewer/2022042223/5ec98a04b83f5f77ec2d4603/html5/thumbnails/8.jpg)
High Memory VM in Azure
![Page 9: Data Warehouse in the Cloud Marketing or Reality? SQL_2013_DWH_IN_THE... · Data Warehouse in the Cloud – Marketing or Reality? Alexei Khalyako Sr. Program Manager ... –Iaas SQL](https://reader033.fdocuments.us/reader033/viewer/2022042223/5ec98a04b83f5f77ec2d4603/html5/thumbnails/9.jpg)
How to deploy
• Powershell script • Windows Azure Gallery
![Page 10: Data Warehouse in the Cloud Marketing or Reality? SQL_2013_DWH_IN_THE... · Data Warehouse in the Cloud – Marketing or Reality? Alexei Khalyako Sr. Program Manager ... –Iaas SQL](https://reader033.fdocuments.us/reader033/viewer/2022042223/5ec98a04b83f5f77ec2d4603/html5/thumbnails/10.jpg)
The Azure Data Warehouse under the hood
![Page 11: Data Warehouse in the Cloud Marketing or Reality? SQL_2013_DWH_IN_THE... · Data Warehouse in the Cloud – Marketing or Reality? Alexei Khalyako Sr. Program Manager ... –Iaas SQL](https://reader033.fdocuments.us/reader033/viewer/2022042223/5ec98a04b83f5f77ec2d4603/html5/thumbnails/11.jpg)
Data Warehouse Lifecycle• Thoughts on the architecture• Creating DB• Connectivity• Populating Database
– Initial data loading OR– Backup/Restore– Incremental data loading– Compression
• Query performance
![Page 12: Data Warehouse in the Cloud Marketing or Reality? SQL_2013_DWH_IN_THE... · Data Warehouse in the Cloud – Marketing or Reality? Alexei Khalyako Sr. Program Manager ... –Iaas SQL](https://reader033.fdocuments.us/reader033/viewer/2022042223/5ec98a04b83f5f77ec2d4603/html5/thumbnails/12.jpg)
Thoughts on the architecture
• Data Loading– Minimize Log impact– Scale loading streams– Do not invent the wheel and follow
the Data loading Performance guide
• Query Performance
! Do not invent the wheel and follow the Data loading
Performance guide
![Page 13: Data Warehouse in the Cloud Marketing or Reality? SQL_2013_DWH_IN_THE... · Data Warehouse in the Cloud – Marketing or Reality? Alexei Khalyako Sr. Program Manager ... –Iaas SQL](https://reader033.fdocuments.us/reader033/viewer/2022042223/5ec98a04b83f5f77ec2d4603/html5/thumbnails/13.jpg)
Windows Azure VM Architecture
• Disks implemented as a shared multi-tenant service
• Built-in triple redundancy, optional geo-redundancy
• Performance less predictable than on-prem
Host machines, storage services, network bandwidth shared between subscribersPerf can depend on where and when VM is provisionedSubject to maintenance operationsGranular control & configurability vs. cost, simplicity, out of box redundancy
Storage Stamp
Stream Layer
Partition Layer
Front-ends
LB
Intra-stamp replication
Stream Layer
Partition Layer
Front-ends
LB
Intra-stamp replication
Storage Stamp
Geo-replication
Storage Location Service
![Page 14: Data Warehouse in the Cloud Marketing or Reality? SQL_2013_DWH_IN_THE... · Data Warehouse in the Cloud – Marketing or Reality? Alexei Khalyako Sr. Program Manager ... –Iaas SQL](https://reader033.fdocuments.us/reader033/viewer/2022042223/5ec98a04b83f5f77ec2d4603/html5/thumbnails/14.jpg)
Tweaks to improve IO Subsystem • Database file initialization
– GPEdit.msc
• Data file placement– SQL Striping for User Data
and TempDB– Aggregated throughput– Set the size and data grow options wisely
Primary
Log
*You may do it differently. Then• Create 350GB DB took ~3 hours
![Page 15: Data Warehouse in the Cloud Marketing or Reality? SQL_2013_DWH_IN_THE... · Data Warehouse in the Cloud – Marketing or Reality? Alexei Khalyako Sr. Program Manager ... –Iaas SQL](https://reader033.fdocuments.us/reader033/viewer/2022042223/5ec98a04b83f5f77ec2d4603/html5/thumbnails/15.jpg)
Scaling IO OptionsWindows Storage Spaces
• Log drive
• Not clear support story
SQL Data Files
• Spread File Group over all drives
![Page 16: Data Warehouse in the Cloud Marketing or Reality? SQL_2013_DWH_IN_THE... · Data Warehouse in the Cloud – Marketing or Reality? Alexei Khalyako Sr. Program Manager ... –Iaas SQL](https://reader033.fdocuments.us/reader033/viewer/2022042223/5ec98a04b83f5f77ec2d4603/html5/thumbnails/16.jpg)
Scaling IO OptionsData disk (read) LOG (write)
SQLIO Single Data Disk(64K)
SQLIO Windows Storage SpacesX3 Disks (64K)
SQLIO SQL Striping x3 Disk(64K) *
CUMULATIVE DATA:throughput metrics:
IOs/sec: 1215.13MBs/sec: 75.94
CUMULATIVE DATA:throughput metrics:
IOs/sec: 2677.69MBs/sec: 167.35
CUMULATIVE DATA:throughput metrics:
IOs/sec: 2742.22MBs/sec: 171.38
SQLIO Single Data Disk(256K)
SQLIO Windows Storage SpacesX3 Disks (256K)
SQLIO SQL Striping x3 Disk(256K)
CUMULATIVE DATA:
throughput metrics:IOs/sec: 288MBs/sec: 71.98
CUMULATIVE DATA:throughput metrics:
IOs/sec: 640.87MBs/sec: 160.21
CUMULATIVE DATA:throughput metrics:
IOs/sec: 599.91MBs/sec: 149.97
* But we can access one file at the time!
![Page 17: Data Warehouse in the Cloud Marketing or Reality? SQL_2013_DWH_IN_THE... · Data Warehouse in the Cloud – Marketing or Reality? Alexei Khalyako Sr. Program Manager ... –Iaas SQL](https://reader033.fdocuments.us/reader033/viewer/2022042223/5ec98a04b83f5f77ec2d4603/html5/thumbnails/17.jpg)
Connectivity Options
Windows Azure VM End Points Point-to-Site /Site-to-Site
*Other options are also available ( FTP)
![Page 18: Data Warehouse in the Cloud Marketing or Reality? SQL_2013_DWH_IN_THE... · Data Warehouse in the Cloud – Marketing or Reality? Alexei Khalyako Sr. Program Manager ... –Iaas SQL](https://reader033.fdocuments.us/reader033/viewer/2022042223/5ec98a04b83f5f77ec2d4603/html5/thumbnails/18.jpg)
What and how we tested
![Page 19: Data Warehouse in the Cloud Marketing or Reality? SQL_2013_DWH_IN_THE... · Data Warehouse in the Cloud – Marketing or Reality? Alexei Khalyako Sr. Program Manager ... –Iaas SQL](https://reader033.fdocuments.us/reader033/viewer/2022042223/5ec98a04b83f5f77ec2d4603/html5/thumbnails/19.jpg)
Getting initial data
• Copy backup to the Data Disks
• Backup/Restore to/from URL
• ETL to the new DB
![Page 20: Data Warehouse in the Cloud Marketing or Reality? SQL_2013_DWH_IN_THE... · Data Warehouse in the Cloud – Marketing or Reality? Alexei Khalyako Sr. Program Manager ... –Iaas SQL](https://reader033.fdocuments.us/reader033/viewer/2022042223/5ec98a04b83f5f77ec2d4603/html5/thumbnails/20.jpg)
URL is fast!Backup to the Local Data Disk
Backup to the URL
DB Size Time Speed
244GB 3 hours 22,978 MB/sec
DB Size Time Speed
244GB 46 min 90,667 MB/sec
![Page 21: Data Warehouse in the Cloud Marketing or Reality? SQL_2013_DWH_IN_THE... · Data Warehouse in the Cloud – Marketing or Reality? Alexei Khalyako Sr. Program Manager ... –Iaas SQL](https://reader033.fdocuments.us/reader033/viewer/2022042223/5ec98a04b83f5f77ec2d4603/html5/thumbnails/21.jpg)
DB and Data Loading
? Data loading ? Tools (BCP, SSIS..)
? Time SLA
? Query Performance? Indexing strategy
? Sizing? Compression
![Page 22: Data Warehouse in the Cloud Marketing or Reality? SQL_2013_DWH_IN_THE... · Data Warehouse in the Cloud – Marketing or Reality? Alexei Khalyako Sr. Program Manager ... –Iaas SQL](https://reader033.fdocuments.us/reader033/viewer/2022042223/5ec98a04b83f5f77ec2d4603/html5/thumbnails/22.jpg)
Loading Data in Azure • Smaller batches (10K -15K rows)• Retry logic• Network latency is high• Parallel loading!!
Start with: SSIS for Hybrid Data MovementSSIS Performance and Operational guide
![Page 23: Data Warehouse in the Cloud Marketing or Reality? SQL_2013_DWH_IN_THE... · Data Warehouse in the Cloud – Marketing or Reality? Alexei Khalyako Sr. Program Manager ... –Iaas SQL](https://reader033.fdocuments.us/reader033/viewer/2022042223/5ec98a04b83f5f77ec2d4603/html5/thumbnails/23.jpg)
Baseline
• Understand Data Sources performance
– Flat File in Azure VM ~60 MB/sec /reads
– SQLIO shows the max throughput of the IO subsystem on the DB side
– App performance can be different
![Page 24: Data Warehouse in the Cloud Marketing or Reality? SQL_2013_DWH_IN_THE... · Data Warehouse in the Cloud – Marketing or Reality? Alexei Khalyako Sr. Program Manager ... –Iaas SQL](https://reader033.fdocuments.us/reader033/viewer/2022042223/5ec98a04b83f5f77ec2d4603/html5/thumbnails/24.jpg)
Parallel LoadingFlat fileMax 60 MB/sec
Flat fileMax 60 MB/sec
8 destinations to keep all CPU busy on the DW site
Mod(7) function
![Page 25: Data Warehouse in the Cloud Marketing or Reality? SQL_2013_DWH_IN_THE... · Data Warehouse in the Cloud – Marketing or Reality? Alexei Khalyako Sr. Program Manager ... –Iaas SQL](https://reader033.fdocuments.us/reader033/viewer/2022042223/5ec98a04b83f5f77ec2d4603/html5/thumbnails/25.jpg)
Begin to load
![Page 26: Data Warehouse in the Cloud Marketing or Reality? SQL_2013_DWH_IN_THE... · Data Warehouse in the Cloud – Marketing or Reality? Alexei Khalyako Sr. Program Manager ... –Iaas SQL](https://reader033.fdocuments.us/reader033/viewer/2022042223/5ec98a04b83f5f77ec2d4603/html5/thumbnails/26.jpg)
Monitoring Loading Performance
You will be followed by TOP waits:ASYNC_NETWORK_IOPAGEIOLATCH_EXWRITELOGPAGEIOLATCH_UPSOS_SCHEDULER_YIELDPAGEIOLATCH_SHPAGELATCH_UPPREEMPTIVE_OLEDBOPS
Network IO
Disk IO
CPU
![Page 27: Data Warehouse in the Cloud Marketing or Reality? SQL_2013_DWH_IN_THE... · Data Warehouse in the Cloud – Marketing or Reality? Alexei Khalyako Sr. Program Manager ... –Iaas SQL](https://reader033.fdocuments.us/reader033/viewer/2022042223/5ec98a04b83f5f77ec2d4603/html5/thumbnails/27.jpg)
Loading: table optionsHeap
• 780 772 573 rows
• Elapsed time: 01:06:15.313
Heap compressed
• 780 772 573 rows
• Elapsed time: 05:12:06.094
![Page 28: Data Warehouse in the Cloud Marketing or Reality? SQL_2013_DWH_IN_THE... · Data Warehouse in the Cloud – Marketing or Reality? Alexei Khalyako Sr. Program Manager ... –Iaas SQL](https://reader033.fdocuments.us/reader033/viewer/2022042223/5ec98a04b83f5f77ec2d4603/html5/thumbnails/28.jpg)
Loading: table optionsHEAP Clustered Index
• Sort!
• Elapsed time: 01:20:12.547
• 780 772 573 rows
• Elapsed time: 01:06:15.313
![Page 29: Data Warehouse in the Cloud Marketing or Reality? SQL_2013_DWH_IN_THE... · Data Warehouse in the Cloud – Marketing or Reality? Alexei Khalyako Sr. Program Manager ... –Iaas SQL](https://reader033.fdocuments.us/reader033/viewer/2022042223/5ec98a04b83f5f77ec2d4603/html5/thumbnails/29.jpg)
Query Performance
• Heap
• Primary Key/Clustered Index
• Compression
![Page 30: Data Warehouse in the Cloud Marketing or Reality? SQL_2013_DWH_IN_THE... · Data Warehouse in the Cloud – Marketing or Reality? Alexei Khalyako Sr. Program Manager ... –Iaas SQL](https://reader033.fdocuments.us/reader033/viewer/2022042223/5ec98a04b83f5f77ec2d4603/html5/thumbnails/30.jpg)
Query performance: results
0.00
2000.00
4000.00
6000.00
8000.00
10000.00
12000.00
Qu
ery
Exec
uti
on
tim
e
Query Type
Heap
Clustered Index
Clustered Index Compressed
![Page 31: Data Warehouse in the Cloud Marketing or Reality? SQL_2013_DWH_IN_THE... · Data Warehouse in the Cloud – Marketing or Reality? Alexei Khalyako Sr. Program Manager ... –Iaas SQL](https://reader033.fdocuments.us/reader033/viewer/2022042223/5ec98a04b83f5f77ec2d4603/html5/thumbnails/31.jpg)
Please welcome on stage SQL 2014
![Page 32: Data Warehouse in the Cloud Marketing or Reality? SQL_2013_DWH_IN_THE... · Data Warehouse in the Cloud – Marketing or Reality? Alexei Khalyako Sr. Program Manager ... –Iaas SQL](https://reader033.fdocuments.us/reader033/viewer/2022042223/5ec98a04b83f5f77ec2d4603/html5/thumbnails/32.jpg)
What’s new?
• Data files to BLOBs
• Updateable Clustered Column Store index
![Page 33: Data Warehouse in the Cloud Marketing or Reality? SQL_2013_DWH_IN_THE... · Data Warehouse in the Cloud – Marketing or Reality? Alexei Khalyako Sr. Program Manager ... –Iaas SQL](https://reader033.fdocuments.us/reader033/viewer/2022042223/5ec98a04b83f5f77ec2d4603/html5/thumbnails/33.jpg)
Loading dataClustered Column store Index
Load test 2 hours 16 min
Heap
1 hour 1 min
![Page 34: Data Warehouse in the Cloud Marketing or Reality? SQL_2013_DWH_IN_THE... · Data Warehouse in the Cloud – Marketing or Reality? Alexei Khalyako Sr. Program Manager ... –Iaas SQL](https://reader033.fdocuments.us/reader033/viewer/2022042223/5ec98a04b83f5f77ec2d4603/html5/thumbnails/34.jpg)
SQL 2014
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
CI 708 92. 385 695 834 413 407 508 971 791 251 529 209 393 345 108 729 725 742 111 221 214
CIComp 388 64. 207 382 483 162 253 485 653 451 241 279 158 139 147 85. 387 405 412 108 120 108
CSI 18. 9.9 20. 16. 20. 7.4 11. 24. 70. 63. 41. 14. 96. 12. 6.9 64. 11. 100 9.7 31. 51. 18.
0.00
500.00
1000.00
1500.00
2000.00
2500.00
Qu
ery
Exe
cuti
on
tim
e
Query type
CI CIComp CSI
x51
x39x2
x55
![Page 35: Data Warehouse in the Cloud Marketing or Reality? SQL_2013_DWH_IN_THE... · Data Warehouse in the Cloud – Marketing or Reality? Alexei Khalyako Sr. Program Manager ... –Iaas SQL](https://reader033.fdocuments.us/reader033/viewer/2022042223/5ec98a04b83f5f77ec2d4603/html5/thumbnails/35.jpg)
Query 19
Estimates vs Actual
![Page 36: Data Warehouse in the Cloud Marketing or Reality? SQL_2013_DWH_IN_THE... · Data Warehouse in the Cloud – Marketing or Reality? Alexei Khalyako Sr. Program Manager ... –Iaas SQL](https://reader033.fdocuments.us/reader033/viewer/2022042223/5ec98a04b83f5f77ec2d4603/html5/thumbnails/36.jpg)
And the winner is…
SQL Server 2014!!
![Page 37: Data Warehouse in the Cloud Marketing or Reality? SQL_2013_DWH_IN_THE... · Data Warehouse in the Cloud – Marketing or Reality? Alexei Khalyako Sr. Program Manager ... –Iaas SQL](https://reader033.fdocuments.us/reader033/viewer/2022042223/5ec98a04b83f5f77ec2d4603/html5/thumbnails/37.jpg)
Summary
• Easy and fast deployment through he Gallery or PS scripts
• Azure Data Warehouse is consistent with the most of the best practices– Query
– Loading
• Low Initial investments and TCO
![Page 38: Data Warehouse in the Cloud Marketing or Reality? SQL_2013_DWH_IN_THE... · Data Warehouse in the Cloud – Marketing or Reality? Alexei Khalyako Sr. Program Manager ... –Iaas SQL](https://reader033.fdocuments.us/reader033/viewer/2022042223/5ec98a04b83f5f77ec2d4603/html5/thumbnails/38.jpg)
THANK YOU!• For attending this session and
PASS SQLRally Nordic 2013, Stockholm