Loading Data Using Sq l Bulk Copy
-
Upload
vikas-c-pulpa -
Category
Documents
-
view
226 -
download
0
Transcript of Loading Data Using Sq l Bulk Copy
-
8/11/2019 Loading Data Using Sq l Bulk Copy
1/21
SQL Azure and SqlBulkCopyPresented by Herve Roggero
Managing Partner, Blue Syntax Consulting
SQL Azure MVP
Email: [email protected]
Twitter: @hroggero
LinkedIn: SQL Azure and SQL Server Security
LinkedIn: Azure Florida Association
www.bluesyntax.net
-
8/11/2019 Loading Data Using Sq l Bulk Copy
2/21
AgendaAvailable tools to load data
INSERT commands
SqlDataAdapterAnalyzing BCP
Analyzing SqlBulkCopy
Performance test
-
8/11/2019 Loading Data Using Sq l Bulk Copy
3/21
Available tools/techniques INSERT
INSERT (2)
INSERT BULK SqlDataAdapter
Migration Wizard
SSIS
BCP / INSERT BULK
SqlBulkCopy
-
8/11/2019 Loading Data Using Sq l Bulk Copy
4/21
What makes up load time? Connection Time
Send Query
Compile Query Execute Query
Send Data
-
8/11/2019 Loading Data Using Sq l Bulk Copy
5/21
INSERT Classic way to insert records
Logged operation
Usually single record at a timeAll packets are encrypted => too verbose
VERY SLOW...
INSERT INTO dbo.MyProducts (Name, ListPrice) VALUES ('Helmet', 25.50)
INSERT INTO dbo.MyProducts (Name, ListPrice) VALUES ('Wheel', 30.00);
-
8/11/2019 Loading Data Using Sq l Bulk Copy
6/21
INSERT (2) New INSERT command in SQL Server 2008
Table Value Constructor
Not as verbose Operation is still logged
Faster than regular INSERT, but still SLOW
INSERT INTO dbo.MyProducts (Name, ListPrice) VALUES
('Helmet', 25.50),
('Wheel', 30.00);
-
8/11/2019 Loading Data Using Sq l Bulk Copy
7/21
INSERT (summary)
INSERT INTO
INSERT INTO
INSERT INTO SQL Azure
SQL Azure
SQL Azure
-
8/11/2019 Loading Data Using Sq l Bulk Copy
8/21
BULK INSERT NOT SUPPORTED in SQL AZURE
INSERT BULK INTO
SQL Server
Raw packets
Raw packets
Raw packetsTERMINATION
File
-
8/11/2019 Loading Data Using Sq l Bulk Copy
9/21
INSERT BULK Supported but requires feeding raw TDS packets
=> Must be used by a Tool
Very efficient and lean Sends 1 INSERT statement
Everything that follows are raw data packets
NO FURTHER COMPILATION/QUERY EXECUTION
SQL Azure waits until Session sends TERMINATION Operation not logged
FAST
-
8/11/2019 Loading Data Using Sq l Bulk Copy
10/21
INSERT BULK (cont.)
INSERT BULK INTO
SQL Azure
Raw packets
Raw packets
Raw packets
TERMINATION
-
8/11/2019 Loading Data Using Sq l Bulk Copy
11/21
BCP Uses INSERT BULK
Performance of BCP depends on INSERT BULK
However BCP can also extract data
FAST
-
8/11/2019 Loading Data Using Sq l Bulk Copy
12/21
Tools Tools that use the BCP program:
SSIS: Bulk Insert Task
SQL Azure Migration Wizard http://sqlazuremw.codeplex.com/
Tools that use the SQL Bulk Copy API
Enzo Data Copy
-
8/11/2019 Loading Data Using Sq l Bulk Copy
13/21
SqlDataAdapter Based on XML Datagrams (differences)
Very memory intensive
Can be very slow depending on approach Provides a way to work in Batches
Sends INSERT operations
SLOW
-
8/11/2019 Loading Data Using Sq l Bulk Copy
14/21
SqlBulkCopy Uses INSERT BULK operation
Programmatic access, options similar to BCP
Can send data in Batch FAST
-
8/11/2019 Loading Data Using Sq l Bulk Copy
15/21
Techniques SummaryTool/Technique Operation Logged SpeedINSERT INSERT Yes SLOW
BULK INSERT BULK INSERT No n/a
INSERT BULK INSERT BULK No FASTBCP INSERT BULK No FAST
SSIS/ MigrationWizard
INSERT BULK No FAST
Data Adapter INSERT Yes SLOW
SqlBulkCopy INSERT BULK No FAST
Enzo Data Copy INSERT BULK No FAST
-
8/11/2019 Loading Data Using Sq l Bulk Copy
16/21
About the performance test Load 40,000 records (on-premise -> SQL Azure)
Uses multithreading (TPL) 8 threads
Comparing Data Adapter
Stored Procedure
SqlBulkCopy API
INSERTs
-
8/11/2019 Loading Data Using Sq l Bulk Copy
17/21
Performance ChartX: Batch Size; Y: Load Time {40K records; DOP = 8}
Batch Size
LoadTime(sec)
2000 3000 40001000500100
10
20
30
Data Adapter
Stored Procedure
SqlBulkCopy
INSERT 2 [limited to batch of 1000]
-
8/11/2019 Loading Data Using Sq l Bulk Copy
18/21
Comparing Tools
40,000 Records (insecs)
2,000,000 Records(in mins)
SSIS 4.2 2.17
SQL Bulk Copy 2.18 4.3
Enzo Data Copy 25 1.41
0
5
10
15
20
25
30
SSIS
SQL Bulk Copy
Enzo Data Copy
-
8/11/2019 Loading Data Using Sq l Bulk Copy
19/21
Demos Sample Code
Uses parallel processing on all methods
Compares: Data Adapter
Inline INSERTs
Stored Procedure INSERTs
SqlBulkCopy
SSIS
Enzo Data Copy (www.bluesyntax.net)
-
8/11/2019 Loading Data Using Sq l Bulk Copy
20/21
Considerations Resilience to Throttling
Adapting to throttling conditions
Changing workload depending on errors Connection Retries
Retrying for connection loss
Causes include throttling, load balancing, network
http://social.technet.microsoft.com/wiki/contents/articles/sql-azure-connection-management.aspx
-
8/11/2019 Loading Data Using Sq l Bulk Copy
21/21
Thank youPresented by Herve Roggero
Managing Partner, Blue Syntax Consulting
SQL Azure MVP
Email: [email protected]
Twitter: @hroggero
LinkedIn: SQL Azure and SQL Server Security
LinkedIn: Azure Florida Association
Blog: www.geekswithblogs.net/hroggerowww.bluesyntax.net