Migrating Data to SQL Azure
-
Upload
lynn-langit -
Category
Technology
-
view
3.732 -
download
3
description
Transcript of Migrating Data to SQL Azure
Lynn LangitSr. Developer EvangelistMicrosoft CorporationBlogs.msdn.com/SoCalDevGal
Migrating Applications to SQL Azure
v1.1
SESSION CODE: COS311
2
Migrating Applications to SQL Azure
OverviewSQL Azure
App Migration Data Access Tier Considerations
Database MigrationMigrating to SQL Azure from MySqlMoving Schema and Data into SQL AzureTransferring Large Data into SQL Azure
Future
3
Developer ExperienceUse existing skills and tools
The Windows Azure Platform
platformAppFabric
Compute Storage Management Relational data Management Connectivity Access control Reporting & BI
Billing & Payments
Flexible APIs
Information Marketplace
4
SQL Azure DatabaseThe first and only true relational database as a service
Elastic Scale Developer Agility
Self-managed
• Database utility; pay as you grow
• Flexible load balancing• Business-ready SLAs• Enable multi-tenant
solutions• World-wide presence
• Easy provisioning and deployment
• Auto high-availability and fault tolerance
• Self-maintaining infrastructure; self-healing
• No need for server or VM administration
• Build cloud-based database solutions on consistent relational model
• Leverage existing skills through existing ecosystem of developer and management tools
• Explore new data application patterns
Application Topologies
Application/ Browser
App Code
(ASP.NET)
App Code(ASP.NET)
TS
QL T
DS
SQL Azure
WindowsAzure
Code Near
App code/ Tools
SQL Azure
Code Far
Hybrid
SQ
L A
zure
/ D
ata
Syn
c
SQL Azure
SQL Server App code/ Tools
App Code
(ASP.NET)
App Code(ASP.NET)
T-S
QL /
TD
S
TS
QL T
DS
WindowsAzure
6
When Migrating Applications
Data Access APIs SupportSecurity ModelConnections & SizingAdmin and Programmability Surface SupportCollation Behavior
7
Data Access APIsSupported APIs
Connection StringADO.Net - Encrypt=True and add @servername to User IDODBC - Encrypt=yes and add @servername to Uid
Supported UnsupportedADO.Net .Net 3.5 SP1 and 4.0 ODBC - SNAC 10Entity Framework .Net 3.5 SP1 and 4.0SQL Server 2008 Driver for PHP v1.1
OleDB
8
Data Access APIs cont.
Protocols
Authentication Mode
Supported UnsupportedTCP/IP over port 1433 Dynamic ports
Named pipesShared memory
Supported UnsupportedSQL Auth Windows Auth
9
Connectivity & SizingRobust Connection Management through Retry Logic
More On this Topic - COS305: Microsoft SQL Azure Development Best PracticesFailover
HW Failures or Upgrades
Resource Management OperationsThrottling and Load Balancing
ALTER DATABASE (T-SQL)Changes to Database Name Changes to MAXSIZE/EDITION
ALTER DATABASE [TestDB] MODIFY (MAXSIZE=50GB)GO
10
Security ManagementAuthentication - SQL Server (only)Manage Logins
CREATE LOGIN [testlogin] WITH PASSWORD='password‘
Manage UserCREATE USER testuser FROM LOGIN testlogin;
Manage Server RolesEXEC sp_addrolemember ‘dbmanager', 'testuser'; [securityadmin role]EXEC sp_addrolemember ‘loginmanager', 'testuser'; [dbcreator]
Granting Database Access to a LoginCreate Login and Database in MasterCreate User and Grant Role in User Database
Viewssys.sql_logins. sys.databases, sys.database_principal
11
Firewall ManagementPortal AccessProgrammatic Access
System Views and Stored Procedures - hereView: sys.firewall_rulesStored Procedure: sp_set_firewall_ruleStored Procedure: sp_delete_firewall_rule
API Implementation Example- here
12
CollationsDefault Collation
Set to SQL_LATIN1_GENERAL_CP1_CI_ASApplies to…
Metadata, TempDb and Default for User Data Object Names (tables, sprocs etc), String variables and Columns when no collation is specified.
Use column and/or expression level collation when neededCREATE TABLE [dbo].[ErrorLog](
[ErrorLogID] [int] IDENTITY(1,1) NOT NULL,[ErrorTime] [datetime] NOT NULL,[UserName] [nvarchar(4000)] COLLATE Turkish_100_CI_AI NOT NULL,[ErrorNumber] [int] NOT NULL,[ErrorMessage] [nvarchar](4000) COLLATE Japanese_CI_AS NOT NULL,
CONSTRAINT [PK_ErrorLog_ErrorLogID] PRIMARY KEY CLUSTERED ([ErrorLogID] ASC) )
declare @str nvarchar(128) COLLATE Greek_CS_AI
13
Feature ParityAdmin Surface – no physical administration
You have a master database You do NOT have server level constructs (sp_configure, endpoints, DBCC commands, server level DMVs and System Views…)
Programmability SurfaceMost RDMS features availablePartially available - USE, XML processing, deprecated T-SQL etc. – list hereSome Features are NOT available today
Full-text Search, Remote Data Access and Linked Servers, Distributed Transactions, Change Tracking, Service Broker etc.
List is available here
14
Migrating ApplicationsApplication Data Access Tier Just Works Need to Investigate
Data Access APIs Support ADO.Net, ODBC, EF and PHP
OleDB
Security Model SQL Auth Windows Auth
Connection Management Apps with transaction retry logic, 3-tiered, stateless or partially statefull apps
Legacy Apps2 tiered apps with online connections
Admin and Programmability Surface Support
Tables, Views, Sprocs, Functions, Triggers, built-in data types etc
Server level options, FTS, Service Broker, Distributed Transactions
Collation Behavior Apps explicit with collation options
Non-English apps depending on db/server collation defaults
15
Database Migration
Moving Schema and / or DataSource Database types
SQL ServerMySQLAccessOther RDMS
16
From SQL Server to SQL Azure
Tool ChoicesSSMS for R2 > Generate Script WizardSSMS for R2 > Import / Export WizardCopy Database commandSSIS for R2SQL Azure Migration Wizard3rd Party
17
SSMS– Generate Script WizardMigration of schema and/or data w/ fine grain control
Pros - Native support for SQL Azure Schema options: “Engine Type = SQL Azure”
Ensures correct options and settings are applied for the TSQL script generation.Requires explicit action on unsupported objects.
Cons - Verbose – INSERT Statements instead of raw data
Data is scripted with a fixed 100 row batch size.Edit “GO” statements between small batches
Round-trip EfficiencyUse “SET NOCOUNT ON”
18
Generate Script Wizard - DEMO
Migrating Databases to SQL Azure
20
Migration Assist for MySql / Access (CTP)
Scenario - Auto porting of schema, database code and data from MySql and Access to SQL Azure
SQL Server Migration Assistant for MySql and Access
Supports MySQL 4.1 and upSupport Access v 97 and upSQL Server versions supported (all editions)
SQL Azure, SQL Server 2005, SQL Server 2008 and 2008 R2
21
MySQL to SQL Azure - DEMO
SQL Server Migration Assistant Version for MySQLVersion for Access
22
MySQL Migration Tool
23
DAC (Data-tier Application) PackageDACPac = Single unit for authoring, deploying, and managing data-tier objects (schema)
Development Lifecycle (VS 2010)Creating / Editing DACs
Schema and DB Code DevelopmentCode Analyses, Deployment Policy SettingsSchema Comparison and more…
Management Lifecycle (SSMS 2008 R2)Extracting DACs (as files)Managing DACs (deploy or upgrade)Registering existing database as DACs
If ‘regular’ into MSDBIf ‘Azure’ into masterIf using UCP into UCP database
24
DACs - DEMO
Migrating Databases to SQL Azure
25
Transferring Large Data to SQL Azure
Best Practices for efficient Data movement to SQL Azure
Use the right tools is key BCPSSIS
Network performance considerationsLatency vs Bandwidth
26
BCP & Bulk Copy APIs for Large Data Transfer
Optimize Databases for Data ImportDisable/Delay creation of none Clustered IndexesOrder Data on the Clustered IndexDisable Triggers and Constraint checking
-N Native mode so no data type conversion needed.
-c Character mode for interoperability-b batchsize will let you control transaction size-h”TABLOCK, ORDER(…)” optimize writes
27
Import/Export Wizard – for data transfer
Scenario - Simplified wizard for migrating schema and data through SSISPros
Great performance out of the boxAllows ‘save as package’ for full control
28
SSIS Data Flow Task (Large Data Xfer)Diverse Source and destinations - To/From Flat Files, ADO.Net, OleDBFully programmable flow
Loops, Sorts, Conditional operators, XML/WebServices Processing etcVS Debugging support with data viewers, watches and conditional breakpoints
Optimize Databases for Data ImportDisable/Delay creation of none Clustered IndexesOrder Data on the Clustered IndexDisable Triggers and Constraint checkingBatch Size: Adjust the transaction sizeBuffer and Blob temp storage area; spread over to multiple drives for large data transfersParallelization – based on execution trees, task will auto parallelize
29
SSIS package - DEMO
Use supported features only, such as bcp, but not BULK INSERT, etc…
30
Parallelization & Network PerfLatency & Bandwidth for high performance import & export
Latency is about laws of physicsLong distances means high latencyParallelization can help mitigate some latency
Bandwidth is key! Once you reach the bandwidth limit parallelization won’t help
See also COS13-INT: Database Performance in a Multi-tenant Environment
31
Datacenter Locations
32
BCP Performance – test set up example
CREATE TABLE [dbo].[table1]( [c01] [int] NOT NULL, [c02] [int] NOT NULL, [c03] [smallint] NOT NULL, [c04] [int] NOT NULL, [c05] [smallint] NOT NULL, [c06] [smallint] NOT NULL, [c07] [char](50) NOT NULL, [c08] [char](24) NOT NULL, … [c17] [char](24) NOT NULL, PRIMARY KEY CLUSTERED ([c01] ASC,[c02] ASC))
CREATE NONCLUSTERED INDEX [index1] ON [dbo].[table1] ([c07] ASC)
Data Distribution:Row Size: 600BytesRow Count: 100KPages: ~7700Total Size: ~60MBEven Distribution on PkeyConstraints and Indexes
Compound Pkey (int+int) None-clustered key on char(50)No triggers or relationships
33
BCP Upload Perf - Rows/Sec
Low Bandwidth, Medium Latency - 114ms
Low Bandwidth, Very High Latency - 289ms
High Bandwidth, High Latency - 169ms
High Bandwidth, Medium Latency - 54ms
0 1000 2000 3000 4000 5000 6000 7000 8000 9000
Parallel Upload Batch Size 1K (sum rows/sec)
Native Sorted - Batch 10K (rows/sec)
Native - Batch 10K (rows/sec)
Native - Batch 100 (rows/sec)
34
BCP Download Perf – Rows/sec
Medium Bandwidth, Medium Latency - 114ms
Medium Bandwidth, Very High Latency - 289ms
High Bandwidth, High Latency - 169ms
High Bandwidth, Medium Latency - 54ms
0 2000 4000 6000 8000 10000 12000 14000 16000
4x Native Sorted - Batch Size 1K (sum rows/sec)
1x Native Sorted - Batch 10K (rows/sec)
35
Guidance on Data Transfer to SQL Azure
Split data transfer into multiple concurrent streamsTune BCP batch size and network packet size option Add non clustered indexes after loading data to SQL Azure.
Two additional indexes created before loading the data increased the final database size by ~50% and increased the time to load the same data by ~170%.
If, while building large indexes, you see a throttling-related error message, retry using the online option.
36
Monitoring Tasks• Space Used
• Monitor space used via ‘sys.dm_db_partition_stats’
• Trap ‘out of space exception’ – SQL Error 40544
• Synchronized Copies• Database copy• SQL Azure Synchronization
Provider•Other monitoring tasks –
use supplied DMVs (list in notes for this slide
37
Database Migration Tools
Tools SSMA Generate Script Wizard
DACs SSIS BCP
MySql and Access to SQL Azure
√ (Schema and Data)
√ (Data Only)
Move SQL Schema √ √ √
Move Data √ √ √
Move Large Data √ √
38
Future for Migration More information on all upcoming features at
DAT209: What’s New in Microsoft SQL AzureMigrating Applications
ConnectivityOleDb Support
SQL Server ParityDB Collation Support
OData Support – SQL Azure LabsFurther reach for SQL Azure data
39
Related ContentBreakout Sessions
AZP301 – A Lap Around the Azure PlatformAZP302 – Using Visual Studio 2010 to Build Applications that run on AzureDAT207 – What’s New in Microsoft SQL AzureAZP303 – Migrating Applications to Microsoft SQL Azure
40
Track Resources
Team Blog – http://blogs.msdn.com/sqlazure
MSDN Dev Center – http://msdn.microsoft.com/azure/sqlazure
Follow us on Twitter - @sqlazure
Windows Azure Platform Training Kit - http://bit.ly/dgvMv0
SQL Azure Labs – http://www.sqlazurelabs.com
41
Resources
www.microsoft.com/teched
Sessions On-Demand & Community Microsoft Certification & Training Resources
Resources for IT Professionals Resources for Developers
www.microsoft.com/learning
http://microsoft.com/technet http://microsoft.com/msdn
Learning
Complete an evaluation via CommNet and Tag to win amazing prizes!
43
TeachingKidsProgramming.org
• Do a Recipe Teach a kid• Free Courseware (recipes)• Ages 10 ++• Microsoft SmallBasic
© 2008 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any
information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.