Windows Azure Storage & SQL Azure

Eric D. BoydFounder and CEO, responsiveX Windows Azure MVP

eric.boyd@responsiveX.com@EricDBoyd – EricDBoyd.com

Agenda

Storage Overview

Getting started

Working with Blobs

Adding Tables and Queues

Build and Deploy your Database

Core Services

Caching CDN

Service Bus Reporting

Data Sync Azure Connect

Identity HPC

Additional Services

Windows Azure

Compute Storage Database

Windows Azure Storage

Storage in the CloudScalable, durable, and availableAnywhere at anytime accessOnly pay for what the service usesInstant Concurrency

Exposed via RESTful Web ServicesUse from Windows Azure ComputeUse from anywhere on the Internet

Wrapped in API by SDKMicrosoft.WindowsAzure.StorageClient

Windows Azure Storage Abstractions

TablesStructured storage. A table is a set of entities; an entity is a set of properties.

QueuesReliable storage and delivery of messages for an application.

BlobsSimple named files along with metadata for the file.

DrivesDurable NTFS volumes for Windows Azure applications to use. Based on Blobs.

Windows Azure Storage Account

Create account in Management Portal

http://windows.azure.com

Geographically located with affinity

You get 2 keys…

API based on REST…

http://btlod.blob.core.windows.net/public/helloworld.txt

http://<account>.<BLOB|TABLE|QUEUE>.core.windows.net/...

Tools to Explore Storage Many tools are available some for free, some from 3rd parties, including:

Visual Studiohttp://myAzureStorage.cloudapp.nethttp://clumsyleaf.comhttp://cerebrata.com

These give access to your storage account

Storage Accountsdemo

Windows Azure Storage AccountCan CDN Enable AccountBlobs delivered via 26+ global CDN nodes

Can co-locate storage account with compute accountExplicitly or using affinity groups

Accounts have two independent 512 bit shared secret keysAllows for key rotation when one is compromised

Limit of 100 TB of data per storage accountEach subscription defaults to five storage accounts

Call tech support to create more

Storage in the Cloud Emulator

Storage Emulator provides a local “Mock” storage

Emulates storage in cloud

Allows offline development

Requires SQL Express 2005/2008 or above

SSMS Provides local view of storage items

http://msdn.microsoft.com/en-us/gg433135

Storage Account

Storage AccountSLA – 99.9% AvailabilityCapacity – Up to 100 TBs per storage accountTransactions – Up to 5,000 requests per secondBandwidth – Up to a few hundred megabytes per second

Single PartitionThroughput up to 60 MB/sUp to 500 transactions (entities or messages) per second

Go Beyond the LimitsPartition between multiple storage accounts and partitionsWhen limit is hit, app may see ‘503 server busy’: applications should implement retry.

Working with BlobsBinary Large Objects include resources we use in our applications like pictures, videos, html, css, etc. that may not fit best in a relational schema

Choice of API’s – REST vs. SDK wrapper To use SDK add references to Microsoft.WindowsAzure.StorageClient Create Storage Account Object Add client for blobs Work with container references and blobs

Table Storage Concepts

EntityTableAccount

contoso

Name =…Email = …

Name =…EMailAdd=

customers

Photo ID =…Date =…

photos

Photo ID =…Date =…

Table Details

InsertUpdate Merge – Partial update

Replace – Update entire entity

UpsertDeleteQueryEntity Group TransactionsMultiple CUD Operations in a single atomic transaction

Create, Query, DeleteTables can have metadata

Not an RDBMS! Table

Entities

Entity PropertiesEntity can have up to 255 propertiesUp to 1MB per entity

Mandatory Properties for every entityPartitionKey & RowKey (only indexed properties)Uniquely identifies an entityDefines the sort order

Timestamp Optimistic ConcurrencyExposed as an HTTP Etag

No fixed schema for other propertiesEach property is stored as a <name, typed value> pairNo schema stored for a tableProperties can be the standard .NET types String, binary, bool, DateTime, GUID, int, int64, and double

No Fixed Schema

FIRST LAST BIRTHDATE

Wade Wegner 2/2/1981

Nathan Totten 3/15/1965

Nick Harris May 1, 1976

FAV SPORT

Canoeing

Querying

FIRST LAST BIRTHDATE

Wade Wegner 2/2/1981

Nathan Totten 3/15/1965

Nick Harris May 1, 1976

?$filter=Last eq ‘Wegner’

Purpose of the Partition KeyEntity LocalityEntities in the same partition will be stored togetherEfficient querying and cache localityEndeavour to include partition key in all queries

Entity Group TransactionsAtomic multiple Insert/Update/Delete in same partition in a single transaction

Table ScalabilityTarget throughput – 500 tps/partition, several thousand tps/accountWindows Azure monitors the usage patterns of partitionsAutomatically load balance partitionsEach partition can be served by a different storage nodeScale to meet the traffic needs of your table

PARTITIONKEY(CATEGORY)

ROWKEY(TITLE)

TIMESTAMP MODELYEAR

BikesSuper Duper Cycle

… 2009

BikesQuick Cycle 200 Deluxe

… 2007

… … … …

Canoes Whitewater … 2009

Canoes Flatwater … 2006

PARTITIONKEY(CATEGORY)

ROWKEY(TITLE)

TIMESTAMP MODELYEAR

Rafts14ft Super Tourer

… 1999

… … … …

SkisFabrikam Back Trackers

… 2009

… … … …

Tents Super Palace … 2008

PARTITIONKEY(CATEGORY)

ROWKEY(TITLE)

TIMESTAMP MODELYEAR

BikesSuper Duper Cycle

… 2009

BikesQuick Cycle 200 Deluxe

… 2007

… … … …

Canoes Whitewater … 2009

Canoes Flatwater … 2006

Rafts14ft Super Tourer

… 1999

… … … …

SkisFabrikam Back Trackers

… 2009

… … … …

Tents Super Palace … 2008

Partitions and Partition Ranges

Server A

Table = Products

Server B

Table = Products[Canoes - MaxKey)

Server A

Table = Products[MinKey - Canoes)

Loosely Coupled Workflow with QueuesEnables workflow between rolesLoad work in a queueProducer can forget about message once it is in queueMany workers consume the queueFor extreme throughput (>500 tps) Use multiple queuesRead messages in batchesMultiple work items per message

Queue

Input Queue (Work Items)

Web Role

Web Role

Web Role

Worker Role

Worker Role

Worker Role

Worker Role

Queue Details

Simple asynchronous dispatch queueNo limit to queue length, subject to storage limit.64kb per messageListQueues - List queues in account

Queue operations CreateQueue DeleteQueueGet/Set MetadataClear Messages

Message operationsPutMessage– Reads message and hides for time periodGetMessages – Reads one or more messages and hides themPeekMessages – Reads one or more messages w/o hiding themDeleteMessage – Permanently deletes messages from queueUpdateMessage – Clients renew the lease and contents

Queue’s Reliable Delivery

Guarantee delivery/processing of messages (two-step consumption)

Consumer dequeues message and it is marked as invisible for a specified “Invisibility Time”

Consumer deletes message when finished processing

If consumer crashes, message becomes visible for another consumer to process

Message lifecycle

Queue

Msg 1

Msg 2

Msg 3

Msg 4

Worker Role

Worker Role

PutMessage

Web Role

GetMessage (Timeout)

RemoveMessage

Msg 2Msg 1

Worker Role

Msg 2

Windows Azure Storage SummaryFundamental data abstractions to build your applicationsBlobs: Files and large objectsDrives: NTFS APIs for migrating applicationsTables: Massively scalable structured storageQueues: Reliable delivery of messages

Easy to use via the Storage Client Library

Core Services

Caching CDN

Service Bus Reporting

Data Sync Azure Connect

Identity HPC

Additional Services

Windows Azure

Compute Storage Database

Start with the SQL Azure Basics

SQL Azure Database isSQL Server database technology delivered as a service on the Windows Azure PlatformIdeal for both simple and complex applicationsEnterprise-ready with automatic support for HADesigned to scale out elastically with demand

Get started quicklyChoose a planChoose a billing optionProvision servers

Ready to get started?Let’s jump into development

Provision Your ServerServer definedService head that contains databases

Connect via automatically generated FQDN (xxx.database.windows.net)

Initially contains only a master database

Provision servers interactivelyLog on to Windows Azure Management Portal

Create a SQL Azure server

Specify admin login credentials

Add firewall rules and enable service access

Automate server provisioningUse Windows Azure Platform PowerShell cmdlets (or use REST API directly)

wappowershell.codeplex.com

SQL Azurecreate server & db

demo

Build Your DatabaseUse familiar technologiesSupports Transact-SQL

Supports popular languages.NET Framework (C#, Visual Basic, F#) via ADO.NET

C / C++ via ODBC

Java via Microsoft JDBC provider

PHP via Microsoft PHP provider

Supports popular frameworksOData, Entity Framework, WCF Data Services, NHibernate

Supports popular toolsSQL Server Management Studio (2008 R2 and later)

SQL Server command-line utilities (SQLCMD, BCP)

CA Erwin® Data Modeler

Embarcadero Technologies DBArtisan®

Differences in comparison to SQL Server Focus on logical vs. physical administration

Database and log files automatically placed

Three high-availability replicas maintained for every database

Databases are fully contained

Tables require a clustered index

Maximum database size is 150 Gb

Unsupported SQL Server featuresBACKUP / RESTORE

USE command, linked servers, distributed transactions, distributed views, distributed queries, four-part names

Service Broker

Common Language Runtime (CLR)

SQL Agent

Build Your DatabaseThin client database developmentIntroducing the SQL Azure management portal

Web designers for tables, views, stored procs

Interactive query editing and execution

Rich client database developmentIntroducing SQL Server Data Tools (SSDT)

Visual Studio IDE for database development

Includes modern designers and projects with declarative, model-driven development

Develop and test in both connected and disconnected states

Platform targeting for both SQL Server (2005 and above) and SQL Azure

Get it free with Web PI, with SQL Server 2012 and with Visual Studio 11

Deploy Your DatabaseData-tier Application Framework (DAC Fx)Alternative to traditional script based approach

Dramatically simplifies deployment, migration and versioning of databases

Provides a single unit of deployment for schema (dacpac) or for schema + data (bacpac)

Supports automatic versioning of database schemas

Supports platform targeting for both SQL Server (2005 and above) and SQL Azure

Build from scratch or extract from existing db

How to get the latest DAC FxWith SQL Server Data Tools

With SQL Server 2012 Management Studio

With SQL Azure Import/Export Service

Via sqldacexamples.codeplex.com

Deploy Your DatabaseInteractive approach for dacpac v1 and v2Manage your server with SQL Azure Management Portal

Deploy dacpac

Load data (scripts, bulk copy, SSIS)

Also supported in SQL Server 2012 Management Studio

Interactive approach for bacpac v2Upload bacpac to blob storage

Log into Windows Azure Management Portal

Import bacpac, table data automatically loaded

Also supports export to blob storage

Upgrading a dacpac or bacpacNot supported in portals yet

Use SQL Server 2012 Management Studio

There are two ways to secure a database:

Within the Database

On the Server

Connect Your ApplicationConnecting to SQL Azure1. TDS (Tabular Data Stream) protocol over

TCP/IP supported

2. SSL required

3. Use firewall rules to connect from outside Microsoft data center

ASP.NET EXAMPLE:

Special considerations1. Legacy tools and providers may

require special format for login: [login]@[server]

2. Idle connections terminated after 30 minutes

3. Long running transactions terminated after 24 hours

4. DoS guard terminates suspect connections with no error message

5. Failover events terminate connections

6. Throttling may cause errors

7. Use connection pooling and implement retry logic to handle transient failures

8. Latency introduced for updates due to HA replicas

9. No cross-database dependencies, result sets from different databases must be combined in application tier

<connectionStrings><addname="AdventureWorks"connectionString=

"Data Source=[server].database.windows.net;Integrated Security=False;Initial Catalog=ProductsDb;User Id=[login];Password=[password];Encrypt=true;"

providerName="System.Data.SqlClient"/></connectionStrings>

Visualize Your DataSQL Azure Reporting isSQL Server Reporting Services technology delivered as a service on the Windows Azure Platform

Ideal for operational reporting against SQL Azure data

Enterprise-ready with automatic support for HA

Designed to scale elastically with demand

Get started quicklyProvision report server via Windows Azure Management Portal

Build reports with Reporting Services Report Designer

Deploy reports to report server

Render reports with Visual Studio ReportViewer controls

Synchronize Your Data

SQL Azure (Hub)

SQL Azure (US)

SQL Azure (WE)

SQL Server

(OnPrem)

SQL Azure Data Sync isMicrosoft Sync Framework technology delivered as a service on the Windows Azure Platform

Ideal for scheduling synchronization between data sets hosted in SQL Azure or SQL Server

Uses a hub and spoke topology

Special considerationsConflict resolution policy configured centrally (hub or client wins)

Sync direction configured between each client and the hub (to hub, from hub, bi-directional)

Sync schedule must be between 5 minutes and 1 month

Data sets include multiple tables and can be filtered, triggers are added to data set tables

Tables added to hub and client schemas

Agent must be installed for on-prem clients

Scale Out Your Data

SQL Azure Federations providesIntegrated database sharding that can scale to hundreds of nodes

Multi-tenancy via flexible repartitioning

Online split operations to minimize downtime

Automatic data discovery regardless of changes in how data is partitioned

Special considerationsA logical database can contain multiple federations

Distribution scheme supports int, bigint, guid, and varbinary types

Filtering routes connection to appropriate shard regardless of changes in partitions

Merge, fan-out queries and automatic distribution of schema changes not supported in initial release

Common Data

Db

Federation Root

A-D E-M T-WN-S X-Z

Hands on Labbuild the backend

one and one half hours

lab 2

© 2011 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.

The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after

the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.