Windows AzureData StorageAnton Boyko.NET Developer

Windows Azure StorageStorage in the CloudScalable, durable, and availableAnywhere at anytime accessOnly pay for what the service uses

Exposed via RESTful Web ServicesUse from Windows Azure ComputeUse from anywhere on the internet

Windows Azure Storage AccountUser specified globally unique account name

North Central US Northern Europe

Western Europe East Asia

South East Asia

US Europe Asia

Can choose geo-location to host storage account:

South Central US

West US East US

Storage in the Development FabricProvides a local “Mock” storageEmulates storage in cloudAllows offline developmentRequires SQL Express 2005/2008 or above

http://msdn.microsoft.com/en-us/gg433135

Storage SecurityWindows Azure Storage provides simple security for calls to storage serviceHTTPS endpoint

Two 512bit symmetric keys per storage accountCan be regenerated independently

More granular security via Shared Access Signatures

Windows Azure Storage Abstractions

TablesStructured storage. A table is a set of entities; an entity is

a set of properties.

QueuesReliable storage and delivery of messages for an application.

BlobsSimple named files along with metadata for the file.

DrivesDurable NTFS volumes for Windows Azure applications to use. Based on Blobs.

Blob Storage

Blob Storage Concepts

BlobContainerAccount

http://<account>.blob.core.windows.net/<container>/<blobname>

Pages/ Blocks

contoso

PIC01.JPGBlock/Page

Block/Page

PIC02.JPGimages

VID1.AVIvideos

Blob Details

Main Web Service

Operations

PutBlobGetBlobDeleteBlobCopyBlobSnapshotBlob LeaseBlob

Blob Details

Associate Metadata with Blob

Standard HTTP metadata/headers (Cache-Control, Content-Encoding, Content-Type, etc)

Metadata is <name, value> pairs, up to 8KB per blob

Either as part of PutBlob or independently

Blob Details

Blob always accessed by

name

Can include ‘/‘ or other delimeter in name e.g. /<container>/myblobs/blob.jpg

GET http://.../products?comp=list&prefix=Tents&delimiter=/

<Blob>Tents/PalaceTent.wmv</Blob><Blob>Tents/ShedTent.wmv</Blob>

Enumerating Blobs

GET Blob operation takes parametersPrefixDelimiterInclude= (snapshots, metadata etc…)

http://adventureworks.blob.core.windows.net/ Products/Bikes/SuperDuperCycle.jpg Products/Bikes/FastBike.jpg Products/Canoes/Whitewater.jpg Products/Canoes/Flatwater.jpg Products/Canoes/Hybrid.jpg Products/Tents/PalaceTent.jpg Products/Tents/ShedTent.jpg

http://.../products?comp=list&prefix=Canoes&maxresults=2

&marker=MarkerValue

<Blob>Canoes/Hybrid.jpg</Blob>

Pagination

Large lists of Blobs can

be paginatedEither set maxresults or;Exceed default value for maxresults (5000)

http://.../products?comp=list&prefix=Canoes&maxresults=2

<Blob>Canoes/Whitewater.jpg</Blob><Blob>Canoes/Flatwater.jpg</Blob><NextMarker>MarkerValue</NextMarker>

Two Types of Blobs Under the Hood

Block BlobTargeted at streaming workloads

Each blob consists of a sequence of blocksEach block is identified by a Block ID

Size limit 200GB per blob

Optimistic Concurrency via Etags

Page BlobTargeted at random read/write workloads

Each blob consists of an array of pages Each page is identified by its offset from the start of the blob

Size limit 1TB per blob

Optimistic or Pessimistic (locking) concurrency via leases

TheBlob.wmv

Uploading a Block Blob

Uploading a large blob

10 GB Movie

Bloc

k Id

1Bl

ock

Id 2

Bloc

k Id

3

Bloc

k Id

N

blobName = “TheBlob.wmv”;PutBlock(blobName, blockId1, block1Bits);PutBlock(blobName, blockId2, block2Bits);…………PutBlock(blobName, blockIdN, blockNBits);PutBlockList(blobName,

blockId1,…,blockIdN);

TheBlob.wmv

BenefitEfficient continuation and retryParallel and out of order upload of blocks

THE BLOB

Windows AzureStorage

Page Blob – Random Read/Write

Create MyBlobSpecify Blob Size = 10 GbytesSparse storage - Only charged for pages with data stored in them

Fixed Page Size = 512 bytesRandom Access Operations

PutPage[512, 2048)PutPage[0, 1024)ClearPage[512, 1536)PutPage[2048,2560)

GetPageRange[0, 4096) returns valid data ranges:

[0,512) , [1536,2560)GetBlob[1000, 2048) returns

All 0 for first 536 bytesNext 512 bytes are data stored in [1536,2048)

0

10 GB

5121024153620482560

10 GB Address Space

Shared Access Signatures

Fine grain access rights to blobs and containersSign URL with storage key – permit elevated rightsRevocationUse short time periods and re-issueUse container level policy that can be deleted

Content Delivery Network (CDN)

High-bandwidth global blob content delivery24 locations globally (US, Europe, Asia, Australia and South America), and growingSame experience for users no matter how far they are from the geo-location where the storage account is hosted

Blob service URL vs. CDN URL:Windows Azure Blob URL: http://images.blob.core.windows.net/Windows Azure CDN URL: http://<id>.vo.msecnd.net/ Custom Domain Name for CDN: http://cdn.contoso.com/

pic1.jpg

Windows Azure CDN

To Enable CDN:Register for CDN via Dev PortalSet container images to public

pic1.jpg

GEThttp://guid01.vo.msecnd.net/images/pic.1jpg

http://sally.blob.core.windows.net/images/pic1.jpg

http://sally.blob.core.windows.net/ http://guid01.vo.msecnd.net/

pic1.jpg

404

TTL Content Delivery Network

Windows Azure Blob Service

EdgeLocation

EdgeLocation

EdgeLocation

Table Storage

Table Storage Concepts

EntityTableAccount

contoso

Name =…Email = …

Name =…EMailAdd=

customers

Photo ID =…Date =…

photos

Photo ID =…Date =…

Table Details

InsertUpdate Merge – Partial updateReplace – Update entire entityDeleteQueryEntity Group TransactionsMultiple CUD Operations in a single atomic transaction

Create, Query, DeleteTables can have metadata

Not an RDBMS! Table

Entities

Entity Properties

Entity can have up to 255 propertiesUp to 1MB per entity

Mandatory Properties for every entityPartitionKey & RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic ConcurrencyExposed as an HTTP Etag

No fixed schema for other propertiesEach property is stored as a <name, typed value> pairNo schema stored for a tableProperties can be the standard .NET types String, binary, bool, DateTime, GUID, int, int64, and double

No Fixed Schema

FIRST LAST BIRTHDATE

Wade Wegner 2/2/1981

Nathan Totten 3/15/1965

Nick Harris May 1, 1976

FAV SPORT

Canoeing

Querying

FIRST LAST BIRTHDATE

Wade Wegner 2/2/1981

Nathan Totten 3/15/1965

Nick Harris May 1, 1976

?$filter=Last eq ‘Wegner’

Query Operators(Table Service Support)• From• Where• Take

(the value specified for the Take operator must be less than or equal to 1000)

• First• FirstOrDefault• Select

(projection is supported)

more details http://msdn.microsoft.com/en-us/library/windowsazure/dd135725.aspx

Purpose of the PartitionKey

Entity LocalityEntities in the same partition will be stored together

Entity Group TransactionsAtomic multiple Insert/Update/Delete in same partition in a single transaction

Table ScalabilityTarget throughput – 500 tps/partition, several thousand tps/accountWindows Azure monitors the usage patterns of partitionsAutomatically load balance partitions

PARTITIONKEY(CATEGORY)

ROWKEY(TITLE) TIMESTAMP MODELYEAR

Bikes Super Duper Cycle … 2009

Bikes Quick Cycle 200 Deluxe … 2007

… … … …Canoes Whitewater … 2009Canoes Flatwater … 2006PARTITIONKEY(CATEGORY)

ROWKEY(TITLE) TIMESTAMP MODELYEAR

Rafts 14ft Super Tourer … 1999

… … … …

Skis Fabrikam Back Trackers … 2009

… … … …Tents Super Palace … 2008

PARTITIONKEY(CATEGORY)

ROWKEY(TITLE) TIMESTAMP MODELYEAR

Bikes Super Duper Cycle … 2009

Bikes Quick Cycle 200 Deluxe … 2007

… … … …Canoes Whitewater … 2009Canoes Flatwater … 2006Rafts 14ft Super Tourer … 1999… … … …

Skis Fabrikam Back Trackers … 2009

… … … …Tents Super Palace … 2008

Partitions and Partition Ranges

Server ATable = Products

Server BTable = Products

[Canoes - MaxKey)

Server ATable = Products

[MinKey - Canoes)

SQL Azure

A Server Is Not A Machine

SQL Server A Machine

SQL Azure Database

ServerA TDS

Endpoint

How It WorksArchitectureClient Layer - Used by application to communicate directly with SQL Database.Services Layer – Gateway between Client layer and Platform layer.Platform Layer – Includes physical servicers and services that support the Services layer.Infrastructure Layer – IT administration of the physical HW and OS.

Client Layer

PHP WCFSQL App and Tools

ODBC ADO .NET

Tabular Data Stream (TDS)

Service Layer

TDS + SSL Endpoint

Create Database…Use Familiar TechnologiesTransact-SQLLanguages

.NET Framework (C#, Visual Basic, F#) via ADO.NETC / C++ via ODBCJava via Microsoft JDBC providerPHP via Microsoft PHP provider

FrameworksOData, Entity Framework, WCF Data Services, NHibernate

ToolsSQL Server Management Studio (2008 R2 and later)SQL Server command-line utilities (SQLCMD, BCP)CA Erwin® Data ModelerEmbarcadero Technologies DBArtisan®

SQL Server ComparisonFocus on logical vs. physical administrationDatabase and log files automatically placedThree high-availability replicas maintained for every databaseTables require a clustered indexMaximum database size is 150 GBUnsupported SQL Server FeaturesUse command, distributed transactions, distributed viewsService BrokerCommon Language Runtime (CLR)SQL Agent

Services Layer

Internet

SQL Database Firewall

SQL Database FirewallSecuring your dataIP Address-based access control for SQL DatabaseRules can be defined at the server and databaseNo IP authorized by defaultConfigurable using the SQL Database Portal and REST APIOption to disable/enable access from applications hosted in Windows Azure

SQL FederationDatabase ScalabilityScale to hundreds of nodes via database shardingMulti-tenancy via flexible repartitioningOnline split operations to minimize downtimeAutomatic data discovery regardless of changes in how data is partitioned

Q & A