Azure Storage

Anton BoykoMicrosoft Azure MVPboyko.ant@live.com

Agenda

Azure Storage• Overview• Blob Storage• CDN• Drive Storage• Queue Storage• Table Storage• File Storage

Azure Storage Performance• Blob Storage• Drive Storage• Table Storage

OVERVIEW

Storage in the cloud

• Exposed via RESTful web services• Available globally

Storage SDK in many languages

C#/.NET Python Ruby Perl

Node.JS Java PHP Erlang

Common LISP

Objective-C

Storage security

• HTTP or HTTPS• Two independent 512bit symmetric master

keys• Shared access signatures for more granular

access

Storage artifacts

Blobs Drives Queues

Tables Files

BLOBS

Blob storage concept

Block/Page

Blob

Container

Account contoso

image

pic01.jpg

block01 block02

pic02.jpg

video

mov01.avi

page01 page02

https://contoso.blob.core.windows.net/image/pic01.jpg

Blobs main operations

• Get• Put• Delete• Copy

Quiz time

QuestionWhy do I need a “copy” command if it will do pretty much the same as “get” and “put” do together?

Answer

Quiz time

QuestionWhy do I need a “copy” command if it will do pretty much the same as “get” and “put” do together?

Answer“Copy” command is server-side copy. I do not need to download and upload my blob. Also it’s possible to use different storage accounts for source and destination (even in different locations).

Blobs main operations

• Get• Put• Delete• Copy• Snapshot• Lease

Blob types

Streaming workloads

Block Random

access workloads

Page

Block blobs

TheBlob.wmv

10 GB Movie

Bloc

k Id

1Bl

ock

Id 2

Bloc

k Id

3

Bloc

k Id

N

blobName = “TheBlob.wmv”;PutBlock(blobName, blockId1, block1Bits);PutBlock(blobName, blockId2, block2Bits);…………PutBlock(blobName, blockIdN, blockNBits);PutBlockList(blobName,

blockId1,…,blockIdN);

TheBlob.wmv

BenefitEfficient continuation and retryParallel and out of order upload of blocks

THE BLOB

Page blobs

Create MyBlobSpecify Blob Size = 10 GbytesSparse storage - Only charged for pages with data stored in them

Fixed Page Size = 512 bytesRandom Access Operations

PutPage[512, 2048)PutPage[0, 1024)ClearPage[512, 1536)PutPage[2048,2560)

GetPageRange[0, 4096) returns valid data ranges:[0,512) , [1536,2560)

GetBlob[1000, 2048) returnsAll 0 for first 536 bytesNext 512 bytes are data stored in [1536,2048)

0

10 GB

512

1024

1536

2048

2560

10 GB Address Space

Quiz time

QuestionHow can we increase upload speed for page blobs? What is the preferred way to upload 1TB VHD to Azure?

Answer

Quiz time

QuestionHow can we increase upload speed for page blobs? What is the preferred way to upload 1TB VHD to Azure?

AnswerUpload in many threads. Upload only non-zero pages.

CONTENT DELIVERY NETWORK

Content Delivery Network

• High-bandwidth global content delivery• Many origins– Web App– Cloud Service– Storage Account

Quiz time

QuestionCan I use CDN for my VM’s in Azure?

Answer

Quiz time

QuestionCan I use CDN for my VM’s in Azure?

Answer• Yes. All VMs sit behind

Cloud Services.

Content Delivery Network

• High-bandwidth global content delivery• Many origins– Web App– Cloud Service– Storage Account

• New origin– Custom URL

pic1.jpgpic1.jpg

GEThttp://guid01.vo.msecnd.net/images/pic.1jpg

http://sally.blob.core.windows.net/images/pic1.jpg

http://sally.blob.core.windows.net/ http://guid01.vo.msecnd.net/

pic1.jpg

404

TTL Content Delivery Network

Azure Blob Storage

EdgeLocation

EdgeLocation

EdgeLocation

Content Delivery Network

DRIVES

Azure Drives• Durable NTFS volume for Azure Instances

– Use existing NTFS APIs to access a network-attached durable drive– Use System.IO from .NET

• Benefits– Move existing apps that are using NTFS more easily to the cloud– Durability and survival of data on instance recycle – Drives can be up to 1TB

• Azure Drive is NTFS VHD Page Blob– Mounts Page Blob over the network as NTFS drive– Local cache on instance for read operations– All flushed and unbuffered writes to drive are made durable to the Page Blob

TABLES

Table storage concept

Entity

Table

Account contoso

users

FName=BobLName=Smith

FName=IngaPhone=123

groups

Name=Admin

Table details

Table• Create• Query• Delete

Entity• Insert• Update

– Merge– Replace

• Upsert• Query• Delete

Entity properties• Entity can have up to 255 properties

– Up to 1MB per entity

• Mandatory Properties for every entity– PartitionKey & RowKey (only indexed properties)

• Uniquely identifies an entity• Defines the sort order

– Timestamp • Optimistic Concurrency• Exposed as an HTTP Etag

• No fixed schema for other properties– Each property is stored as a <name, typed value> pair– No schema stored for a table– Properties can be the standard .NET types

String, binary, bool, DateTime, GUID, int and double

Purpose of Partition Key• Entity Locality

– Entities in the same partition will be stored together– Efficient querying and cache locality– Endeavour to include partition key in all queries

• Entity Group Transactions– Atomic multiple Insert/Update/Delete in the same partition in a single transaction

• Table Scalability– Target throughput – 2 000 tps/partition, 20 000 tps/account– Azure monitors usage patterns of partitions– Automatically load balance partitions– Each partition can be served by a different storage node– Scale to meet traffic needs of your table

Poker scoring table

By players• PK = PlayerID• RK = GameID• Score = 42

By games• PK = GameID• RK = PlayerID• Score = 42

there is no silver bullet

Blog tables

Posts

• PK = Date• RK = Time• Authors• Tags• Text

PostsByAuthors

• PK = UserID• RK = Post PK +

RK

PostsByTags

• PK = Tag• RK = Post PK +

RK

IT’S SHOW TIME

DRIVES

Single drive

4 * 1/4 drives

BLOBS

Upload single-threadMbytes uploaded: 142.49238300323486total time in ms: 7377avg ms per Mbyte: 51.771188357714024

Mbytes uploaded: 1424.9273290634155total time in ms: 70467avg ms per Mbyte: 49.45304828023543

Upload multi-threadtotal files uploaded: 365total Mbytes uploaded: 142.4734115600586total time in ms: 4576avg ms per Mbyte: 32.11827350727137

total files uploaded: 365total Mbytes uploaded: 1424.9266424179077total time in ms: 36323avg ms per Mbyte: 25.491136819762723

Copy blobMbytes uploaded: 127000total time in ms: 177avg ms per Mbyte: 0.0013937007874015748

Mbytes uploaded: 127000total time in ms: 1836avg ms per Mbyte: 0.014456692913385827

TABLES

Insert single entityrecords inserted: 500total time in ms: 2920avg ms per record: 5.84

Insert batch of 50records inserted: 500total time in ms: 470avg ms per record: 0.94

Insert batch of 100 (9100 times)records inserted: 910000total time in ms: 1072417 (~18 min)avg ms per record: 1.1784802197802198

Select all single-threadrecords selected: 1500queries count: 2total time in ms: 274avg ms per record: 0.18266666666666667

Select single partitionrecords selected: 5000queries count: 5total time in ms: 815avg ms per record: 0.163

Select all single-threadrecords selected: 1142100queries count: 1143total time in ms: 335824 (~6 min)avg ms per record: 0.29404080203134575

Select all multi-threadrecords selected: 1142100queries count partition: 306queries count data: 1259total time in ms: 48134 (~48 sec)avg ms per record: 0.04214517117590403

total http queries: +37%total time in ms: -76%avg ms per record: -76%

Select by custom propertyrecords selected: 284927queries count: 285total time in ms: 74065avg ms per record: 0.2599437750722115

QuestionsAnton BoykoMicrosoft Azure MVPboyko.ant@live.com

facebook.com/boyko.antyoutube.com/user/boykoant

Google: boykoant