Azure Data StorageKen Cenerelli@KenCenerelli

Microsoft Azure

Agenda

Blobs Files Queues Tables

Agenda

SQL

Database

SQL Server

VMDocumentDB Search HDInsight

Microsoft Azure

Storage Blob

Two Types of Blobs Under the Hood

Two Types of Blobs Under the Hood

Block Blob:

Two Types of Blobs Under the Hood

Page Blob:

Blob Storage Concepts

Blob Details –

Shared Access Signatures

Policy based:

Stored Access PolicyUse for longer dated revocable permission sets

Points to a Container level policy

Can only revoke by deleting the blob or waiting for expiry

Use case: Providing revocable permissions to certain users/groups

Shared Access Signatures – Two broad approaches

Ad-hoc:

Shared Access SignatureUse for very short dated single use scenarios

Can only revoke by deleting the blob or waiting for expiry

Use case: Single use URLs

E.g. Provide URL to mobile client to upload to container

Always endeavour to use least permission set possible

Microsoft Azure

Storage Files

Azure Files

Shared Network File Storage for Azure

Availability, durability, scalability are managed automatically

Supports two interfaces: SMB and REST

Azure Files – Usage

Share data across VMs and applications

Share settings throughout services

Share tools amongst developers in a group

Dev/Test/Debug

Azure Files vs Blobs

Description Azure Blobs Azure Files

Durability

Options

LRS, ZRS, GRS (and RA-GRS for higher

availability)LRS, GRS

Accessibility REST APIsSMB 2.1 (standard file system APIs)

REST APIs

Connectivity REST – WorldwideSMB 2.1 - Within region

REST – Worldwide

Endpointshttp://myaccount.blob.core.windows.net/

mycontainer/myblob

\\myaccount.file.core.windows.net\myshare\myfile.txt

http://myaccount.file.core.windows.net/myshare/myfile.txt

DirectoriesFlat namespace however prefix listing

can simulate virtual directoriesTrue directory objects

Case Sensitivity of Names Case sensitive Case insensitive, but case preserving

Capacity Up to 500TB containers 5TB file shares

Throughput Up to 60 MB/s per blob Up to 60 MB/s per share

Object size Up to 1 TB/blob Up to 1 TB/file

Billed capacity Based on bytes written Based on file size

Microsoft Azure

Storage Queue

Queue length reflects how well the backend processing nodes are doing.

Decouples the application and makes scaling easier.

Flexibility of efficient resource usage within an application.

Absorb traffic bursts and reduce the impact of individual component failures.

Why use a Queue?

Storage Account: All access to Azure Storage is done through

a storage account. It can store millions of messages until the

storage account’s maximum capacity of 500 TB is reached.

Queue: A queue contains a set of messages.

Message: A message, in any format, of up to 64KB.

Queue Components

Queues are addressable using the following URL format:

http://{storage-account}.queue.core.windows.net/{queue}

Queue URL format

Faster Web Applications with queues

using asynchronous workloads

Demo

Messages are ordered but not guaranteed FIFO.

Message will be processed at least once.

Message may be processed more than once.

.DequeueCount increases every time.

Messages are stored up to 7 days.

Queue Considerations

Microsoft Azure

Storage Table

Table Storage Concepts

Table Storage Details

Table Storage Details

Table Storage Details – Entity Properties

Table Storage Details

Table Storage Details – Purpose of the PartitionKey

Microsoft Azure SQL Database

From private to public Cloud

Microsoft Relational Storage Options

A Continuous offering

A Server is not a machine

SQL Server database technology “as a Service”

Fully Managed

Enterprise-ready with automatic support for HA, DR, Backups, replication and more

SQL Database – The Basics

Scale out with ElasticScale

Built-in regional database replicas for additional protection

Uptime SLA of 99.99%

SQL Database – The Basics

The Service head contains databases

Connect via automatically generated FQDN: {name}.database.windows.net

Initially contains only a master database

SQL Database – Server Definition

Auto backups, transactional logs every 5 min

Backups in Azure Storage and geo-replicated

Creates a side-by-side copy, non-disruptive

Backups retention policy: 7, 14 or 35 days

Automated export of logical backups for long-term backup protection

Point-in-time restore - “oops recovery”

Available in all tiers: Basic, Standard and Premium

Built on geo-redundant Azure Storage

Recover to any Azure region

Geo-restore – Emergency data recovery when you need it most

Focus on logical vs. physical administration

Database and log files automatically placed

Three high-availability replicas maintained for every database

Tables require a clustered index

Maximum database size is 500 GB

Azure SQL Database vs. SQL Server

Transact-SQL

Use Familiar Technologies

(obviously)

.NET Framework (C#, Visual Basic, F#): ADO.NET

C / C++: ODBC

Java: Microsoft JDBC provider

PHP: Microsoft PHP provider

Use Familiar Technologies - Languages

OData

Entity Framework

WCF Data Services

NHibernate (etc.)

Use Familiar Technologies - Frameworks

SQL Server Management Studio (>=2008 R2)

SQL Server command-line utilities (SQLCMD, BCP)

Visual Studio IDE for database development

Use Familiar Technologies - Tools

Use command, distributed transactions, distributed views

Service Broker

Common Language Runtime (CLR)

SQL Agent

SQL Profiler

Native Encryption

Unsupported SQL Server Features

Elastic Scale

Classic 3-tier enterprise architecture:

Scale out the front ends to multiple instances is easy

Scale the data-tier is more challenging

Web

Role

L

SQL

Worker

Role

Canonical cloud app architecture

SQL Database Considerations and Best Practices

Elastic Scale across thousands of databases via custom sharding

Scale out via .NET Client libraries consumed by customer applications to support sharded database pattern

Enables developer and manageability functions

Supports split, merge, and move operations on data

Vertical: Scale-up or scale-down

Horizontal: Scale-out or scale-in

Basic

Standard

Premium

Basic Basic Basic Basic Basic Basic

Premium

Standard

Scale out/in

Sca

le u

p/d

ow

n

Scalability options in Azure SQL DB

Web

Role

J

SQL

Worker

Role

J

J

Elastic Scale architecture

SQL Server VM

Run SQL on VM

Run any SQL product on cloud VM

Support for SQL Server, Oracle, MySQL

Ready to go VM images available in the Gallery

Persistent storage using attached disk in blob storage

Has all the benefits and powers of VMs combined with the full features of a SQL Engine

SQL Database SQL IaaS

Why (at least) two offerings of SQL in Azure?

SQL Database SQL IaaS

Other features SQL IaaS supports that SQL Database doesn’t (yet)

Microsoft Azure DocumentDB

Fully managed, scalable JSON document database service

Microsoft Azure Data Services

fully managed, scalable, queryable, schemafree JSON

document database service for modern applications

transactional processing

rich query

managed as a service

elastic scale

internet accessible http/rest

schema-free data model

arbitrary data formats

DocumentDB – Reasons to use

Ways to work with data

RESTful access methodsFor Create/Read/Update/Delete (CRUD) operations

DocumentDB SQLA query language with SQL-derived syntax

Executing logic in the database

Stored procedures

Triggers

User-defined functions (UDFs)

- Allow extending DocumentDB SQL

All written in JavaScript

Azure Search

Azure Search

Embed a sophisticated search experience into web and mobile applications without having to worry about the complexities of full-text search and without having to deploy, maintain or manage any infrastructure.

Azure Search

Perfect for enterprise cloud developers, cloud software vendors, cloud architects who need a fully-managed search solution.

Simple HTTP/JSON API for creating indexes, pushing documents, searching

Keyword search with user-friendly operators (+, -, *, “”, etc.)

Hit highlighting

Faceting (histograms over ranges, typically used in catalog browsing)

Search Functionality

Suggestions (auto-complete)

Rich structured queries (filter, select, sort) that combines with search

Scoring profiles to model search result relevance

Geo-spatial support integrated in filtering, sorting and ranking (such as finding all restaurants within 5 KM of your current location)

Search Functionality

What are customers doing with Azure Search?

high

high heelshigh topshigh arch

Azure Search doesn’t

provide any UI components

Users expect suggestions, so

Azure Search supports this

HDInsight

Data size

Access

Updates

Structure

Integrity

Scaling

Relational DB vs. Hadoop

Distributed Storage

(HDFS)

Query

(Hive)

Distributed Processing

(MapReduce)

OD

BC

Legend■ Core Hadoop

■ Data processing

■ Data Movement

■ Packages

Hadoop ecosystem

Hadoop Core +

Hive, Pig, HBase

C#, F#, .NET

Azure Storage (WASB)

Office 365 Power BI

(Excel, PowerQuery,

PowerView,

BI Sites)

World's Data (Azure Data

Marketplace)

HDInsight and Hadoop

ODBCSqoop for SQL

ServerPowerShell

Microsoft Azure

The Big Picture

The Big Picture

The traditional world:

The Big Picture

A modern view:

The Big Picture

Microsoft Azure technologies:

The Big Picture

Why Use a NoSQL Technology on Azure?

The Big Picture

Choosing a Data Technology

Resources

Resources

Microsoft Azure Essentials: Fundamentals of Azure

Michael S. Collier and Robin E. Shahan

February 2015

246 pages

http://bit.ly/1zRa1UW

Resources

• Azure Data Services series by Adarsha Datta

• http://bit.ly/1aTDSBA

• Get Ready for Global Azure Bootcamp

• http://bit.ly/1BkD0Lf

• Azure Documentation Center

• http://azure.microsoft.com/en-us/documentation/

• http://www.hadooponazure.com/