Where Is My Data - ILTAM Session

Tamir DresherSenior Software ArchitectJuly 2, 2014

Where is my Data? (In the Cloud)

About Me

• Software architect, consultant and instructor• Software Engineering Lecturer @ Ruppin Academic Center• Technology addict• 10 years of experience• .NET and Native Windows Programming

@[email protected]://www.TamirDresher.com.

http://www.twitter.com/tamir_dresher


mailto:[email protected]

http://www.tamirdresher.com/

Agenda

• Storage• Blob• Relational DB• NoSql DB• MapReduce

3

Storage

4

Where is my data Storage

5

Numbers – 1 Second is

• 1,132 Instagram photos uploaded


6


• 1,132 Instagram photos uploaded • 1,365 Tumblr posts


7


• 1,132 Instagram photos uploaded • 1,365 Tumblr posts • 7,241 Tweets sent


http://www.internetlivestats.com/twitter-statistics/


8


• 1,132 Instagram photos uploaded • 1,365 Tumblr posts • 7,241 Tweets sent • 44,512 Google searches




http://www.internetlivestats.com/google-search-statistics/

9


• 1,132 Instagram photos uploaded • 1,365 Tumblr posts • 7,241 Tweets sent • 44,512 Google searches• 84,921 YouTube videos viewed• http://www.internetlivestats.com/one-second/• http://onesecond.designly.com/




http://www.internetlivestats.com/google-search-statistics/

http://www.internetlivestats.com/one-second/

http://onesecond.designly.com/

http://onesecond.designly.com/

10

Storage Prices

Types of information• Product catalogs• Employee data• User profiles• Images• Session state• Shopping cart• Game scores and state

11

• Social feeds• Query output results• Airline seating charts• Inventory management system• Game leaderboards• Performance counters• Weather • Stock quotes


12

Gartner Magic QuadrantIaaS PaaS

North America Europe Asia Pacific

S. Central – U.S. Region

W. Europe Region

N. Central – U.S. Region

N. Europe Region

S.E. AsiaRegion

E. AsiaRegion

Data centers

Windows Azure Growing Global PresenceMicrosoft Azure Storage

East – U.S. Region

West – U.S. Region

Brazil South Region

Japan WestRegion

Japan EastRegion

China North

China East

Storage SLA – 99.99%52.56 minutes per year

http://azure.microsoft.com/en-us/support/legal/sla

http://azure.microsoft.com/en-us/support/legal/sla

14

AZURE BLOBS

What is a BLOB

• BLOB – Binary Large OBject• Storage for any type of entity such as binary files and text

documents• Distributed File Service (DFS)– Scalability and High availability

• BLOB file is distributed between multiple server and replicated at least 3 times

15

Where is my data BLOB

Azure Blob Storage Concepts

BlobContainerAccount

http://<account>.blob.core.windows.net/<container>/<blobname>

Pages/ Blocks

contoso

PIC01.JPG

Block/Page

Block/Page

PIC02.JPG

images

VID1.AVIvideos

16


Amazon Simple Storage Service(S3) Concepts

ObjectBucketAccount

http://<bucket>. s3.amazonaws.com/<object>

contoso

PIC01.JPG

PIC02.JPG

images

VID1.AVIvideos

17


Blob Operations

18

PutBlobGetBlobDeleteBlobCopyBlobSnapshotBlob LeaseBlob Windows Azure

Storage

REST


DEMOCreating a Blob

19

BLOBS - Azure

• Block blob - up to 200 GB in size• Page blobs – up to 1 TB in size• Total Account Capacity - 500 TB

20


21

BLOBS - AWS

• Object size – up to 5 TB• AWS account can own up to 100 buckets at a time, unlimited

objects• 99.999999999% durability, 99.99% availability• Reduced Redundancy Storage (RRS) - 99.99% durability and

99.99%• Amazon Glaciar - low-cost storage service as a storage option

for data archival.


22

Pricing - AWS

• pay for what you use• Components:– Storage capacity used (per GB per month)– Data transfer out (per GB per month)– Requests (per n thousand requests per month)

• http://aws.amazon.com/s3/pricing/

Where is my data BLOB Pricing

http://aws.amazon.com/s3/pricing/

23

Pricing - Azure

• pay for what you use or 6,12 months plan• Components– Storage capacity used (per GB per month)– Replication option (LRS, GRS, RA-GRS)– Number of requests (per n thousand requests per month)– Data egress (per GB per month)

• http://azure.microsoft.com/en-us/pricing/details/storage/

Where is my data BLOB Pricing

http://azure.microsoft.com/en-us/pricing/details/storage/

24

RELATIONAL DB

Relational Database Service (RDS)

• MySQL, Oracle, or Microsoft SQL Server in the cloud• No administrative overheads• Dedicated Hardware• High Availability• pay-as-you-grow pricing• Familiar Development Model*

* Despite missing features and some limitations - http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_SQLServer.html

25

Where is my data Relational DB

http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_SQLServer.html

http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_SQLServer.html

SQL Azure

• SQL Server in the cloud• No administrative overheads• Shared or Reserved (Dedicated) Hardware• High Availability• pay-as-you-grow pricing• Familiar Development Model*

* Despite missing features and some limitations - http://msdn.microsoft.com/en-us/library/ff394115.aspx

26


http://msdn.microsoft.com/en-us/library/ff394115.aspx

http://msdn.microsoft.com/en-us/library/ff394115.aspx

DEMOCreating and Using SQL Azure

27

PricingSQL - Azure

28


Pricing - RDS

29


• pay for what you use• Components:– Storage capacity used (per GB-month and per million I/O requests)– Deployment type - Single-AZ/Multi-AZ (AZ-Availabiity Zone)– DB instance hours (per hour)– Additional backup storage (per GB-month(– Data transfer in / out (per GB per month)

• http://aws.amazon.com/rds/pricing/

http://aws.amazon.com/rds/pricing/

Case Study - :// . /https haveibeenpwned com

30

Where is my data SQL Azure

https://haveibeenpwned.com/


• http://www.troyhunt.com/2013/12/working-with-154-million-records-on.html

• How do I make querying 154 million email addresses as fast as possible?

• if I want 100GB of SQL Server and I want to hit it 10 million times, it’ll cost me $176 a month (now its ~20$)

31

Where is my data SQL Azure


http://www.troyhunt.com/2013/12/working-with-154-million-records-on.html

http://www.troyhunt.com/2013/12/working-with-154-million-records-on.html

32

NoSql - Azure Tables, DynamoDB

NoSql

• Relational technology has long been the dominant approach for data. • Large amount of data– Scaling across many servers is challenging.

• Different kind of data on Relational DB– JSON documents – Graphs

• ACID – Atomicity, Consistency, Isolation, Durability.• CAP - Consistency, Availability, Partition tolerance.• BASE - Basic Availability, Soft-state, Eventual consistency.

33

Where is my data NoSql

NoSql

34

Where is my data NoSql

Table Storage Concepts

EntityTableAccount

contoso

Name =…Email = …

Name =…EMailAdd=

customers

Photo ID =…Date =…

photos

Photo ID =…Date =…

35

Where is my data NoSql Azure Tables

Table Storage

• Not RDBMS – No relationships between entities– NoSql

• Entity can have up to 255 properties - Up to 1MB per entity• Mandatory Properties for every entity– PartitionKey & RowKey (only indexed properties)

• Uniquely identifies an entity• Same RowKey can be used in different PartitionKey• Defines the sort order

– Timestamp - Optimistic Concurrency

• Strongly consistent36


No Fixed Schema

FIRST LAST BIRTHDATE

Wade Wegner 2/2/1981

Nathan Totten 3/15/1965

Nick Harris May 1, 1976

FAV SPORT

Canoeing

37


Table Object Model

• ITableEntity interface –PartitionKey, RowKey, Timestamp, and Etag properties– Implemented by TableEntity and DynamicTableEntity

38

// This class defines one additional property of integer type, // since it derives from TableEntity it will be automatically // serialized and deserialized. public class SampleEntity : TableEntity{ public int SampleProperty { get; set; } }


Sample – Inserting an Entity into a Table

39

// You will need the following using statementsusing Microsoft.WindowsAzure.Storage;using Microsoft.WindowsAzure.Storage.Table;

// Create the table client.CloudTableClient tableClient = storageAccount.CreateCloudTableClient();CloudTable peopleTable = tableClient.GetTableReference("people");peopleTable.CreateIfNotExists();

// Create a new customer entity.CustomerEntity customer1 = new CustomerEntity("Harp", "Walter");customer1.Email = "[email protected]";customer1.PhoneNumber = "425-555-0101";

// Create an operation to add the new customer to the people table.TableOperation insertCustomer1 = TableOperation.Insert(customer1);

// Submit the operation to the table service.peopleTable.Execute(insertCustomer1);


Retrieve

40

// Create the table client.CloudTableClient tableClient = storageAccount.CreateCloudTableClient();CloudTable peopleTable = tableClient.GetTableReference("people");

// Retrieve the entity with partition key of "Smith" and row key of "Jeff"TableOperation retrieveJeffSmith = TableOperation.Retrieve<CustomerEntity>("Smith", "Jeff");

// Retrieve entityCustomerEntity specificEntity = (CustomerEntity)peopleTable.Execute(retrieveJeffSmith).Result;


Table Storage – Important Points

• Azure Tables can store TBs of data• Tables Operations are fast• Tables are distributed –PartitionKey defines the partition– A table might be stored in different partitions on different storage

devices.

41


Pricing

42



43




• How do I make querying 154 million email addresses as fast as possible?

• [email protected] – the domain is the partition key and the alias is the row key

• if I want 100GB of storage and I want to hit it 10 million times, it’ll cost me $8 a month

• SQL Server will cost $176 a month - 22 times more expensive

44



DynamoDB

• Item can have up to 64KB per entity• Item stored on SSDs and are replicated across multiple Availability

Zones in a Region• Item has a primary key can either be a single-attribute hash key or

a composite hash-range key• Supports secondary indexes

45

Where is my data NoSql AWS

DynamoDB

DynamoDB

• Eventually-consistent reads (by default), and strongly-consistent reads (optional)

• Provisioned Throughput - the request throughput you want your table to be able to achieve– 10 units of Write Capacity (enough capacity to do up to 36,000 writes per hour)*– 50 units of Read Capacity (enough capacity to do up to 180,000 strongly

consistent reads, or 360,000 eventually consistent reads, per hour)

46


DynamoDB

Pricing

• Pay for what you use • Components:– Provisioned throughput capacity (per hour)– Indexed data storage (per GB per month)– Data transfer out (per GB per month)

• http://aws.amazon.com/dynamodb/pricing/

47


DynamoDB

http://aws.amazon.com/dynamodb/pricing/

DynamoDB

• Item can have up to 64KB per entity• Item stored on SSDs and are replicated across multiple Availability

Zones in a Region• Item has a primary key can either be a single-attribute hash key or

a composite hash-range key• Supports secondary indexes

48


DynamoDB

49

MapReduce on the Cloud

Hadoop in the cloud

• Hadoop on Azure Cloud• Some Facts:– 2013 Global mobile data traffic reached 1.5 exabytes per month – Cisco predicts 1.1 zettabytes (1000 exabyte) of internet traffic in 2016

50

Where is my data MapReduce

http://www.cisco.com/c/en/us/solutions/collateral/service-provider/visual-networking-index-vni/white_paper_c11-520862.html

MapReduce – The BigData Power

• Map – takes input and output key;value pairs

51

(Key1,Value1)(Key2,Value2)::(Keyn,Valuen)


MapReduce – The BigData Power

• Reduce – take group of values per key and produce new group of values

52

Key1:[value1-1,Value1-2…]

Key2:[value2-1,Value2-2…]

Keyn:[valueN-1,ValueN-2…]

[new_value1-1,new_value1-2…]

[new_value2-1,new_value2-2…]

[new_valueN-1,new_valueN-2…]

: :


FIRST, STORE THE DATA

Server

ServerServer

MapReduce - How Does It Work?

Files

Server


SECOND, TAKE THE PROCESSING TO THE DATA

So How Does It Work?

// Map Reduce function in JavaScript

var map = function (key, value, context) {var words = value.split(/[^a-zA-Z]/);for (var i = 0; i < words.length; i++) {

if (words[i] !== "")context.write(words[i].toLowerCase(),1);}}};

var reduce = function (key, values, context) {var sum = 0;while (values.hasNext()) {sum += parseInt(values.next());

}context.write(key, sum);};

ServerServer

ServerServer

RUNTIME

Code


55

Elastic Map Reduce (EMR)

Where is my data MapReduce EMR

• Amazon Hadoop on the Cloud• Hortonworks and Microsoft Hadoop to Windows• Cluster of EC2 • Pricing:– hourly rate for every instance hour (by instance type)– Additional EMR price per EC2 instance– http://aws.amazon.com/elasticmapreduce/pricing/

http://aws.amazon.com/elasticmapreduce/pricing/

56

HDInsight

Where is my data MapReduce HDInsight

• MS Hadoop on (not only) Azure Cloud• Hortonworks and Microsoft Hadoop to Windows• Native integration with .NET

Finding common friends

• Facebook shows you how many common friends you have with someone

• There were 1,310,000,000 active users in facebook with130 friends on average (01.01.2014)

• Calculating the mutual friends

57

Where is my data HDInsight

Finding common friends

• We can represent Friend Relationship as:

• Note that a Friend relationship is Symmetrical – if A is a friend of B then B is a friend of A

58


Someone [List of his\her friends]

Common Friends

Example of Friends file

• U1 -> U2 U3 U4• U2 -> U1 U3 U4 U5• U3 -> U1 U2 U4 U5• U4 -> U1 U2 U3 U5• U5 -> U2 U3 U4

59

Where is my data HDInsight Common Friends

Designing our MapReduce job

• Each line from the file will input line to the Mapper• The Mapper will output key-value pairs• Key: (user, friend)– Sorted, friend might be before user

• value: list of friends

60


Designing our MapReduce job - Mapper

• Each line from the file will input line to the Mapper• The Mapper will output key-value pairs• Key: (user, friend)– Sorted, friend might be before user

• value: list of friends

• Having the key sorted will help us with the reducer, same pairs will be provided together

61


Mapper Example

62


Mapper Output: Given the Line:

(U1 U2) U2 U3 U4(U1 U3) U2 U3 U4(U1 U4) U2 U3 U4

U1U2 U3 U4

Mapper Example

63




U1U2 U3 U4

(U1 U2) -> U1 U3 U4 U5(U2 U3) -> U1 U3 U4 U5(U2 U4) -> U1 U3 U4 U5(U2 U5) -> U1 U3 U4 U5

U2 U1 U3 U4 U5

Mapper Example – final result

64




U1U2 U3 U4


U2 U1 U3 U4 U5


U3 -> U1 U2 U4 U5

Mapper Output: Given the Line:(U1 U4) -> U1 U2 U3 U5(U2 U4) -> U1 U2 U3 U5(U3 U4) -> U1 U2 U3 U5(U4 U5) -> U1 U2 U3 U5

U4 -> U1 U2 U3 U5

(U2 U5) -> U2 U3 U4(U3 U5) -> U2 U3 U4(U4 U5) -> U2 U3 U4

U5 -> U2 U3 U4

Designing our MapReduce job - Reducer

• The input for the reducer will be structured as:(friend1, friend2) (friend1 friends) (friend2 friends)

• The reducer will find the intersection between the lists• Output:

(friend1, friend2) (intersection of friend1 and friend2 friends)

65


Reducer Example

66


Reducer Output: Given the Line:

(U1 U2) -> (U3 U4) (U1 U2) -> (U1 U3 U4 U5) (U2 U3 U4)(U1 U3) -> (U2 U4) (U1 U3) -> (U1 U2 U4 U5) (U2 U3 U4)(U1 U4) -> (U2 U3) (U1 U4) -> (U1 U2 U3 U5) (U2 U3 U4)(U2 U3) -> (U1 U4 U5) (U2 U3) -> (U1 U2 U4 U5) (U1 U3 U4 U5)(U2 U4) -> (U1 U3 U5) (U2 U4) -> (U1 U2 U3 U5) (U1 U3 U4 U5)(U2 U5) -> (U3 U4) (U2 U5) -> (U1 U3 U4 U5) (U2 U3 U4)(U3 U4) -> (U1 U2 U5) (U3 U4) -> (U1 U2 U3 U5) (U1 U2 U4 U5)(U3 U5) -> (U2 U4) (U3 U5) -> (U1 U2 U4 U5) (U2 U3 U4)(U4 U5) -> (U2 U3) (U4 U5) -> (U1 U2 U3 U5) (U2 U3 U4)

Creating c# MapReduce

67


Creating c# MapReduce - Mapper

68


public class CommonFriendsMapper:MapperBase{ public override void Map(string inputLine, MapperContext context) { var strings = inputLine.Split(new []{' '}, StringSplitOptions.RemoveEmptyEntries); if (strings.Any()) { var currentUser = strings[0]; var friends = strings.Skip(1); foreach (var friend in friends) { var keyArr = new[] {currentUser, friend}; Array.Sort(keyArr); var key = String.Join(" ", keyArr); context.EmitKeyValue(key, string.Join(" ",friends)); } } }}

Creating c# MapReduce - Reduce

69


public class CommonFriendsReducer:ReducerCombinerBase{ public override void Reduce(string key, IEnumerable<string> strings, ReducerCombinerContext context) { var friendsLists = strings .Select(friendList => friendList.Split(' ')) .ToList(); var intersection = friendsLists[0].Intersect(friendsLists[1]); context.EmitKeyValue(key, string.Join(" ", intersection)); }}

Creating c# MapReduce – Hadoop Job

70


HadoopJobConfiguration myConfig = new HadoopJobConfiguration(); myConfig.InputPath = "wasb:///example/data/friends/friends"; myConfig.OutputFolder = "wasb:////example/data/friends/output"; Environment.SetEnvironmentVariable("HADOOP_HOME", @"c:\hadoop"); Environment.SetEnvironmentVariable("Java_HOME", @"c:\hadoop\jvm"); var hadoop = Hadoop.Connect(clusterUri, clusterUserName, hadoopUserName, clusterPassword, azureStorageAccount, azureStorageKey, azureStorageContainer, createContinerIfNotExist); var jobResult = hadoop.MapReduceJob.Execute<CommonFriendsMapper, CommonFriendsReducer>(myConfig); int exitCode = jobResult.Info.ExitCode; // (0 – success, otherwise – failure)

Pricing

71


10 node cluster that will exist for 24 hours:• Secure Gateway Node - free.• head node - 15.36 USD per 24-hour day• 1 data node - 7.68 USD per 24-hour day• 10 data nodes - 76.80 USD per 24-hour day• Total: $92.16 USD

72

WRAP UP

Comparing the alternatives

73

Storage Type When Should you Use ImplicationsBLOB Unstructured data

Files- Application Logic Responsibility- Consider using HDInsight(Hadoop)

Relational DB Structured Relational DataACID transactions

- SQL DML+DDL- Could affect scalability- BI Abilities- Reporting

Azure Tables, DynamoDB

Structured DataLoose SchemaGeo Replication (High DR)Auto Sharding

- OData, REST- Application Logic- Responsibility(Multiple Schemas)

Where is my data Wrap Up

What have we seen

• Blobs• Relational DB• NoSql• MapReduce in the Cloud

74


What’s Next

• NoSql – MongoDB, Cassandra, CouchDB, RavenDB• Hadoop ecosystem – Hive, Pig, SQOOP, Mahout• Cache Options - Amazon ElastiCache, Azure Cache, InRole

Cache, Redis• http://blogs.msdn.com/b/windowsazure/• http://blogs.msdn.com/b/windowsazurestorage/• http://blogs.msdn.com/b/bigdatasupport/

75


http://blogs.msdn.com/b/windowsazure/

http://blogs.msdn.com/b/windowsazure/

http://blogs.msdn.com/b/windowsazurestorage/

http://blogs.msdn.com/b/windowsazurestorage/

http://blogs.msdn.com/b/bigdatasupport/

http://blogs.msdn.com/b/bigdatasupport/

Presenter contact detailsc: +972-52-4772946t: @tamir_dreshere: [email protected]: TamirDresher.comw: www.codevalue.net



mailto:[email protected]

http://www.codevalue.net/

Where Is My Data - ILTAM Session

Technology

Transcript of Where Is My Data - ILTAM Session