SQL Azure Federations

Patrick RileyOctober 2011, KY .NET UG

Data Virtualization

Federated System

SQL Azure Federations

“shard” [shahrd]

noun1. A single physical database in a

federated infrastructure2. A horizontal partition in a

database

“shard” “shard·ing” [shahrd] [shahrd-ing]verb1. The process of breaking an

application’s logical database into smaller chunks of data

2. Distributing chunks of data across multiple physical databases to achieve application scalability

Federation Concepts

Federation: an object which defines the federation and the federation key

Federation Root: The database which houses the federation object

Federation Key: the key used for partitioning data (ex: Customer ID) and the distribution style (range)

Atomic Unit: A single value of the federation key, always housed in the same federation member

Federation Members (shards): a physical container for a range of atomic units

Federated Tables: tables in federation members which contain partitioned data

Reference Tables: small-scale lookup data copied to each federation member

Into the weeds…Federation Syntax-- in master database, create federation root databaseCREATE DATABASE marchmadness_db-- connect to the new marchmadness_db, create the federation. this creates the first federation member. CREATE FEDERATION section_federation(s_id RANGE BIGINT)-- connect to the federation member and deploy schema to the federation memberUSE FEDERATION section_federation(0) …GOCREATE TABLE section_row (…, section_id BIGINT NOT NULL) FEDERATED ON (s_id = section_id)CREATE TABLE section_fan (…, section_id BIGINT NOT NULL) FEDERATED ON (s_id = section_id)-- create a reference table to be replicated to all membersCREATE TABLE teams (…)--When you want to scale out sections, connect to marchmadness_db and SPLIT the federationALTER FEDERATION section_federation SPLIT AT(s_id=101)-- When you want to scale in., connect to marchmadness_db and MERGE the federation dataALTER FEDERATION section_federation MERGE AT (s_id=201) 

//Create command with filtering

SqlCommand cm = new SqlCommand(“USE FEDERATION section_federation(s_id=55) ”+ ”WITH FILTERING=ON”)

cm.ExecuteNoneQuery();

SqlCommand cm = new SqlCommand(“SELECT … FROM dbo.section_row WHERE …”) cm.ExecuteQuery();…

Enzo SQL Shard Library

• Open source hybrid shard approach for SQL Azure and SQL Server

• Abstracts the physical storage• Enterprise Library caching• Task Parallel Library for fan out

queries• Round robin inserts• Limitation of caching the shard

directory• Online repartitioning will

present challenges

Entity Framework Support

• Coming soon• Challenge of managing

connections• MARS currently not supported

The “No SQL” Gene

• Google BigTable, HBase/Hadoop, Hypertable

• Amazon Dynamo, Voldemort, Cassandra, Riak

Common Characteristics• Key Value store• Runs on a large number of

commodity machines • Data is partitioned and replicated• Consistency requirement is relaxed

(because the CAP theorem, you can’t have Consistency, Availability, and Partitioning at the same time)

Links: Cihan’s blog:http://blogs.msdn.com/b/cbiyikoglu/Enzo SQL Server and SQL Azure Library:http://enzosqlshard.codeplex.com/Cloud Ninja Azure Federations sample : http://shard.codeplex.com/SQL Azure Federations with Entity Framework:http://windowsazurecat.com/2011/09/sql-azure-federations-entity-framework-code-first/PDC 2010 Presentation Lev Novik: http://channel9.msdn.com/events/PDC/PDC10/CS02

Contact: patrick@patrickmriley.nethttp://patrickmriley.net

Questions?

References:http://publib.boulder.ibm.com/infocenter/db2luw/v8/index.jsp?topic=/com.ibm.db2.ii.doc/opt/tlsset06.htmhttp://www.queplix.com/index.php?option=com_content&view=article&id=88http://blogs.msdn.com/b/cbiyikoglu/archive/2010/10/30/building-scalable-database-solution-in-sql-azure-introducing-federation-in-sql-azure.aspxhttp://www.bluesyntax.net/files/EnzoFramework.pdfhttp://blogs.msdn.com/b/cbiyikoglu/archive/2011/01/18/sql-azure-federations-robust-connectivity-model-for-federated-data.aspxhttp://social.technet.microsoft.com/wiki/contents/articles/how-to-shard-with-sql-azure.aspxhttp://enzosqlshard.codeplex.com/http://geekswithblogs.net/hroggero/Default.aspxhttp://horicky.blogspot.com/2009/11/nosql-patterns.html