Virtual techdays INDIA │ 9-11 February 2011 virtual techdays Data grail: Data Market on Windows...

virtual techdaysINDIA │ 9-11 February 2011

virtual techdays

Data grail: Data Market on Windows Azure

Sudhindra Kovalam │ Developer, Icertis Inc.

An Introduction to Windows Azure Marketplace DataMarket V1 Why is Data important & Why would I Need something like DataMarket in the first place? Ok Cool, What sorts of data is available on the DataMarket

Developing Applications on DataMarket Using Power Pivot plugin for Microsoft Excel Developing simple applications using Data Market

Inside Windows Azure DataMarket Some more details about DataMarket DataMarket internals

Wrap-up


S E S S I O N A G E N D A

Data is the new “OIL” - Gerd Leonhard Many of the decisions we take in our day to day life are based on some sort of statistical

data. As an end-user, you always want to have some statistic or data to make decisions.

Discoverability and reliability Many sources on the internet such as blogs, hosted documents, websites etc. Making sense out of the data and basing your decisions on that data.


A N I N T R O D U C T I O N T O D AT A M A R K E T


virtual techdays

Example : A Site Selection application for a Retail store Chain


virtual techdays

W h a t’ s D a t a M a r k e t


DEVELOPING APPLICATIONS USING DATAMARKET

How does DataMarket help?

I Need Data!semantics

billing

schema

discovery hosting


virtual techdays

ENTER: MARKETPLACE DATAMARKET

DataMarket is an information marketplace for ISVs and IWs that provides both trusted public domain and premium commercial data via integrated consumption experiences and easy data discovery, exploration, and purchasing.

Content Providers

Global reach

Unified billing & provisioning platform

Easy content onboarding

Data security / authorization model

Flexible pricing, auditing, logging

ISVs and Devs

Consistent, flexible , context optimized APIs - OData

Single Contract – One Stop shop for data and insight

Easy access to premium data

Unified billing and provisioning platform

Information Workers

Easy discovery

Rich Apps to consume Data

Microsoft Office, Dynamics, Bing + 3rd party ISV Applications

Ability to mash up public and private data

Flexible pricing – pay as you go


DEVELOPING APPLICATIONS USING DATAMARKET

How does DataMarket help?

I Need Data!

semantics

billing

schema

discovery hosting

Consumer Spending

Climate

Real Estate Value

DataMarket

Marketplace

Billing

Visualization

Serv

ices f

or

Con

ten

t P

rovid

ers

Standard Data Access

(OData)


Marketplace DataMarket Content Providers

DataMarket Launch45+ Content partners60+ Data offerings100+ Offers coming soon

http://www.melissadata.com/

http://alteryx.com/

http://alteryx.com/

http://www.digitalglobe.com/digitalglobe2/index.php

http://www.digitalglobe.com/digitalglobe2/index.php

http://cdyne.com/

http://cdyne.com/

http://cdyne.com/


virtual techdays

Demo : Windows Azure Marketplace DataMarket Walkthrough

Using PowerPivot plugin for Excel 2010 PowerPivot gives users the power to create compelling self-service BI solutions Get PowerPivot from http://www.powerpivot.com/download.aspx

Via Managed Code Add “Service Reference” to Dataset end point to consume the dataset via code.


Exploring Data Market Datasets

http://www.powerpivot.com/download.aspx

http://www.powerpivot.com/download.aspx


virtual techdays

Demo : Using PowerPivot for Excel 2010

ODATA Protocol for querying and updating data. Based on AtomPub, HTTP, JSON Provides access to information from a variety of applications, services, and stores. Released under the Open Specification Promise. Visit www.odata.org for more details.

ODATA Clients Existing - Java, PHP, AJAX, .NET, Silverlight, Excel New – iPhone, Windows Phone 7 CTP, Linqpad, etc.


Developing Applications using Data Market

ODATA

http://www.odata.org/

Key Features of DataMarket APIs The Metadata service document is exposed. Useful in building a generic application which

would work across multiple DataMarket datasets. Service side filtering.

Fixed and Flexible Query Types (Controlled by the content provider) Fixed Querying

For e.g. https://api.datamarket.azure.com/Data.ashx/adatum.com/CensusInfo/GetCensusData Proxy classes cannot be generated by doing a “Add Service reference” The proxy class file is available on the dataset Details page after the subscription is purchased or subscribed to.

Flexible Querying For e.g. https://

api.datamarket.azure.com/Data.ashx/thephone-company.com/accountbalance/GetAccountBalance?phonenumber=5551234567

We can do a “Add Service reference for such datasets.


Developing Applications using Data Market

Key Features of Azure DataMarket

https://api.datamarket.azure.com/Data.ashx/adatum.com/CensusInfo/GetCensusData

https://api.datamarket.azure.com/Data.ashx/adatum.com/CensusInfo/GetCensusData

https://api.datamarket.azure.com/Data.ashx/thephone-company.com/accountbalance/GetAccountBalance?phonenumber=5551234567




virtual techdays

Demo : Consuming a DataMarket dataset via Code


Basic Building Blocks

Windows Azure compute infrastructure (VMs) SQL Azure elastic databases Windows Azure AppFabric

Access Control System (ACS)

Public APIs

DataMarket infrastructure

Web roles for exposed datasets (web services / public DataMarket APIs)

Web roles for marketplace website(ASP.NET)


Processing Requests

Public API endpoints

Work dispatcher

Prioritized queue Dataset worker Dataset



.

.

.


Caching requests

Prioritized queueData worker /

Mapping layer

Data in database

Data as web

service

Caching layer

In-memory cache

Distributed cache (Windows Azure Tables)


Connecting to Content Provider’s Data

Data worker / Mapping layer

Data source connector

SQL Azure connector

Web service connector

3rd party cloud connector

Data in database

Data as web

service

Data in 3rd

party cloud

Caching layer

SQL

REST / SOAP

REST / SOAP

OData

Mapping store CSDL


Scaling Out User DBs

System database Part1



Partitioning function

System data processing pipeline

Database access management

Partition 2


Monitoring

Discover issues before customer! Build monitoring as Windows Azure web role

Deployed to multiple data centers Executes sample queries on regular base Stores result in Windows Azure Table

Logs transferred on a regular basis to SQL Azure Reporting services to create reports Alerting on condition matching

Prioritized queue

Data worker / Mapping

layer

Data in database

Data as web

service


Securing High value data

Data processing pipeline Content provider

information

Data in databaseData as web

service

Cryptocomponent

Problem: how to secure 3rd party connection information?

Solution: Certificates! Windows Azure has dedicated

certificate pipeline Only small number of people need

access to them Connection information

encrypted with final key during publication process

Certificates to decrypt connection information on the fly

Connecting to data sources


The Big Picture

Public API endpoints

Data connector /

data mapping

SQL Azure

Data as web

service

Data in 3rd party

cloud

System databases

Request queues / processing

CachingLogging and monitoringBilling

Database access management

OData

Enrolling as a Data consumer? Visit https://datamarket.azure.com

Enrolling as a Content Provider You are in control of the data, price, and terms Easily onboard data from SQL Azure or existing web services Receive periodic reports and payouts Get in touch with the Data Market On Boarding team : [email protected]


How Do I Enroll

mailto:[email protected]

Questions? Use the “Ask a Question” button on the VTD site Or use the “Q & A Panel” on the Live Meeting console.

Contact Details Blog : http://sudheerkovalam.wordpress.com Email: [email protected]


http://sudheerkovalam.wordpress.com/


virtual techdays

Thank you

Virtual techdays INDIA │ 9-11 February 2011 virtual techdays Data grail: Data Market on Windows...

Documents

Transcript of Virtual techdays INDIA │ 9-11 February 2011 virtual techdays Data grail: Data Market on Windows...