ETL in data warehouse as a service enviornment

12
ETL in Data Warehouse As a Service SANIL MHATRE | DATABASE ENGINEERING | LUMERIS @SQLSUPERGURU | HTTP://SQLWITHSANIL.COM/

Transcript of ETL in data warehouse as a service enviornment

Page 1: ETL in data warehouse as a service enviornment

ETL in Data Warehouse As a ServiceSANIL MHATRE | DATABASE ENGINEERING | LUMERIS

@SQLSUPERGURU | HTTP://SQLWITHSANIL.COM/

Page 2: ETL in data warehouse as a service enviornment

Data Warehouse As a Service

Client specific

Bulk Data

Client specific Real time data

Common Data Sources

Client Specific Data Marts

USER INTERFACE

ANALYTICS

Page 3: ETL in data warehouse as a service enviornment

Current Technology Stack

• SQL Server Database Engine

• SQL Server Integration Services

• SQL Server Analysis Services

• SQL, MDX, C# , ASP.NET

• Team Foundation Server, Visual Studio

• DACPAC & ISPAC Deployments

Page 4: ETL in data warehouse as a service enviornment

Future • Hadoop

• Hbase

• MongoDB

• Java

• HiveQL, Pig Latin, Spark

• Team Foundation Server, GIT

• Jenkins, ANT

Page 5: ETL in data warehouse as a service enviornment

SSIS Catalog & Project Deployment Model

◦ SSISDB - SQL Server Database , a new repository model for Integration Services (SQL Server 2012 onwards)

◦ Folder Hierarchy for Projects and Environments

◦ Project Based Deployment

◦ Versioning

◦ Validation

◦ Execution & Logging

◦ Environments

◦ Reports

◦ Permissions

Page 6: ETL in data warehouse as a service enviornment

SSIS Catalog Creation

Page 7: ETL in data warehouse as a service enviornment

Project Deployment ModelIntegration Services Project Conversion Wizard

Page 8: ETL in data warehouse as a service enviornment

Deploying to the CatalogIntegration Services Deployment Wizard

Page 9: ETL in data warehouse as a service enviornment

ConfigurationNo Wizard ! But pretty intuitive

Page 10: ETL in data warehouse as a service enviornment

Executing Package via SQL Agent Job

Page 11: ETL in data warehouse as a service enviornment

ResourcesSSIS catalog - https://msdn.microsoft.com/en-us/library/hh479588.aspx

Creating the SSIS Catalog - https://msdn.microsoft.com/en-us/library/gg471509.aspx

Backup, Restore & Moving the SSIS catalog - https://msdn.microsoft.com/en-us/library/hh213291.aspx

SSIS Project Deployment Model - https://msdn.microsoft.com/en-us/library/hh213290.aspx

Converting SSIS Packages to Project Deployment Model - https://msdn.microsoft.com/en-us/library/ff929158.aspx

Page 12: ETL in data warehouse as a service enviornment

Thank youQuestions ?

Comments ?

Feedback ?

@SQLSUPERGURU | http://sqlwithsanil.com/