OLAP Data Mart Data Warehouse Client OLTP System(s)
-
Upload
laurence-gardner -
Category
Documents
-
view
222 -
download
1
Transcript of OLAP Data Mart Data Warehouse Client OLTP System(s)
![Page 1: OLAP Data Mart Data Warehouse Client OLTP System(s)](https://reader033.fdocuments.us/reader033/viewer/2022042522/56649dde5503460f94ad706d/html5/thumbnails/1.jpg)
![Page 2: OLAP Data Mart Data Warehouse Client OLTP System(s)](https://reader033.fdocuments.us/reader033/viewer/2022042522/56649dde5503460f94ad706d/html5/thumbnails/2.jpg)
Power Query in Modern Corporate BI Marco [email protected]
![Page 3: OLAP Data Mart Data Warehouse Client OLTP System(s)](https://reader033.fdocuments.us/reader033/viewer/2022042522/56649dde5503460f94ad706d/html5/thumbnails/3.jpg)
Who is speaking
BI Consultant & MentorFounder of www.sqlbi.comBook Writer
[email protected]: @marcorus
![Page 4: OLAP Data Mart Data Warehouse Client OLTP System(s)](https://reader033.fdocuments.us/reader033/viewer/2022042522/56649dde5503460f94ad706d/html5/thumbnails/4.jpg)
What is Power Query
![Page 5: OLAP Data Mart Data Warehouse Client OLTP System(s)](https://reader033.fdocuments.us/reader033/viewer/2022042522/56649dde5503460f94ad706d/html5/thumbnails/5.jpg)
What is Power QueryExcel Add-In for Excel 2010 and Excel 2013Free download
FeaturesData DiscoveryData Transformation
Hidden featuresLanguage “M”
![Page 6: OLAP Data Mart Data Warehouse Client OLTP System(s)](https://reader033.fdocuments.us/reader033/viewer/2022042522/56649dde5503460f94ad706d/html5/thumbnails/6.jpg)
Power Query ElementsConnectorsTransformations (UI)The “M” LanguageSharing Queries and Functions
![Page 7: OLAP Data Mart Data Warehouse Client OLTP System(s)](https://reader033.fdocuments.us/reader033/viewer/2022042522/56649dde5503460f94ad706d/html5/thumbnails/7.jpg)
Connectors
![Page 8: OLAP Data Mart Data Warehouse Client OLTP System(s)](https://reader033.fdocuments.us/reader033/viewer/2022042522/56649dde5503460f94ad706d/html5/thumbnails/8.jpg)
Connectors
![Page 9: OLAP Data Mart Data Warehouse Client OLTP System(s)](https://reader033.fdocuments.us/reader033/viewer/2022042522/56649dde5503460f94ad706d/html5/thumbnails/9.jpg)
SAP BusinessObjects BI Universe
![Page 10: OLAP Data Mart Data Warehouse Client OLTP System(s)](https://reader033.fdocuments.us/reader033/viewer/2022042522/56649dde5503460f94ad706d/html5/thumbnails/10.jpg)
Transformations
![Page 11: OLAP Data Mart Data Warehouse Client OLTP System(s)](https://reader033.fdocuments.us/reader033/viewer/2022042522/56649dde5503460f94ad706d/html5/thumbnails/11.jpg)
Transformations
![Page 12: OLAP Data Mart Data Warehouse Client OLTP System(s)](https://reader033.fdocuments.us/reader033/viewer/2022042522/56649dde5503460f94ad706d/html5/thumbnails/12.jpg)
The “M” Language
![Page 13: OLAP Data Mart Data Warehouse Client OLTP System(s)](https://reader033.fdocuments.us/reader033/viewer/2022042522/56649dde5503460f94ad706d/html5/thumbnails/13.jpg)
MFunctional language similar to F#No “real” iterative statements
Lambda functions…
Flexible, can be invoked dynamicallyExpression.Evaluate(…)
![Page 14: OLAP Data Mart Data Warehouse Client OLTP System(s)](https://reader033.fdocuments.us/reader033/viewer/2022042522/56649dde5503460f94ad706d/html5/thumbnails/14.jpg)
Sharing Queries & Functions
![Page 15: OLAP Data Mart Data Warehouse Client OLTP System(s)](https://reader033.fdocuments.us/reader033/viewer/2022042522/56649dde5503460f94ad706d/html5/thumbnails/15.jpg)
Sharing Queries and FunctionsYou can share queries and functions
Requires Office 365 account with Power BI
Published queries can be “certified”Requires Data Steward role
Accessible through “Online Search”Query is copied and executed locallyNo execution server-side
![Page 16: OLAP Data Mart Data Warehouse Client OLTP System(s)](https://reader033.fdocuments.us/reader033/viewer/2022042522/56649dde5503460f94ad706d/html5/thumbnails/16.jpg)
Corporate BI Architecture
![Page 17: OLAP Data Mart Data Warehouse Client OLTP System(s)](https://reader033.fdocuments.us/reader033/viewer/2022042522/56649dde5503460f94ad706d/html5/thumbnails/17.jpg)
The Classical BI Solution Architecture
OLAP
OLAPData Mart
Data Mart
DataWarehouse
Client
OLTP System(s)
![Page 18: OLAP Data Mart Data Warehouse Client OLTP System(s)](https://reader033.fdocuments.us/reader033/viewer/2022042522/56649dde5503460f94ad706d/html5/thumbnails/18.jpg)
The Self-Service BI Force
OLAP
OLAPData Mart
Data Mart
DataWarehouse
Client
OLTP System(s)
Self Service BI
![Page 19: OLAP Data Mart Data Warehouse Client OLTP System(s)](https://reader033.fdocuments.us/reader033/viewer/2022042522/56649dde5503460f94ad706d/html5/thumbnails/19.jpg)
What might happen
OLAP
OLAPData Mart
Data Mart
DataWarehouse
OLTP
OLTP
OLTPClient
OLTP System(s)
PowerPivot
External Sources
Power Query transformation or Power Pivot
![Page 20: OLAP Data Mart Data Warehouse Client OLTP System(s)](https://reader033.fdocuments.us/reader033/viewer/2022042522/56649dde5503460f94ad706d/html5/thumbnails/20.jpg)
What should happen
OLAP
OLAPData Mart
Data Mart
DataWarehouse
OLTP
OLTP
OLTPClient
OLTP System(s)
PowerPivot
External Sources
CO
RPO
RAT
E V
IEW
S
CERTIFIED
QUERIES
Query provided by IT
Power Query transformation or Power Pivot
![Page 21: OLAP Data Mart Data Warehouse Client OLTP System(s)](https://reader033.fdocuments.us/reader033/viewer/2022042522/56649dde5503460f94ad706d/html5/thumbnails/21.jpg)
Support for Self-Service BI
![Page 22: OLAP Data Mart Data Warehouse Client OLTP System(s)](https://reader033.fdocuments.us/reader033/viewer/2022042522/56649dde5503460f94ad706d/html5/thumbnails/22.jpg)
Provide Data to Self-Service BIUse ViewsDo not expose table directlyEvery view define a contract with a consumerKeep views valid over time – do not break existing reports and extraction
Use Natural (Business) KeysSurrogate Keys are useless to integrate data from different sources
Expose both Data Warehouse and Data MartsData Marts are usually report-orientedData Warehouse might offer better data for Self-Service BI (similar to a private Data Mart)OLTP only for real-time data (limit history available)
![Page 23: OLAP Data Mart Data Warehouse Client OLTP System(s)](https://reader033.fdocuments.us/reader033/viewer/2022042522/56649dde5503460f94ad706d/html5/thumbnails/23.jpg)
Consider Self-Service ETLSimple transformations can be “self-service”No longer necessary to apply whatever transformation in DWH/DMCombine columns, concatenate names, denormalization of lookup tablesAesthetic transformations easily done with tools like Power Query
Complex transformations should be centralizedBusiness logic belongs to the Data WarehouseWeek Calendar is ready to use, no ambiguities in usage
![Page 24: OLAP Data Mart Data Warehouse Client OLTP System(s)](https://reader033.fdocuments.us/reader033/viewer/2022042522/56649dde5503460f94ad706d/html5/thumbnails/24.jpg)
Metadata RepositorySimplify extraction of data from Data WarehouseNavigate in available dataUse ViewsClassify SourcesControl who extracts what with automatic tools
Searching for dataNavigate in available dataCloud services available (Azure DataMarket, datamarket.com, …)Searchable metadata repository required for data warehouse Consider Power Query repository in Office 365
![Page 25: OLAP Data Mart Data Warehouse Client OLTP System(s)](https://reader033.fdocuments.us/reader033/viewer/2022042522/56649dde5503460f94ad706d/html5/thumbnails/25.jpg)
Optimize Data for Columnar DatabasesControl Data NormalizationViews should present denormalized entitiesColumnar databases use high compression for low granularity columnsRelationships are much more expensive
Do not over-denormalizeHaving multiple entities in the same table makes it hard correlating data from other tablesIt is important to find the right balance – star schema is still a good guideline
![Page 26: OLAP Data Mart Data Warehouse Client OLTP System(s)](https://reader033.fdocuments.us/reader033/viewer/2022042522/56649dde5503460f94ad706d/html5/thumbnails/26.jpg)
Power Query Usage
![Page 27: OLAP Data Mart Data Warehouse Client OLTP System(s)](https://reader033.fdocuments.us/reader033/viewer/2022042522/56649dde5503460f94ad706d/html5/thumbnails/27.jpg)
Power Query RoleSelf-Service ETLTransform data using a client toolPossible execution server-side (data refresh in Power BI)Share queries and expressions (simple text file, repository available in Office 365)
Integrate queries in Power Pivot data modelsSame language regardless of data sourceGraphical user interface and interactive editorDoes not hide the query in the data modelTechnically it would be possible to analyze dependencies in a “M” expression (not yet implemented)
![Page 28: OLAP Data Mart Data Warehouse Client OLTP System(s)](https://reader033.fdocuments.us/reader033/viewer/2022042522/56649dde5503460f94ad706d/html5/thumbnails/28.jpg)
Surface exposed to Power QueryExpose TablesPROS: automatic detection of relationshipsCONS: lack of decoupling layer in Corporate BIEasier to implement, new table/columns are immediately accessible
Expose ViewsPROS: better tracking of tables/columns used – enable dependency analysisCONS: lack of automatic detection of relationshipsAct as a decoupling layer between Corporate BI and Self-Service BI, useful for handle data structure change over time
![Page 29: OLAP Data Mart Data Warehouse Client OLTP System(s)](https://reader033.fdocuments.us/reader033/viewer/2022042522/56649dde5503460f94ad706d/html5/thumbnails/29.jpg)
Data Mart vs. Data WarehouseData WarehouseExpose natural (business) keysRaw data, useful for unplanned analysisShould not be the default choice, but it could be wrong hiding it completely
Data MartExpose surrogate keysViews of single tables should:• expose application keys• hide surrogate keys for SCD Type I• show surrogate keys for SCD Type II
Might include report business logicShould be easier to use to obtain same numbers in existing Corporate reports
![Page 30: OLAP Data Mart Data Warehouse Client OLTP System(s)](https://reader033.fdocuments.us/reader033/viewer/2022042522/56649dde5503460f94ad706d/html5/thumbnails/30.jpg)
Share Queries in Power QueryPublish Queries on Power BIDifferent visibility levels: private, group, public (within the company)
Share and Search EasilyFeatures integrated in Power Query ribbon
Validation Provided by Data StewardData Stewards can certify a querySearch can be limited to certified queries
![Page 31: OLAP Data Mart Data Warehouse Client OLTP System(s)](https://reader033.fdocuments.us/reader033/viewer/2022042522/56649dde5503460f94ad706d/html5/thumbnails/31.jpg)
Demo
Data Stewardship
![Page 32: OLAP Data Mart Data Warehouse Client OLTP System(s)](https://reader033.fdocuments.us/reader033/viewer/2022042522/56649dde5503460f94ad706d/html5/thumbnails/32.jpg)
Data StewardResponsible of queries/issues related to dataIdentifies and acquires new data sourcesCreates and maintains master data definitionsMonitor published data sources for usage/relevance/qualityMaintains business metadata (descriptions) for published data sources
Connection to the IT departmentShould be a business user – knows data and business processes
![Page 33: OLAP Data Mart Data Warehouse Client OLTP System(s)](https://reader033.fdocuments.us/reader033/viewer/2022042522/56649dde5503460f94ad706d/html5/thumbnails/33.jpg)
Other Corporate Data Sources
![Page 34: OLAP Data Mart Data Warehouse Client OLTP System(s)](https://reader033.fdocuments.us/reader033/viewer/2022042522/56649dde5503460f94ad706d/html5/thumbnails/34.jpg)
Reporting ServicesOData FeedUse standard protocolCan be imported directly in Power PivotPower Query does not support OData generated by Reporting Services (currently = May 2014)
SecurityData visibility handled by the report (might have complex logic)Granularity exposed according to report designIn Power Query it would be possible to call same report with different parameters changing URL
![Page 35: OLAP Data Mart Data Warehouse Client OLTP System(s)](https://reader033.fdocuments.us/reader033/viewer/2022042522/56649dde5503460f94ad706d/html5/thumbnails/35.jpg)
Analysis ServicesOne Query, One TableAt the moment, no direct support for DAX/MDX• Coming soon, similar user interface as SAP BusinessObjects BI connectorCan be used with a query in SQL Server and a Linked ServerIn this way, it is supported also in Power Query scheduled refresh
Column NamesComplete dimension/hierarchy/level name exposed as column name in Power PivotIn Tabular, table name is always present before column nameA query can rename all the columns in a proper way
![Page 36: OLAP Data Mart Data Warehouse Client OLTP System(s)](https://reader033.fdocuments.us/reader033/viewer/2022042522/56649dde5503460f94ad706d/html5/thumbnails/36.jpg)
OData Integration in Corporate BI
![Page 37: OLAP Data Mart Data Warehouse Client OLTP System(s)](https://reader033.fdocuments.us/reader033/viewer/2022042522/56649dde5503460f94ad706d/html5/thumbnails/37.jpg)
OData feeds in Power BIPublish OData through Data Management GatewayExpose data in a controlled waySimplify connection to data source in complex environmentsSupports only SQL Server and Oracle
LimitationsCurrency/Money data type is exposed as decimalRequires physical access to database serverCurrently it is accessible only by Power Query
Excel and Power Pivot do not support Organizational AccountCannot be used to refresh Excel Expose data to organizational accounts
Does not expose relationships between tables
![Page 38: OLAP Data Mart Data Warehouse Client OLTP System(s)](https://reader033.fdocuments.us/reader033/viewer/2022042522/56649dde5503460f94ad706d/html5/thumbnails/38.jpg)
Corporate OData Feeds
38
Data Management Gateway
Power Query
(1) Using Power Query Anna connects to OData feed
(3) The Data Management Gateway connects to SQL Server using either Windows account or Database account setup by IT when creating the feed
(4) Returns Result
(5) Returns OData feed
Power BI Cloud Services
(2) Redirect to gateway
![Page 39: OLAP Data Mart Data Warehouse Client OLTP System(s)](https://reader033.fdocuments.us/reader033/viewer/2022042522/56649dde5503460f94ad706d/html5/thumbnails/39.jpg)
Expose OData in Integration ServicesSSIS Transformations On-DemandPublish SSIS Packages as a view in SQL ServerCan be exposed as OData Feed Sources using Data Management Gateway (DMG)Enable Power BI refresh of Data Sources not supported by SQL ServerEnable Power Query access to unsupported data sources
Latency and ArgumentsAcceptable SSIS execution latency (2-3 seconds on-prem, something more through DMG)SSIS parameter are defined in the view exposed (cannot be passed at execution time)
More info:http://office.microsoft.com/en-us/publish-ssis-packages-as-odata-feed-sources-HA104079177.aspx
![Page 40: OLAP Data Mart Data Warehouse Client OLTP System(s)](https://reader033.fdocuments.us/reader033/viewer/2022042522/56649dde5503460f94ad706d/html5/thumbnails/40.jpg)
SSIS Package
Expose SSIS Package as SQL Query/View
Data Source
Data Streaming Destination
OLE DB Provider for SSIS
Linked Server
SQL Query
![Page 41: OLAP Data Mart Data Warehouse Client OLTP System(s)](https://reader033.fdocuments.us/reader033/viewer/2022042522/56649dde5503460f94ad706d/html5/thumbnails/41.jpg)
![Page 42: OLAP Data Mart Data Warehouse Client OLTP System(s)](https://reader033.fdocuments.us/reader033/viewer/2022042522/56649dde5503460f94ad706d/html5/thumbnails/42.jpg)
![Page 43: OLAP Data Mart Data Warehouse Client OLTP System(s)](https://reader033.fdocuments.us/reader033/viewer/2022042522/56649dde5503460f94ad706d/html5/thumbnails/43.jpg)
![Page 44: OLAP Data Mart Data Warehouse Client OLTP System(s)](https://reader033.fdocuments.us/reader033/viewer/2022042522/56649dde5503460f94ad706d/html5/thumbnails/44.jpg)
![Page 45: OLAP Data Mart Data Warehouse Client OLTP System(s)](https://reader033.fdocuments.us/reader033/viewer/2022042522/56649dde5503460f94ad706d/html5/thumbnails/45.jpg)
![Page 46: OLAP Data Mart Data Warehouse Client OLTP System(s)](https://reader033.fdocuments.us/reader033/viewer/2022042522/56649dde5503460f94ad706d/html5/thumbnails/46.jpg)
Data Refreshin Power BI
![Page 47: OLAP Data Mart Data Warehouse Client OLTP System(s)](https://reader033.fdocuments.us/reader033/viewer/2022042522/56649dde5503460f94ad706d/html5/thumbnails/47.jpg)
Data Refresh from Power BI Sites
Data Management
Gateway
Excel Workbook in Sharepoint
OnlineGateway
Cloud Service
(1) Data refresh is triggered from Power BI Sites
(2) Connects to Gateway Cloud Service
(3) Checks whether user is authorized to perform a refresh
(4) Sends command (SQL statement, connection string) to on-premise Gateway
(5) Sends SQL to SQL Server
(6) Return Results
(7) Compress, chunk and return the data
(8) Returns data to Excel Workbook
![Page 48: OLAP Data Mart Data Warehouse Client OLTP System(s)](https://reader033.fdocuments.us/reader033/viewer/2022042522/56649dde5503460f94ad706d/html5/thumbnails/48.jpg)
Data Refresh with Power QuerySupport in Power BIScheduled Refresh on Power Query (no direct refresh)Only Oracle and SQL Server data sourcesConsider moving data to Windows Azure SQL Database and/or SQL Server on IaaS for direct refresh
Refresh on the CloudWorkload on source databases at refresh timePossible latency for accessing data on-premiseTransfer data source on the cloud, transformation on the cloud (after query folding)
![Page 49: OLAP Data Mart Data Warehouse Client OLTP System(s)](https://reader033.fdocuments.us/reader033/viewer/2022042522/56649dde5503460f94ad706d/html5/thumbnails/49.jpg)
ConclusionCorporate BI supports Self-Service BIConsume Data Warehouse and Data MartsExpose Views
Power Query can be managed by ITCertify QueryConsume Views
SQL Server as an additional data gatewayPower BI refresh only supports SQL Server and OracleUse linked server to access other providersUse SSIS to extend access to real-time/on-demand transformations
![Page 50: OLAP Data Mart Data Warehouse Client OLTP System(s)](https://reader033.fdocuments.us/reader033/viewer/2022042522/56649dde5503460f94ad706d/html5/thumbnails/50.jpg)
Related contentSession code
Title Time
[DBI-B410] Deep Dive into Power Query Formula Language
Tuesday May 13th 3.15-4.30 PM
[DBI-B324] Microsoft Power Query for Excel: Discover and Refine Your Data
Monday 3.00 PM–4.15 PM
[DBI-B212] BI Power Hour Tuesday May 13th 5PM-6.15 PM
[DBI-B211] All You Need to Know about Microsoft Power BI for Office 365
Monday 1.15 PM-2.30 PM
[DBI-B320] Interactive Data Visualization with Power View
Tuesday May13th 8.30-9.45 AM
![Page 51: OLAP Data Mart Data Warehouse Client OLTP System(s)](https://reader033.fdocuments.us/reader033/viewer/2022042522/56649dde5503460f94ad706d/html5/thumbnails/51.jpg)
Track resources
Download Microsoft SQL Server 2014 http://www.trySQLSever.com
Try out Power BI for Office 365! http://www.powerbi.com
Sign up for Microsoft HDInsight today! http://microsoft.com/bigdata
![Page 52: OLAP Data Mart Data Warehouse Client OLTP System(s)](https://reader033.fdocuments.us/reader033/viewer/2022042522/56649dde5503460f94ad706d/html5/thumbnails/52.jpg)
Resources
Learning
Microsoft Certification & Training Resources
www.microsoft.com/learning
msdn
Resources for Developers
http://microsoft.com/msdn
TechNet
Resources for IT Professionals
http://microsoft.com/technet
Sessions on Demand
http://channel9.msdn.com/Events/TechEd
![Page 53: OLAP Data Mart Data Warehouse Client OLTP System(s)](https://reader033.fdocuments.us/reader033/viewer/2022042522/56649dde5503460f94ad706d/html5/thumbnails/53.jpg)
Complete an evaluation and enter to win!
![Page 54: OLAP Data Mart Data Warehouse Client OLTP System(s)](https://reader033.fdocuments.us/reader033/viewer/2022042522/56649dde5503460f94ad706d/html5/thumbnails/54.jpg)
Evaluate this session
Scan this QR code to evaluate this session.
![Page 55: OLAP Data Mart Data Warehouse Client OLTP System(s)](https://reader033.fdocuments.us/reader033/viewer/2022042522/56649dde5503460f94ad706d/html5/thumbnails/55.jpg)
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.