DirectQuery vs In-Memory in BISM - Ms SQL Girl · The new and flashy Tabular Model for Analysis...
Transcript of DirectQuery vs In-Memory in BISM - Ms SQL Girl · The new and flashy Tabular Model for Analysis...
![Page 1: DirectQuery vs In-Memory in BISM - Ms SQL Girl · The new and flashy Tabular Model for Analysis Services has been highly pitched with the in-memory (VertiPaq) capability allowing](https://reader033.fdocuments.us/reader033/viewer/2022053013/5f10b1457e708231d44a5b22/html5/thumbnails/1.jpg)
1 MsSQLGirl.com | @MsSQLGirl
DirectQuery vs In-Memory in BISM Julie Koesmarno www.MsSQLGirl.com
Sydney SQL User Group @MsSQLGirl
![Page 2: DirectQuery vs In-Memory in BISM - Ms SQL Girl · The new and flashy Tabular Model for Analysis Services has been highly pitched with the in-memory (VertiPaq) capability allowing](https://reader033.fdocuments.us/reader033/viewer/2022053013/5f10b1457e708231d44a5b22/html5/thumbnails/2.jpg)
2 MsSQLGirl.com | @MsSQLGirl
About Me
8+ years experience with SQL Server (since
SQL Server 2000)
MCITP 2008
Past projects: large scale 24/7 Data Integration,
Data Management and Data Cleansing system
Current project: Data Warehouse and Analysis
Services Tabular Model project
2
![Page 3: DirectQuery vs In-Memory in BISM - Ms SQL Girl · The new and flashy Tabular Model for Analysis Services has been highly pitched with the in-memory (VertiPaq) capability allowing](https://reader033.fdocuments.us/reader033/viewer/2022053013/5f10b1457e708231d44a5b22/html5/thumbnails/3.jpg)
3 MsSQLGirl.com | @MsSQLGirl
Agenda
Terminology
DirectQuery vs In-Memory
Hybrid Mode
Out of Scope:
Sharepoint Integration
3
![Page 4: DirectQuery vs In-Memory in BISM - Ms SQL Girl · The new and flashy Tabular Model for Analysis Services has been highly pitched with the in-memory (VertiPaq) capability allowing](https://reader033.fdocuments.us/reader033/viewer/2022053013/5f10b1457e708231d44a5b22/html5/thumbnails/4.jpg)
4 MsSQLGirl.com | @MsSQLGirl
Abstract
The new and flashy Tabular Model for Analysis Services has been highly pitched with the
in-memory (VertiPaq) capability allowing for complex queries to run very fast
What are the advantages and disadvantages of using in-memory cache to store and
query data?
Is DirectQuery the answer to address in-memory cache mode limitations?
What are the restrictions on DirectQuery tabular model?
When can a hybrid mode be used?
How would the selected mode can affect the Processing option during Deployment?
In this session, all the above questions will be answered with live demos, comparing the
two modes and the scenarios that are applicable to them. It also discusses how partition
plays an important role in DirectQuery mode and choosing the correct processing mode
when deploying a Tabular Model solution.
4 MsSQLGirl.com | @mssqlgirl
![Page 5: DirectQuery vs In-Memory in BISM - Ms SQL Girl · The new and flashy Tabular Model for Analysis Services has been highly pitched with the in-memory (VertiPaq) capability allowing](https://reader033.fdocuments.us/reader033/viewer/2022053013/5f10b1457e708231d44a5b22/html5/thumbnails/5.jpg)
5 MsSQLGirl.com | @MsSQLGirl
v
Terminology
Tabular Model Architecture and Solution
![Page 6: DirectQuery vs In-Memory in BISM - Ms SQL Girl · The new and flashy Tabular Model for Analysis Services has been highly pitched with the in-memory (VertiPaq) capability allowing](https://reader033.fdocuments.us/reader033/viewer/2022053013/5f10b1457e708231d44a5b22/html5/thumbnails/6.jpg)
6 MsSQLGirl.com | @MsSQLGirl 6
DirectQuery
Vertipaq
In-Memory
xVelocity Cached
![Page 7: DirectQuery vs In-Memory in BISM - Ms SQL Girl · The new and flashy Tabular Model for Analysis Services has been highly pitched with the in-memory (VertiPaq) capability allowing](https://reader033.fdocuments.us/reader033/viewer/2022053013/5f10b1457e708231d44a5b22/html5/thumbnails/7.jpg)
7 MsSQLGirl.com | @MsSQLGirl
Tabular Model Architecture
7
Tabular Model
7
DirectQuery In-Memory
Third Party Application Excel Power View Reporting Services
SQL Server
Databases Non SQL Server
Databases
OData Files Cloud Services
![Page 8: DirectQuery vs In-Memory in BISM - Ms SQL Girl · The new and flashy Tabular Model for Analysis Services has been highly pitched with the in-memory (VertiPaq) capability allowing](https://reader033.fdocuments.us/reader033/viewer/2022053013/5f10b1457e708231d44a5b22/html5/thumbnails/8.jpg)
8 MsSQLGirl.com | @MsSQLGirl
Tabular Model Architecture
8
Tabular Model
8
In-Memory
Third Party Application Excel Power View Reporting Services
SQL Server
Databases Non SQL Server
Databases
OData Files Cloud Services
![Page 9: DirectQuery vs In-Memory in BISM - Ms SQL Girl · The new and flashy Tabular Model for Analysis Services has been highly pitched with the in-memory (VertiPaq) capability allowing](https://reader033.fdocuments.us/reader033/viewer/2022053013/5f10b1457e708231d44a5b22/html5/thumbnails/9.jpg)
9 MsSQLGirl.com | @MsSQLGirl
Tabular Model Architecture
9
Tabular Model
9
Third Party Application Power View Reporting Services
SQL Server
Databases
DirectQuery
![Page 10: DirectQuery vs In-Memory in BISM - Ms SQL Girl · The new and flashy Tabular Model for Analysis Services has been highly pitched with the in-memory (VertiPaq) capability allowing](https://reader033.fdocuments.us/reader033/viewer/2022053013/5f10b1457e708231d44a5b22/html5/thumbnails/10.jpg)
10 MsSQLGirl.com | @MsSQLGirl
Tabular Model Solutions
10
Authoring
(SSDT) Deployment Option
Connectivity
Database
Properties
demo
![Page 11: DirectQuery vs In-Memory in BISM - Ms SQL Girl · The new and flashy Tabular Model for Analysis Services has been highly pitched with the in-memory (VertiPaq) capability allowing](https://reader033.fdocuments.us/reader033/viewer/2022053013/5f10b1457e708231d44a5b22/html5/thumbnails/11.jpg)
11 MsSQLGirl.com | @MsSQLGirl
v
Execution
DirectQuery vs In-Memory
![Page 12: DirectQuery vs In-Memory in BISM - Ms SQL Girl · The new and flashy Tabular Model for Analysis Services has been highly pitched with the in-memory (VertiPaq) capability allowing](https://reader033.fdocuments.us/reader033/viewer/2022053013/5f10b1457e708231d44a5b22/html5/thumbnails/12.jpg)
12 MsSQLGirl.com | @MsSQLGirl
Querying
12
Data Access
Query
Data Sources
12
DirectQuery
Process
In-Memory
DAX / MDX
VertiPaq SE
SQL
DAX
![Page 13: DirectQuery vs In-Memory in BISM - Ms SQL Girl · The new and flashy Tabular Model for Analysis Services has been highly pitched with the in-memory (VertiPaq) capability allowing](https://reader033.fdocuments.us/reader033/viewer/2022053013/5f10b1457e708231d44a5b22/html5/thumbnails/13.jpg)
13 MsSQLGirl.com | @MsSQLGirl
DirectQuery Security
13 13
Role
DirectQuery Impersonation
Impersonate Current User
Default (Data Source login)
Data Source Impersonation
Windows User
Service Account
DirectQuery
SQL
Query
![Page 14: DirectQuery vs In-Memory in BISM - Ms SQL Girl · The new and flashy Tabular Model for Analysis Services has been highly pitched with the in-memory (VertiPaq) capability allowing](https://reader033.fdocuments.us/reader033/viewer/2022053013/5f10b1457e708231d44a5b22/html5/thumbnails/14.jpg)
14 MsSQLGirl.com | @MsSQLGirl
In-Memory Security
14
Role
Row Level Security
Data Source Impersonation
14
VertiPaq SE
In-Memory
Process
Query
![Page 15: DirectQuery vs In-Memory in BISM - Ms SQL Girl · The new and flashy Tabular Model for Analysis Services has been highly pitched with the in-memory (VertiPaq) capability allowing](https://reader033.fdocuments.us/reader033/viewer/2022053013/5f10b1457e708231d44a5b22/html5/thumbnails/15.jpg)
15 MsSQLGirl.com | @MsSQLGirl
Other Security Consideration?
Identity flow in Sharepoint 2010
http://www.sqlpass.org/summit/2011/Speakers/CallForSpeakers/SessionDetail.
aspx?sid=2027
15
BISM file
RSDS file
Simple Tabular Model Connection String
![Page 16: DirectQuery vs In-Memory in BISM - Ms SQL Girl · The new and flashy Tabular Model for Analysis Services has been highly pitched with the in-memory (VertiPaq) capability allowing](https://reader033.fdocuments.us/reader033/viewer/2022053013/5f10b1457e708231d44a5b22/html5/thumbnails/16.jpg)
16 MsSQLGirl.com | @MsSQLGirl
v
Design
DirectQuery vs In-Memory
![Page 17: DirectQuery vs In-Memory in BISM - Ms SQL Girl · The new and flashy Tabular Model for Analysis Services has been highly pitched with the in-memory (VertiPaq) capability allowing](https://reader033.fdocuments.us/reader033/viewer/2022053013/5f10b1457e708231d44a5b22/html5/thumbnails/17.jpg)
17 MsSQLGirl.com | @MsSQLGirl
Formula Compatibility
Semantic differences
xVelocity in-memory analytics engine (VertiPaq)
SQL Server
No Calculated Columns in DirectQuery
17
![Page 18: DirectQuery vs In-Memory in BISM - Ms SQL Girl · The new and flashy Tabular Model for Analysis Services has been highly pitched with the in-memory (VertiPaq) capability allowing](https://reader033.fdocuments.us/reader033/viewer/2022053013/5f10b1457e708231d44a5b22/html5/thumbnails/18.jpg)
18 MsSQLGirl.com | @MsSQLGirl
DAX Functions
Some DAX Functions are not supported in
DirectQuery:
No equivalent calculations in relational engine
No equivalent SQL expressions
Performance of converted expressions
E.g. TOTALYTD, SAMEPERIODLASTYEAR
More information on BOL:
http://msdn.microsoft.com/en-us/library/hh213006.aspx
18
![Page 19: DirectQuery vs In-Memory in BISM - Ms SQL Girl · The new and flashy Tabular Model for Analysis Services has been highly pitched with the in-memory (VertiPaq) capability allowing](https://reader033.fdocuments.us/reader033/viewer/2022053013/5f10b1457e708231d44a5b22/html5/thumbnails/19.jpg)
19 MsSQLGirl.com | @MsSQLGirl
v
Administration
DirectQuery vs In-Memory
![Page 20: DirectQuery vs In-Memory in BISM - Ms SQL Girl · The new and flashy Tabular Model for Analysis Services has been highly pitched with the in-memory (VertiPaq) capability allowing](https://reader033.fdocuments.us/reader033/viewer/2022053013/5f10b1457e708231d44a5b22/html5/thumbnails/20.jpg)
20 MsSQLGirl.com | @MsSQLGirl
Processing Options (In-Memory)
Mode Database Table Partition
Process Default × × ×
Process Full × × ×
Process Data × ×
Process Clear × × ×
Process Defrag ×
Process Add ×
Process Recalc ×
20
Further reading:
http://msdn.microsoft.com/en-us/library/hh758414.aspx
![Page 21: DirectQuery vs In-Memory in BISM - Ms SQL Girl · The new and flashy Tabular Model for Analysis Services has been highly pitched with the in-memory (VertiPaq) capability allowing](https://reader033.fdocuments.us/reader033/viewer/2022053013/5f10b1457e708231d44a5b22/html5/thumbnails/21.jpg)
21 MsSQLGirl.com | @MsSQLGirl
Partitioning in DirectQuery
One partition in Tabular Model
Can take advantage of Partitioning and ColumnStore
Index technology on SQL Server 2012
Partitioning / ColumnStore management done at the
source database level
21
![Page 22: DirectQuery vs In-Memory in BISM - Ms SQL Girl · The new and flashy Tabular Model for Analysis Services has been highly pitched with the in-memory (VertiPaq) capability allowing](https://reader033.fdocuments.us/reader033/viewer/2022053013/5f10b1457e708231d44a5b22/html5/thumbnails/22.jpg)
22 MsSQLGirl.com | @MsSQLGirl
Partitioning in In-Memory
In-Memory can have multiple mutually exclusive
partitions
To eliminate unnecessary processing and
processor load on the AS servers
22
C u r r e n t
Mar 2012
Jan 2012
Feb 2012
…
In-Memory
Partitions Process
![Page 23: DirectQuery vs In-Memory in BISM - Ms SQL Girl · The new and flashy Tabular Model for Analysis Services has been highly pitched with the in-memory (VertiPaq) capability allowing](https://reader033.fdocuments.us/reader033/viewer/2022053013/5f10b1457e708231d44a5b22/html5/thumbnails/23.jpg)
23 MsSQLGirl.com | @MsSQLGirl
v
Pros and Cons
DirectQuery vs In-Memory
![Page 24: DirectQuery vs In-Memory in BISM - Ms SQL Girl · The new and flashy Tabular Model for Analysis Services has been highly pitched with the in-memory (VertiPaq) capability allowing](https://reader033.fdocuments.us/reader033/viewer/2022053013/5f10b1457e708231d44a5b22/html5/thumbnails/24.jpg)
24 MsSQLGirl.com | @MsSQLGirl
In-Memory - Pros
ALL functionality of Tabular Model
Calculated Columns, ALL DAX functions
Row Level security
xVelocity in-memory analytics engine
Many choices of client tool
Excel, Power View, SSRS, MDX client*
Many supported data sources
24
![Page 25: DirectQuery vs In-Memory in BISM - Ms SQL Girl · The new and flashy Tabular Model for Analysis Services has been highly pitched with the in-memory (VertiPaq) capability allowing](https://reader033.fdocuments.us/reader033/viewer/2022053013/5f10b1457e708231d44a5b22/html5/thumbnails/25.jpg)
25 MsSQLGirl.com | @MsSQLGirl
In-Memory - Cons
AS requires Memory & CPU resources
Caching
Processing
Querying
Diligent Partitioning and Processing
Out of date data
25
![Page 26: DirectQuery vs In-Memory in BISM - Ms SQL Girl · The new and flashy Tabular Model for Analysis Services has been highly pitched with the in-memory (VertiPaq) capability allowing](https://reader033.fdocuments.us/reader033/viewer/2022053013/5f10b1457e708231d44a5b22/html5/thumbnails/26.jpg)
26 MsSQLGirl.com | @MsSQLGirl
DirectQuery - Pros
Real time access
Scalable
SQL Server permission
SQL Server optimisation
Consistency
One Partition to manage
No processing required
26
![Page 27: DirectQuery vs In-Memory in BISM - Ms SQL Girl · The new and flashy Tabular Model for Analysis Services has been highly pitched with the in-memory (VertiPaq) capability allowing](https://reader033.fdocuments.us/reader033/viewer/2022053013/5f10b1457e708231d44a5b22/html5/thumbnails/27.jpg)
27 MsSQLGirl.com | @MsSQLGirl
DirectQuery - Cons
Restricted DAX functions
Restricted client tools: PowerView, SSRS
No Row Level Security
Unless implemented at the Source Database level
No Calculated Columns
27
![Page 28: DirectQuery vs In-Memory in BISM - Ms SQL Girl · The new and flashy Tabular Model for Analysis Services has been highly pitched with the in-memory (VertiPaq) capability allowing](https://reader033.fdocuments.us/reader033/viewer/2022053013/5f10b1457e708231d44a5b22/html5/thumbnails/28.jpg)
28 MsSQLGirl.com | @MsSQLGirl
v
Hybrid Mode
The best of both worlds
![Page 29: DirectQuery vs In-Memory in BISM - Ms SQL Girl · The new and flashy Tabular Model for Analysis Services has been highly pitched with the in-memory (VertiPaq) capability allowing](https://reader033.fdocuments.us/reader033/viewer/2022053013/5f10b1457e708231d44a5b22/html5/thumbnails/29.jpg)
29 MsSQLGirl.com | @MsSQLGirl
Tabular Model Solutions
29
DirectQuery Mode
On
Off
Connection String:
DirectQueryMode=DirectQuery
DirectQueryMode=In-Memory
Query Mode
DirectQuery
DirectQuery with In-Memory
In-Memory with DirectQuery
In-Memory
![Page 30: DirectQuery vs In-Memory in BISM - Ms SQL Girl · The new and flashy Tabular Model for Analysis Services has been highly pitched with the in-memory (VertiPaq) capability allowing](https://reader033.fdocuments.us/reader033/viewer/2022053013/5f10b1457e708231d44a5b22/html5/thumbnails/30.jpg)
30 MsSQLGirl.com | @MsSQLGirl
Which Hybrid Mode?
Query Mode Excel Connection String
DirectQuery with In-Memory DirectQueryMode = InMemory
In-Memory with DirectQuery No need to specify DirectQueryMode
30
PowerView always uses the default Query Mode
(i.e. the first bit of the Query Mode) – can’t
change this!
SSRS works with both Query Mode and
Connection String combinations
![Page 31: DirectQuery vs In-Memory in BISM - Ms SQL Girl · The new and flashy Tabular Model for Analysis Services has been highly pitched with the in-memory (VertiPaq) capability allowing](https://reader033.fdocuments.us/reader033/viewer/2022053013/5f10b1457e708231d44a5b22/html5/thumbnails/31.jpg)
31 MsSQLGirl.com | @MsSQLGirl
Partitioning for Hybrid Mode
One Partition for DirectQuery
Processing Option: Never process this partition
Multiple Partitions for In-Memory (No Overlap)
31
…
C u r r e n t
Aug 2012
Jun 2012
Jul 2012 In-Memory
Partitions
DirectQuery
Partitions
![Page 32: DirectQuery vs In-Memory in BISM - Ms SQL Girl · The new and flashy Tabular Model for Analysis Services has been highly pitched with the in-memory (VertiPaq) capability allowing](https://reader033.fdocuments.us/reader033/viewer/2022053013/5f10b1457e708231d44a5b22/html5/thumbnails/32.jpg)
32 MsSQLGirl.com | @MsSQLGirl
Implications
Inconsistency
Data – stale data in Cache
Calculations – semantic differences xVelocity vs SQL
Security
DirectQuery Design
Flexibility at run time / client tools
In-Memory (Excel) or DirectQuery (Real time)
http://www.mssqlgirl.com/hybrid-mode-in-tabular-bi-semantic-model-part-1.html
http://www.mssqlgirl.com/hybrid-mode-in-tabular-bi-semantic-model-part-2.html
32
![Page 33: DirectQuery vs In-Memory in BISM - Ms SQL Girl · The new and flashy Tabular Model for Analysis Services has been highly pitched with the in-memory (VertiPaq) capability allowing](https://reader033.fdocuments.us/reader033/viewer/2022053013/5f10b1457e708231d44a5b22/html5/thumbnails/33.jpg)
33 MsSQLGirl.com | @MsSQLGirl
v
Wrap Up
![Page 34: DirectQuery vs In-Memory in BISM - Ms SQL Girl · The new and flashy Tabular Model for Analysis Services has been highly pitched with the in-memory (VertiPaq) capability allowing](https://reader033.fdocuments.us/reader033/viewer/2022053013/5f10b1457e708231d44a5b22/html5/thumbnails/34.jpg)
34 MsSQLGirl.com | @MsSQLGirl
Tabular Model Modes
In-Memory Mode (default)
DirectQuery Mode
DirectQuery only
DirectQuery with In-Memory
In-Memory with DirectQuery
34
![Page 35: DirectQuery vs In-Memory in BISM - Ms SQL Girl · The new and flashy Tabular Model for Analysis Services has been highly pitched with the in-memory (VertiPaq) capability allowing](https://reader033.fdocuments.us/reader033/viewer/2022053013/5f10b1457e708231d44a5b22/html5/thumbnails/35.jpg)
35 MsSQLGirl.com | @MsSQLGirl
Do I DirectQuery?
SQL Server database is optimised
Columnstore Index
Partitioning
Contains all required business rules
High Volume
Can import non-SQL data into one SQL Server
database which will be the source
35
![Page 36: DirectQuery vs In-Memory in BISM - Ms SQL Girl · The new and flashy Tabular Model for Analysis Services has been highly pitched with the in-memory (VertiPaq) capability allowing](https://reader033.fdocuments.us/reader033/viewer/2022053013/5f10b1457e708231d44a5b22/html5/thumbnails/36.jpg)
36 MsSQLGirl.com | @MsSQLGirl
Further Reading
Cathy Dumas’ blog
http://cathydumas.com/
SQLBI – (Marco Russo & Alberto Ferrari)
http://www.sqlbi.com/
DAX resources (with links to Tabular Model)
http://social.technet.microsoft.com/wiki/contents/articles/1088.dax-resource-center.aspx
Julie Koesmarno’s blog
http://www.MsSQLGirl.com
36
![Page 37: DirectQuery vs In-Memory in BISM - Ms SQL Girl · The new and flashy Tabular Model for Analysis Services has been highly pitched with the in-memory (VertiPaq) capability allowing](https://reader033.fdocuments.us/reader033/viewer/2022053013/5f10b1457e708231d44a5b22/html5/thumbnails/37.jpg)
37 MsSQLGirl.com | @MsSQLGirl
Q & A
37
http://www.mssqlgirl.com
http://au.linkedin.com/in/juliekoesmarno
@mssqlgirl
![Page 38: DirectQuery vs In-Memory in BISM - Ms SQL Girl · The new and flashy Tabular Model for Analysis Services has been highly pitched with the in-memory (VertiPaq) capability allowing](https://reader033.fdocuments.us/reader033/viewer/2022053013/5f10b1457e708231d44a5b22/html5/thumbnails/38.jpg)
38 MsSQLGirl.com | @MsSQLGirl
v
Thank you!
![Page 39: DirectQuery vs In-Memory in BISM - Ms SQL Girl · The new and flashy Tabular Model for Analysis Services has been highly pitched with the in-memory (VertiPaq) capability allowing](https://reader033.fdocuments.us/reader033/viewer/2022053013/5f10b1457e708231d44a5b22/html5/thumbnails/39.jpg)
39 MsSQLGirl.com | @MsSQLGirl 39 MsSQLGirl.com |
@mssqlgirl