Essbase Statistics DW: How to automatically administrate
Essbase using ODI
Ricardo Giampaoli
Rodrigo Radtke
DevEpm.com
@RZGiampaoli
@RodrigoRadtke
@DEVEPMAbout the Speakers
Giampaoli, Ricardo
• Oracle Ace
• Master in Business Administration and IT management
• EPM Consultant @ Dell
• Essbase/Planning/OBIEE/ODI Certified Specialist
• Blogger @ devepm.com
Radtke, Rodrigo
• Oracle Ace
• Graduated in Computer Engineering
• Software Developer Sr. Advisor at Dell
• ODI, Oracle and Java Certified
• Blogger @ devepm.com
DevEpm.com
@RZGiampaoli
@RodrigoRadtke
@DEVEPMWhat we'll learn
• Essbase Performance
• Essbase Caches
• Java Essbase API
• Essbase DW
• ODI Automation
• Results
DevEpm.com
@RZGiampaoli
@RodrigoRadtke
@DEVEPMEssbase Performance
• The Essbase performance has a direct relationship with the settings that we apply for each cube that should be based on the design and usage
• The main problem with Essbase statistics is that they are a snapshot of the cube, and we would have a better understanding of the cube if we had the historical information about it
• To have the historical information we need to constantly get and store the statistics somewhere (DW)
• We can analyze the historical information and keep improving the Essbase cubes especially true for BSO, the focus of this presentation
Essbase
DW
Stage Area
Analysis
ODI
DevEpm.com
@RZGiampaoli
@RodrigoRadtke
@DEVEPMEssbase Caches
• Essbase use caches to speed up calculations/retrieves/data load/data extract
Cache Type Description
Index A buffer in memory that holds index pages (.ind files)
Data file A buffer in memory that holds compressed data files (.pag files). Essbase allocates
memory to the data file cache during data load, calculation, and retrieval operations, as
needed. (only when direct I/O is in effect)
Data A buffer in memory that holds uncompressed data blocks. Essbase allocates memory to
the data cache during data load, calculation, and retrieval operations, as needed
Calculator A buffer in memory that Essbase uses to create and track data blocks during calculation
operations
Dynamic
calculator
A buffer in memory that Essbase uses to store all of the blocks needed for a calculation of
a Dynamic Calc member in a dense dimension
Essbase
DW
Stage Area
Analysis
ODI
DevEpm.com
@RZGiampaoli
@RodrigoRadtke
@DEVEPMIndex Cache
• The index Cache is used to hold in memory all used indexes since the application is turned on.
• The maximum index cache size is calculated by multiplying the number of existing Blocks * 112 bytes• A Block is the combination of all store like members from all dense dimensions.
Essbase creates a new Block when a new unique combination of data in the sparse dimensions is inserted/calculated
• The number of existing Blocks can be acquired by:
EAS MAXL ESSCMD JAVA
Right click in a cube edit
properties->Statistics
query database
sample.basic get
dbstats data_block;
select sample basic;
getdbstats;
(IEssCube) cube.getPropertyNames();
Essbase
DW
Stage Area
Analysis
ODI
DevEpm.com
@RZGiampaoli
@RodrigoRadtke
@DEVEPMIndex Cache
• The optimal index cache will vary depending of the design and use of the cube. • If the cube has a lot of historical data
the chances of these index never hit the cache are big
• For that we can define a historical percentage that we can remove from the maximum index cache to find a more realistic cache
• This historical percentage can be obtained by analyzing the outline and see how many members are in use
Years
FY10
FY11
FY12
…
FY17
FY18
FY10 Scenario
Actuals
Q1FCST
Q2FCST
Q3FCST
Q4FCST
Full Index
40%
30%
Recom
mended
Index (
30%)
Essbase
DW
Stage Area
Analysis
ODI
DevEpm.com
@RZGiampaoli
@RodrigoRadtke
@DEVEPMData File Cache
• The data file cache is used to hold compressed data files (essn.pag) in memory and is used only when direct I/O option is in effect.
• It is calculated by summing all essn.pag files, if possible; otherwise, as large as possible
• The size of the essn.pag files can be acquired by:
•EAS MAXL ESSCMD JAVA
Right click in a cube edit
properties->Storage
query database
sample.basic list all
file information;
listfiles "" "sample"
"basic";
maxl.execute("query database " +
s_application + "." + s_cube + " list all
file information");
Essbase
DW
Stage Area
Analysis
ODI
DevEpm.com
@RZGiampaoli
@RodrigoRadtke
@DEVEPMData Cache
• The data cache is used hold uncompressed data blocks in memory.
• The recommended size would be 12.5 % of the summing of all essn.pag files (or 0.125 * Data File Cache size)
• Same case as Index cache, just the working blocks are used (no historical)
• The size of the essn.pag files can be acquired by:
EAS MAXL ESSCMD JAVA
Right click in a cube edit
properties->Storage
query database
sample.basic list all
file information;
listfiles "" "sample"
"basic";
maxl.execute("query database " +
s_application + "." + s_cube + " list all
file information");
Essbase
DW
Stage Area
Analysis
ODI
DevEpm.com
@RZGiampaoli
@RodrigoRadtke
@DEVEPMCalculator Cache
• A buffer in memory that Essbase uses to create and track data blocks during calculation operations
• Essbase split the sparse dimensions in 2 groups:• Bitmap dimensions: Any sparse dimensions that can be fit inside the bitmap. Each
member combination placed in the bitmap occupies 1 bit of memory, and the entire dimension must fit inside the bitmap.
• Anchoring dimensions: The remaining sparse dimensions that do not fit into the bitmap.
• Because the calculator cache controls the size of the bitmap, the number of sparse dimensions that can fit in the bitmap and anchoring dimensions depends on the size of the calculator cache
Essbase
DW
Stage Area
Analysis
ODI
DevEpm.com
@RZGiampaoli
@RodrigoRadtke
@DEVEPMCalculator Cache
• Essbase starts with the first sparse dimension in the outline and fits as many as possible into the bitmap.
• Essbase stops the process when it cannot fit another complete sparse dimension into the bitmap.
• All the remaining sparse dimensions becomes anchoring dimensions
• The number of bitmaps used is determined by the maximum number of dependent parents for any members in the anchoring dimension• The max number of dependent parents is the same as the number of Levels in
a dimension
Essbase
DW
Stage Area
Analysis
ODI
DevEpm.com
@RZGiampaoli
@RodrigoRadtke
@DEVEPM
BITMAPBITMAP
BITMAPBITMAP
Calculator Cache
• Depending of the cache setting we can have 3 options:
Sparse
Dims
# of
Members
Dependent
Parents
S1 10 NA
S2 10 NA
S3 20 NA
S4 30 NA
S5 200 3
BITMAP
Number of Bitmaps
Dependent Parents + 2
Bitmap Size = (S1*S2*S3*S4)/8
Option Calculator Cache
Single
anchoring,
multiple
bitmaps
(10*10*20*30)/8=7500
3+2=5
7500*5=37500 bytes
Single
anchoring,
single bitmap
(10*10*20*30)/8=7500
1
7500*1=7500 bytes
Multiple
anchoring,
single bitmap
(10*10*20)/8=250
1
250*1=250 bytes
Essbase
DW
Stage Area
Analysis
ODI
DevEpm.com
@RZGiampaoli
@RodrigoRadtke
@DEVEPMCalculator Cache
• To calculate the Calculator cache we need to know:• The amount of member in each sparse dimensions
• The amount of levels in the biggest sparse dimension
• The biggest sparse dimensions should be placed in the last place on the outline
• These information can be acquired by:
EAS MAXL ESSCMD JAVA
Amount of members
Right click in a cube edit
properties->Dimensions
None None selection.executeQuery(dimension.getName(),
IEssMemberSelection.QUERY_TYPE_DESCENDANTS, 0,
null, null, null);
Levels
Double click in outline->
count the parents
None None IEssDimension dimension = (IEssDimension)
otl.getDimensions().getAt(i);
IEssIterator level = dimension.getLevels();
Essbase
DW
Stage Area
Analysis
ODI
DevEpm.com
@RZGiampaoli
@RodrigoRadtke
@DEVEPMDynamic Calculator Cache
• A buffer in memory that Essbase uses to store all of the blocks needed for calculate a Dynamic Calc member in a dense dimension
• To find the optimal size of this cache we need:• CALCLOCKBLOCK size: it is half the number of expanded blocks that fit into
the data cache
• Expanded Blocks including: Store, Never Share, Dynamic Calc members and dynamic time series members
• The largest expanded block across all databases on the machine.
• The maximum number of expected concurrent users
• Can be analysed by gathering the session info into a table and analyse the patterns but for this presentation is a constant number based in experience
Essbase
DW
Stage Area
Analysis
ODI
DevEpm.com
@RZGiampaoli
@RodrigoRadtke
@DEVEPMDynamic Calculator Cache
• To calculate the Maximum Calculator Cache we need to multiply:• CALCLOCKBLOCK: (Data Cache in bytes (already calculated) / Size of the
expanded Block in bytes) / 2
• The largest expanded Block in bytes on the server
• The maximum number of expected concurrent Users (Constant)
• These information can be acquired by:
EAS MAXL ESSCMD JAVA
Data Cache
Right click in a cube edit
properties->Storage
query database
sample.basic list
all file information;
listfiles ""
"sample"
"basic";
maxl.execute("query database " +
s_application + "." + s_cube + " list all file
information");
Amount of members
Right click in a cube edit
properties->Dimensions
None None selection.executeQuery(dimension.getNam
e(),
IEssMemberSelection.QUERY_TYPE_DESCEN
DANTS, 0, null, null, null);
Essbase
DW
Stage Area
Analysis
ODI
DevEpm.com
@RZGiampaoli
@RodrigoRadtke
@DEVEPMEssbase Caches Calculation
Cache Type Description
Index number of existing Blocks * 112 bytes = the size of database index
Data file Combined size of all essn.pag files, if possible; otherwise, as large as possible
Data 0.125 * the value of Data File Cache size
Calculator Bitmap size in bytes * Number of bitmaps
Bitmap size in bytes Max((member combinations on the bitmap dimensions / 8), 4)
Number of bitmaps Maximum number of dependent parents in the anchoring
dimension + 2 constant bitmaps
Dynamic
calculator
C * S * U
C is the value of the appropriate CALCLOCKBLOCK setting. (Data Cache in bytes / Size of the
expanded Block in bytes) / 2
S is the size of the largest expanded block across all databases on the machine. Multiply the
number of members in each dense dimension together and multiply by 8 bytes
U is the maximum number of expected concurrent users
Essbase
DW
Stage Area
Analysis
ODI
DevEpm.com
@RZGiampaoli
@RodrigoRadtke
@DEVEPMAutomating using Java Essbase API
• Almost all the information required for this presentation could be exported from Essbase by Maxl/Esscmd commands to a text files
• Java can easily connect and manipulate Essbase through its API• Java can also connect to Oracle Database, run OS commands and more all in one
single code
• Java may be easily deployed to ODI using Procedures and scheduled using ODI Operator
• Combining ODI and Java code creates a powerful and seamlesslyintegration going beyond the database boundaries
Essbase
DW
Stage Area
Analysis
ODI
DevEpm.com
@RZGiampaoli
@RodrigoRadtke
@DEVEPMIE
ssbase
IEssDomain
IEssMaxlSession
IEssOlapServer IEssOlapApplication IEssCube
IEssDimension
IEssCubeOutline IEssMember
IEssMemberSelection
Java Basic
Ora
cle
Data
Sourc
e
Connection PreparedStatement
• Since we are developing one single code we need to connect in Essbase to retrieve the information and load that in the Oracle database
• Essbase API is very similar with what we see in EAS• The structure of the classes are the “same” as we
have in a Essbase server (Server->App->Cube->Otl)
• Make it easy to find out what you are looking
Essbase
DW
Stage Area
Analysis
ODI
DevEpm.com
@RZGiampaoli
@RodrigoRadtke
@DEVEPMEssbase Stats Table
• Essbase has a lot of different stats and information that we can extract from it• Each one has it own amount of columns and metrics
• If we create one table for each kind of structure it will be very trick to maintain and harder to create any generic code
• The best way to extract information is to have just one table where we have the properties in the rows instead of columns• This way we have just one structure for all kind of information, no matter the amount of
columns that returns
Essbase
DW
Stage Area
Analysis
ODI
DevEpm.com
@RZGiampaoli
@RodrigoRadtke
@DEVEPMJava Oracle Connection
• Connecting to an Oracle DB is very simple• We just need the DB URL,
Username and Password
• Since we have a generic table we can have just one prepare statement and use it to insert all kind of Stats
Essbase
DW
Stage Area
Analysis
ODI
DevEpm.com
@RZGiampaoli
@RodrigoRadtke
@DEVEPMJava Essbase Connection
• Connecting to Essbase is also easy the only additional information is that you also need to pass the Provider Service• It may be “Embedded” or by Provider
Service
• Basically we do a sign in to the provider service and then select the olap server that we want to use
Essbase
DW
Stage Area
Analysis
ODI
DevEpm.com
@RZGiampaoli
@RodrigoRadtke
@DEVEPMJava Essbase Get DB Statistics
1. First we get all the cubes from all the apps
2. For each cube we get the its properties which is an array in a Key and a Value format
3. This information is added to the prepare Statement and executed (inserted) into the DB
Essbase
DW
Stage Area
Analysis
ODI
DevEpm.com
@RZGiampaoli
@RodrigoRadtke
@DEVEPMJava Essbase Get File Information
1. In Java we can issue any Maxl command using the “IEssMaxlSession”class
2. The result set contains Columns and rows, similar with what we get in EAS
• We need to Loop through the rows and get the columns that we need
3. This information is added to the prepare Statement and executed (inserted) into the DB
Essbase
DW
Stage Area
Analysis
ODI
DevEpm.com
@RZGiampaoli
@RodrigoRadtke
@DEVEPMJava Essbase Get Outline Information
1. We can get member information using IEssMemberSelection with a custom query or find the member directly in the outline using find member
2. The results contains a set of members that we may loop and analyze its properties• In this case we categorized the results by
storage type
Essbase
DW
Stage Area
Analysis
ODI
DevEpm.com
@RZGiampaoli
@RodrigoRadtke
@DEVEPMJava in ODI
• ODI runs Java code using Java BeanShell Technology• Tip: Create your code in a Java
IDE, test it and copy the “Main” into ODI Procedure
• Other advantage to use ODI is that we can get the connection information from topology and reuse everywhere within the package• Define the connection in the
command on source
• Get the information in the commando on target
Essbase
DW
Stage Area
Analysis
ODI
DevEpm.com
@RZGiampaoli
@RodrigoRadtke
@DEVEPMEssbase DW
• After we gather all the information we needed inside a generic stage table, it is time for us to split, organize and load this information in their historical DW tables
• For this presentation we have 3 different kind of information:• DB Statistics
• File Statistics
• Outline Statistics
• Since their structure are very different we will generate one table for each statistics
Essbase
DW
Stage Area
Analysis
ODI
DevEpm.com
@RZGiampaoli
@RodrigoRadtke
@DEVEPMEssbase DW
• Since we create a generic table to hold all extracted information in rows now we need to PIVOT the data into columns and load it into the historical tables
1. Define the columns to be Pivoted
2. Use an consolidation function on the data column
1. SUM, AVG, MIN, MAX, COUNT…
2. Specify the data to me Pivoted
3. The data MUST be a constant in the “IN” Clouse
3. Data is Pivoted
4. Data is loaded to DW table using Control Append
Essbase
DW
Stage Area
Analysis
ODI
DevEpm.com
@RZGiampaoli
@RodrigoRadtke
@DEVEPMEssbase DW
• With all 3 tables loaded with historical data we can now calculate all the caches we need
Essbase
DW
Stage Area
Analysis
ODI
DevEpm.com
@RZGiampaoli
@RodrigoRadtke
@DEVEPMCalculations
Calc Metric Calculation Result
Index Max Index (TOTAL_BLOCKS * 112) /1024 8.206.282 Kb
Data File Max Data DATA_FILE_TOTAL 3.825.872 Kb
Data Max Data (DATA_FILE_TOTAL * 0.125) 478.234 Kb
Calculator
BITMAP Ceils(exp(sum(ln(MEMBERS))))/838.250.576 b
Anchor Levels + 28
Result BITMAP * Anchor / 1024 298.832 Kb
Dynamic
Calculator
CalcLock((Data Cache * 1024) /
(CEIL(exp(sum(ln(TOTAL-SHARED-LABEL))))*8))/2
598
Max Block Max((CEIL(exp(sum(ln(TOTAL-SHARED-LABEL))))*8)) 409.488 b
Max User Static number = 50 50
Results (CalcLock * Max Block * Max User) /1024 / 1024 11.677 Mb
Essbase
DW
Stage Area
Analysis
ODI
DevEpm.com
@RZGiampaoli
@RodrigoRadtke
@DEVEPMGlue all together in ODI
• To Glue all together in ODI is simple• We may have all the java codes in
steps
• Our 3 pivot queries in 3 insert steps
• Our 5 caches in 5 steps as well creating the analyzes
• And finally a email step to send the information
Essbase
DW
Stage Area
Analysis
ODI
DevEpm.com
@RZGiampaoli
@RodrigoRadtke
@DEVEPMAnalyzes
• We could automatically set the caches using maxl/esscmd commands but this could cause a few problems:• The apps and in some case the Essbase server
needs to be restarted
• Even these settings being what Oracle recommends, each cube has its own characteristics then is too dangerous to automatically set them
• This information greatly helps the Admins to maintain the apps in a more systematic way
Essbase
DW
Stage Area
Analysis
ODI
DevEpm.com
@RZGiampaoli
@RodrigoRadtke
@DEVEPMConclusion
• Automate a task that is hard to keep vigilance (Pro active admin)
• Allow us to track cube growing (preventive hardware actions)
• Identify business seasonality (plan settings ahead based in the past)
• Can be extended for sessions and Essbase statistical tables for deeper analyzes
• Great repository to find abnormal situations
DevEpm.com
@RZGiampaoli
@RodrigoRadtke
@DEVEPM
Ricardo Giampaoli – TeraCorp
Rodrigo Radtke de Souza - Dell
Thank you!
Top Related