BW Layered Scalable Architecture (LSA) 07
Transcript of BW Layered Scalable Architecture (LSA) 07
SAP Skills 2009 ConferenceBW Layered Scalable Architecture (LSA) –Die Referenzarchitektur zur Standardisierung von unternehmensweiten Business-Warehouse-Implementierungen
Juergen Haupt, Architect, SAP NetWeaver RIG BI EMEA02.07.2009
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 2
1. BI Maturity and Data Logistics – Motivation2. BW Layered Scalable Architecture (LSA) –
The Reference Architecture for BW on Corporate/ Global scale2.1. LSA Building Blocks2.2. LSA Data Layers2.3. LSA Data Domains2.4. LSA & Master Data
3. LSA Implementation – Unified Data Warehousing4. BW LSA Assistent Building Blocks
1. Storage - RDBMS & Columnar DMS2. Landscape: BW – Centralization & Federation
5. Summary
Agenda
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 3
BI & Data Warehouse Market
TheMarket
CompetitiveView DW/BI
Source: Gartner 2008
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 4
BI Value & Data Logistics
BI Maturity&
DataLogistic
Excellence
DW/BIChasms
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 4
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 5
Bill Inmon’s Corporate Information Factory & Enterprise Data Warehouse (EDW)
Copyright ©1999 by William H. Inmon
Enterprise Data Warehouse (EDW):A single instantiation of a data warehouse layer for the entire corporation or big parts of the organization is often called the Enterprise Data Warehouse
EDW-Keywordsoffer a ‘single version of truth’extract once & deploy manysupport the ‘unknown’
re-build new-build
controlled redundancyprovide a corporate memory
Conceptual DetailsIntegrated historical completecomprehensiveapplication neutralgranularcorporate ownednon-volatile…
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 6
The Chasm on the Way to the EDW Shore EDW + X : The Challenging X of Today’s EDWs
Broader scopeCross corporate BI/ reportingOften local BI/ standard-reporting
Mission/ business critical99.6% availability from year’s perspective8 am local time report availability
Highly restrictive TCO & TCD expectations
EDW
Principles +
Under complex conditions
Often 24x7, global operations
High volumes (data, meta data, user)
Highly volatile environment Continuous roll outContinuous BI projects
X
Today’s EDWs can only deliver on promise if development, maintenance, operations and administration is highly standardized and automated and latest technology is leveraged:
EDW + X = BW + BW Layered Scalable Architecture (LSA)
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 7
Non-SAP-Sourcedata model
BW Model-Driven Data Warehousing-BW Means DWH Standardization & Automation
Buildingblocks DWH Stores:
DTPs, DSOs
ArchitectedData Marts:InfoCubes/ BWA
SAP-Sourcedata model
Non-SAP Source BSAP Source A
BW Model-driven DWH:1. DWH data modeling
Reference data modelInfoObjects + relations
2. DWH structure modelingInfoProviders
3. DWH operations modelingExtract, load, transformTransfer, Error-handlingAdmin, Monitor
ETL/ StagingPSA, InfoPackagesData Services
map models:provided by BW
extractor
map models:user-defined
extractor
Subject-areas
BW is from all perspectives fully model-driven
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 8
Crossing the EDW Chasm:The BW Layered Scalable Architecture
SAP Customer Expectations on BW EDW:
Nestlé
Kraft FoodsArla Foods
Adidas
EdekaBeiersdorf
HenkelJapan
TobaccoPhilips
SamsungNovartis
SyngentaBASF
LandHessen
Shell Downstream
Mc Kesson
Statoil
Best Practices &Best Practices &EDW Principles EDW Principles
BW Layered Scalable Architecture (LSA) –The Reference Architecture for BW on a Corporate Scale
SAP Consulting & Industri
esSAP Dev & RIG
LSA
Nike
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 9
1. BI Maturity and Data Logistics – Motivation2. BW Layered Scalable Architecture (LSA) –
The Reference Architecture for BW on Corporate/ Global scale2.1. LSA Building Blocks2.2. LSA Data Layers2.3. LSA Data Domains2.4. LSA & Master Data
3. LSA Implementation – Unified Data Warehousing4. BW LSA Assistent Building Blocks
1. Storage - RDBMS & Columnar DMS2. Landscape: BW – Centralization & Federation
5. Summary
Agenda
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 10
The BW Layered Scalable Architecture (LSA)describes the design of
service-level oriented, scalable, best practice BW architectures founded on accepted EDW principles*.
The BW LSA serves as a reference architecture to design transparent, complete, comprehensive
customer DWH architectures (Customer LSA).
The Customer LSA describes corporate standards to build BI applications in a
performant, maintainable, flexible manner.
Enterprise Data Warehouse (EDW) And The BW Layered Scalable Architecture (LSA)
* As introduced in Bill Inmon‘s Corporate Information Factory (CIF)
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 11
Layered Scalable ArchitectureStandardized, Transparent, Efficient
architecturedarchitecturednonnon--architecturedarchitectured
Non-Architectured
D a
t a
f l o
w
D a
t a
f l o
w
LSA Architectured
Large scale BWData
Warehouses (EDWs) should
follow architecture
principles like we can observe
constructing large buildings:
standardized, scalable, no squiggles,
efficient usage of materials, earth quake
save.
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 12
DEPLOYING WEBI ON SAPSTANDARDIZATION – NOT NICE, BUT TRANSPARENT
The LSA is all about
Standardization
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 13
BI-Project-Design
BW LSA And Customer LSA
BW LSA: The Reference Architecture
Customer LSA : Standards - Handbook
BI-Project-DesignBI Project Design
Step 3:PerfectPerfect
Customer LSA
Step 4:UpdateUpdate
Customer LSA
Step 1:DesignDesign
Customer LSA
Step 2:ApplyApply
Customer LSA
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 14
Large Scale BW Scenarios and Customer Focus
1. Consolidation BW – Replacing several BWs into/ by a single BW
2. Migration BW – Redesigning/ Reengineering a BW
3. Primary BW – BW as primary source for all BI applications/ reporting and allorganizations
4. Integration BW – BW as integrated source for cross-organizational BI/ Reporting in addition to an existing BW/ DWH landscape
The scenarios overlap each other. What varies is the primary focus of the customer what again derives from the different starting position. .
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 15
Large Scale BW ScenariosDifferent Starting Points
BW1
BW2
BW3
BWn
C-BW M-BWBW-Old
P-BWnewSAP
ERP(s)
BW1
BW2
BWn
DWHn
I-BW+
Consolidation BW Migration BW
Primary BW Integration BW
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 16
BW LSA: The Reference Architecture
LSABuilding Blocks
Reference
LSA Implementation
Reference
LSAOperations Reference
Describescore structures &
definitions
Describesdesign standards to build BI applications founded on building
blocks
DescribesSupport
Scenarios
Life CyclesInformation Meta Object LSA
Meta Data ManagementNaming ConventionsOrganization (InfoAreas)
AdministrationData BaseHousekeepingMonitoring
Transport
Security
Data Staging/ Management for transactional & master data
Persistent ObjectsFlows - scheduled/ recoveryTransformationProgramming (Abap)Organization (Process Ch.)
BW LSA
Landmark Building BlocksLayersDomains Data Model &
Data Integration Assistant Building Blocks
Data QualityLandscapeETL Storage Ownership/ OrganizeDevelopment concept
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 17
1. BI Maturity and Data Logistics – Motivation2. BW Layered Scalable Architecture (LSA) –
The Reference Architecture for BW on Corporate/ Global scale2.1. LSA Building Blocks2.2. LSA Data Layers2.3. LSA Data Domains2.4. LSA & Master Data
3. LSA Implementation – Unified Data Warehousing4. BW LSA Assistent Building Blocks
1. Storage - RDBMS & Columnar DMS2. Landscape: BW – Centralization & Federation
5. Summary
Agenda
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 18
LSA Building Blocks –The Architecture Cornerstones
The LSA Building Blocks
are the cornerstones of your future architecture thus having a decisive influence on the overall success of your future BW EDW
describe the general BW EDW layout independent from concrete BI applications and BI projects.
Landmark Building Blocks deal with architecture areas that need fundamental decisions before doing implementations – they play the same role like the supporting structures (pillars & ceilings) of buildings.
Assistant Building Blocks deal with architecture areas that are normally less prior with respect to the point in time you should decide and roll out corporate standards.
LSABuilding Blocks
Reference
Describescore structures &
definitions
Landmark Building BlocksLayersDomains Data Model &
Data Integration Assistant Building Blocks
Data QualityLandscapeETL Storage Ownership/ OrganizeDevelopment concept
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 19
LSA Landmark Building BlocksLayers & Domains
There are two areas to standardize the architecture of persistent data stores:
1. Structure the data stores in a flow from PSA to InfoCubes with respect to their role and the offered services – define data layers
2. Structure (split/ collect) the data within the layers to guarantee Layer and advanced Services – define data domains
LSA ArchitecturedNon-ArchitecturedDomain
Layer
data
flow
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 20
LSA Landmark Building Blocks BW Data Model Strategy
What has a SAP BW data model strategy to consider? SAP BW has to cover reporting requirements from various organizational units1. Support of corporate and local business scenarios (SAP BW InfoProvider)
2. Support of corporate and local master data (SAP BW InfoObjects)
Group
Division A Division B Division C Division D
: SAP BW Scenarios (InfoCubes/ DS-Objects)
EnterpriseScenario
SubOrg Scenarios
SubOrg Scenarios
SAP BW data model:
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 21
LSA Assistant Building Blocks I
ETL (Extraction, Transformation, Load) – Do you expect extensively data from non-SAP sources? If the answer is ‘yes’, it is meaningful for you investigating an ETL-tool like SAP Data Integrator. If SAP systems are the initial and primary sources for your future BW EDW, you just don’t care. May be later on.
Data Quality – Do you have considerable data quality issues? If the answer is ‘yes’, it makes sense for you thinking about a Data Quality tool. Again, if integrated SAP systems are the initial and primary sources for your future BW EDW, you normally don’t care. May be later on.
Landscape - often reduced to ‘Do I need a single or a multiple BW instance’landscape. This topic has become more and more an assistant one, because of the arrival of new technologies and the transparent support by BW (BW Accelerator, Near Line Storage s. ‘Storage’).
The landscape architecture comes into focus if we have to support mission critical BI or to observe legal restrictions or with other customer specific requirementsif it comes to agile BI and local autonomy (federated landscapes, SAP Newton appliances and BW EDW)
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 22
LSA Assistant Building Blocks II
Storage – Traditionally all data of a BW DWH are hosted by an RDBMS. This is for large scale BW EDWs not adequate (also having a smaller BW you should rethink this strategy):
Rarely used data should be hosted by a Near Line Storage (NLS) tool. NLS tools compress your data offering space reduction up 95% (e.g. SAND) without destroying your Service Level Agreements (SLAs). The BW Accelerator (BWA) must be part of the overall architecture.
Ownership/ Organization – Designing, implementing and operating a BW EDW need strong governance. A BI Competency Center (BICC) should be established if not existing yet.
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 23
1. BI Maturity and Data Logistics – Motivation2. BW Layered Scalable Architecture (LSA) –
The Reference Architecture for BW on Corporate/ Global scale2.1. LSA Building Blocks2.2. LSA Data Layers2.3. LSA Data Domains2.4. LSA & Master Data
3. LSA Implementation – Unified Data Warehousing4. BW LSA Assistent Building Blocks
1. Storage - RDBMS & Columnar DMS2. Landscape: BW – Centralization & Federation
5. Summary
Agenda
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 24
‘Layered’ means horizontal structuring/ modelingThe Data-logistic is organized in a unified, service-oriented way. Parameters are:
Coverage, comprehensiveness (Process, User demands) GranularityHistory (completeness)Reusability (BI application scalability)Recovery (robustness, availability)QualityIntegrationAccess-rights (End-user)Life-cycle.....
LSA Landmark Building Blocks Data Layer/ Layering of Data
Non-Architectured
D a
t a
f l o
w
D a
t a
f l o
w
Value of Data Layers:+ Highly Transparent & Flexible
+Development, Maintenance+Administration, Operations+Database-Integration
LSA Architectured
Layer
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 25
LSA Reference Layers
LSA
Reporting Layer (Architected Data Marts)
Business Transformation LayerBusiness Transformation Layer
Operational D
ata Store
Operational D
ata Store
Data Propagation LayerData Propagation Layer
Quality & Harmonisation LayerQuality & Harmonisation Layer
Corporate MemoryCorporate Memory
Data Acquisition LayerData Acquisition Layer
Virtualization Layer
1:1 from extraction,temporary
source system service level,long term, comprehensive, complete, master the unknown
create harmonised view, guarantee quality
EDW layersApplication neutralCorporate owned Granular
BI Applications/Architected
Data Marts Layers
digestible, integrated, unified data, ready to consume
apply business logic
reporting, analysis ready abstraction near real time, operational like
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 26
LSA Reference LayersAcquisition Layer and Subsequent Layers
The LSA suggests different 3 potential layers (targets) on top of the Acquisition Layer.
1. Corporate Memory
2. Propagation Layer (note: Harmonization Layer is optional, depending on situation)
3. Operational Data Store
Why so many layers? LSA
Reporting Layer (Architected Data Marts)– Shared Master Data (InfoObjects)
Business Transformation Layer
Operational D
ata Store
Operational D
ata Store
Data Propagation Layer
Harmonization LayerCM
Data Acquisition Layer
Other view on LSA, which keepsImplementation already in mind
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 27
LSA Reference LayersAcquisition Layer and Subsequent Layers
In a BW we have to fulfill different competing SLAs (Service Level Agreements)Experience shows that a single staging approach with single persistent stores for all purposes cannot achieve this! This applies the more challenging the conditions are for BW.Note: This means on the other hand side that if we found less complex conditions the LSA Building Blocks set up can be more simple. This applies as well for an overall customer LSA set up as for specific data sources within a full blown customer LSA !
What are complex conditions andchallenging requirements?
24x7, time zoneshigh volume, not split able loadssmall recovery window (e.g. 6h)out sourcing (skills?) off shoring (skills?) high operational robustnesshigh report availability (e.g. >96%)high application flexibility....
LSA
Reporting Layer (Architected Data Marts)– Shared Master Data (InfoObjects)
Business Transformation Layer
Operational D
ata Store
Operational D
ata Store
Data Propagation Layer
Harmonization LayerCM
Data Acquisition Layer
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 28
LSA Reference LayersAcquisition Layer To Corporate Memory Layer
SourceSystems
EDW
Data flowData flow
CorporateMemory
BusinessTransformation
Layer
Reporting Layer
PropagationLayer
AcquisitionLayer
BI-Applications
Harmonize/Quality
Corporate Memory Objects complete history of loaded datautmost comprehensive manner
1:1 from Acquisition as rule of thumb in addition harmonized data may be
stored in a dedicated Corporate Memory Object after complex harmonization/ transformation
for backup reasons provide source-system service level
Enable managing all tasks, which are unforeseeable (reorganizations, basic changes..) and/ or do not happen on a regular base (recovery, new init from source)
If we have the same DataSource offered by multiple source-systems we should go for a single CM DSO. (Data lifecycle management must exist!)
NLS is the right place for CM data
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 29© SAP 2008 / Page 29
LSA Reference LayersCorporate Memory Flier
DataSource specificWrite-optimized (wo) DSO for delta loads, normal DSO + wo DSO for fullnot directly in flow to applications (branched out)
store & deploy
completehistory
comprehensive (all fields of a DataSource)additional fields to simplify administration & access
content
Request by request; no overwrite/ update of loaded recordsupdate
Criteria Characteristics
potential sources Data Acquisition Layer, (Harmonization Layer)
potential targets Data Propagation Layer (Business Transformation Layer, Reporting Layer)
reusability Yes
transformations 1:1
granularity All extracted records for delta loads (copy of delta queue)
main services source system like SLA‘Single Point of Truth’ of extracted data for delta/ changed data for fullultimate area for application recovery, new-build and DWH reorganizationmanage the unknown
Data life cycle Should mainly reside on NLS (Near Line Storage)
DW
H S
ervi
ces
Impl
emen
tO
p.
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 29
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 30
BW LSA – Layer Description Example
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 31
Detailing LSA Reference LayersAcquisition to Propagator Layer
SourceSystems
EDW
Data flowData flow
CorporateMemory
BusinessTransformation
Layer
Reporting Layer
PropagationLayer
AcquisitionLayer
BI-Applications
Harmonize/Quality
PropagatorsThis flow describes daily, weekly,.. recurring staging of data to feed finally the BI application layersPropagator DSOs offer data, which are easy to digest for BI applications on top: Easy to digest means standards like:
Unified (additive) delta i.e. data can be direct processed into InfoCubes/ BWA index
Data are integrated if the BI applications ask for integrated data
Data are local if the BI applications ask for local, not corporate integrated data
Data have no flavor with respect to specific business rule transformations but offer additional data with respect to the loaded data, which are commonly or frequently needed by the BI applications
Manageable portions of data to fulfill Report Availability, Recovery, Administration SLAs(-> Domains)
......
LSA Example
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 32
Detailing LSA Reference LayersPropagation Layer & Digestible DataThe core service of a Propagation Layer is to offer digestible data to applications i.e.:
harmonized data in the broadest sense1. integrating data: common semantics, common values2. smoothing data: common semantics, technically unified values (e.g. compounding)
unified data – consistent behaviour of propagator DSOs regardless the DataSource
trimmed to fit DataSources and Data persistency‘s toReduce data complexity for applications1. Extending data from a DataSources by looking up information, which applications
frequently ask for. Note: introduced parent-child relationship must be stable otherwise realignment issues!
2. Merging different but highly related data from different DataSources in a single propagator, If application always or frequently request them together (e.g. HR InfoTypes, avoiding extractor enhancements)
Provide sound data portions for better support of application services (availability etc) 3. Collecting data from the same (or similar) DataSource but from different source systems
to less or a single source system independent propagator (s) ( LSA domains)4. Splitting data from a DataSources into multiple persistency’s with identical structure (
LSA domains)
LSA Example
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 33
Detailing LSA Reference LayersPropagation Layer & Trimming Data
DataSource ASource 2
DataSource ASource 2
DataSource ASource 1
DataSource ASource 1
‘DataSource A’Propagator
‘DataSource A’Propagator
3. Collect
DataSource BDataSource BDataSource ADataSource A
‘DataSource A & B’Propagator
‘DataSource A & B’Propagator
2.Merge
DataSource ADataSource A
‘DataSource A +’Propagator
‘DataSource A +’Propagator
1.ExtendAdd data
DataSource ADataSource A
‘DataSource A’Propagator 1‘DataSource A’Propagator 1
4.Split
‘DataSource A’Propagator 2‘DataSource A’Propagator 2
Note on Collecting and Splitting DataSources: This is very close related to LSA Domains!Both may not be applied without regarding
volume of data!
LSA Example
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 34
LSA Reference LayersData Propagation Layer Flier
‚normal‘ DSO in overwritesemantical/ logical partitioned for large scale DWH/ time-zone support
store & deploy
Minimum history defined by requirements of target-applications/ dependent fromCorporate Memory existence
history
DataSource specificas comprehensive as possible, if propagator is expecting volatile requiermentsMerge of different DataSources to reduce complexity
content
driven by BI application requirements (report availability)update
Criteria Characteristics
potential sources Data Acquisition Layer, Harmonization Layer, Corporate Memory
potential targets Business Transformation Layer, Reporting Layer
reusability Yes
transformations Additional, stable fields to increase (re-)useability & accessibility (e.g. currencytranslation). No application-specific rules!
granularity single records, granularity defined by DataSource business-key
main services ‘Single Point of Truth’ for BI applications (Business Transf. & Reporting Layers)Provide digestible (additive delta, content, performance) data for BI applicationsapplication recovery, rebuilt
housekeeping Regular delete of DSO change-log content
DW
H S
ervi
ces
Impl
.O
p.LSA Example
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 35
Detailing LSA Reference LayersSub-Layers of Reporting Layer
Access Abstraction-/Virtual Layer
Reporting Layer
Analytics-/Dimensional Layer
Flexible Reporting/Granular Reporting
LayerCharacteristics:
highly granular highly comprehensive
short life cycleflat, multidimensional
Characteristics:less granular
less comprehensivelong life cycle
multidimensionaloptimized performance
Characteristics:abstract from physics
‘virtual’flexible ‘view’ generation
protect front-end investments
Planning Layer
Characteristics:dedicated for planning direct input structures
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 36
From Reference LSA to Customer LSA Example: Customer Defined Layers
LSA
Reporting Layer (Architected Data Marts)
Reporting Layer (Architected Data Marts)
Business Transformation Layer
Operational D
ata StoreO
perational Data Store
Data Propagation Layer
Quality & Harmonisation Layer
Corporate Memory
Data Acquisition LayerData Acquisition Layer
Reference LSA
Customer LSA
AcquisitionLayer
CorporateMemory
Layer
PropagationLayer
BusinessTransformation
Layer
FlexibleLayer
DimensionalLayer
VirtualLayer
(YADSSnnn)
YCDSSnnn
YPDSSnnn YBAPPnnn
YFAPPnnn
YVAPP1nn
YVAPP2nn
*
YDAPPnnn
D a t a f l o w lookup
1:1Unlink
UnflavoredIntegratedGranular
Ready to consume
Apply business-logic
ReportingGranular
ReportingPerformance
Long term
AbstractionFlexible
CompleteComprehensiveMost granular
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 37
1. BI Maturity and Data Logistics – Motivation2. BW Layered Scalable Architecture (LSA) –
The Reference Architecture for BW on Corporate/ Global scale2.1. LSA Building Blocks2.2. LSA Data Layers2.3. LSA Data Domains2.4. LSA & Master Data
3. LSA Implementation – Unified Data Warehousing4. BW LSA Assistent Building Blocks
1. Storage - RDBMS & Columnar DMS2. Landscape: BW – Centralization & Federation
5. Summary
Agenda
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 38
LSA Landmark Building Blocks Data Domains
Non-Architectured
D a
t a
f l o
w
D a
t a
f l o
w
LSA Architectured
Layer
Domain
Domains means structuring/ modeling of data within the layers Transparent, disjunct structuring of transactional data using stable criterion.Target is the support of:
Independency/ autonomy of organizations 24x7, time-zonesScalability / performance/ low latency(parallel vers sequential)Challenging recovery-windowEmbedding into RDBMS Implementation & Operational robustness
Value of Data Domains:+ Transparency & Flexibility
+Development, Maintenance+Administration, Operations
+ Scalability & Robustness+Application+Load/ Query Performance+Database-Integration
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 39
EDW Incremental Implementation
2. Incremental in terms of organizational coverage – organizational roll-out
An EDW implementation is always an incremental one, never a big-bang.
1. Incremental in terms of functional coverage – BI application roll-out
Dem
and Planning
Generating D
emand
Procure 2 P
ay
CO
PA
..... .....
EDW
nBI Application Coverage & EDW
Org U
nit A
Org U
nit B
Org U
nit C
Org U
nit N
..... .....EDW
nOrganizational Coverage & EDW
Needs Scalability that is addressed by (EDW)-
layers
Needs Scalability that is addressed by
domains (& data model)
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 40
BW EDW Data Domains Consolidation BW
EuropeJapan
Asia Pacific
ERPERP ERPERPERPERP
ERPERP
ERPERP
BWBWBWBW
BWBW
BWBW
BWBW ERPERP
North America
South America
A BW EDW replaces a bunch of existing BWs and/ or legacy DWHs (BI Consolidation) spread across the organization
To enable comparable services like we had in a distributed, multiple DWH instance world (yes, there are some nice things) we introduce Data Domains in a BW EDW that
divide the transactional data but use identical meta data & master data definitions..BWBW
Using Domains in a BW EDW stands for manageability & flexibilityDomains allow SLAs in a BW EDW like in a distributed BW world
X
X
Domain AmericasDomain Americas
XX
Domain EuropeDomain Europe
X
X
Domain Asia PacificDomain Asia PacificBW EDWBW EDW
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 41
BW EDW Data Domains Support of Enterprise ERP Rollout (Primary BW)
EuropeJapan
Asia Pacific
North America
South America
A single BW EDW shall offer standard reporting & analytics for all organizational units in a global ERP rollout.
To enable comparable services like we had in a distributed, multiple BW instance world we introduce Data Domains in a BW EDW that divide the transactional data but
use identical meta data & master data definitions.
Using Domains in a BW EDW stands for manageability & flexibilityDomains allow SLAs in a BW EDW like in a distributed BW world
AMSAMS GermanyGermany APAAPAUSUS EMEAEMEA
Global ERP
BW EDWBW EDW
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 42
BW EDW Data Domains Divide Data by Sources-‘Quality’ (Integration BW)
BW EDWBW EDW
mainERP
mainERP
remoteERP 1
remoteERP 1
remoteERP 2
remoteERP 2
Main Domain Main Domain Remote Domain Remote Domain
less stable, no controlstable, controlled
Using Domains in a BW EDW stands for manageability & flexibilityDomains allow SLAs in a BW EDW like in a distributed BW world
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 43
In Short: Domains makes always sense keeping large BW EDWs manageable & flexible
Dom
ain
Wes
tD
omai
n W
est
Dom
ain
Cen
tral
Dom
ain
Cen
tral
Dom
ain
East
Dom
ain
East
BW EDWBW EDWCompany
is operating In North America
Using Domains in a BW EDW stands for manageability & flexibilityDomains allow SLAs in a BW EDW like in a distributed BW world
North America
South America
Domain South Domain South AMSAMS
Company isexpanding to
South America
Dom
ain
Dom
ain
Wes
tW
est
Dom
ain
Dom
ain
Cen
tral
Cen
tral
Dom
ain
Dom
ain
East
East
BW EDWBW EDWEurope
North America
South America
Domain South Domain South AMSAMS
Dom
ain
Dom
ain
Wes
tW
est
Dom
ain
Dom
ain
Cen
tral
Cen
tral
Dom
ain
Dom
ain
East
East
Domain Domain EUEU
Company isexpanding to
Europe
BW EDWBW EDW
LSA Example
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 44
Transparent, Scalable Structuring Your BW: LSA Domains & Layers
LSA
Reporting Layer (Architected Data Marts)Reporting Layer (Architected Data Marts)
Business Transformation LayerBusiness Transformation Layer
Data Propagation LayerData Propagation Layer
Quality & Harmonisation LayerQuality & Harmonisation Layer
Corporate
Memory
Corporate
Memory
Data Acquisition LayerData Acquisition Layer
Access Abstraction Layer(MultiProvider)
Access Abstraction Layer(MultiProvider)
Single Source System (Layer)Single Source System (Layer)
LSA
Reporting Layer (Architected Data Marts)Reporting Layer (Architected Data Marts)
Business Transformation LayerBusiness Transformation Layer
Data Propagation LayerData Propagation Layer
Quality & Harmonisation LayerQuality & Harmonisation Layer
Corporate
Memory
Corporate
Memory
Data Acquisition LayerData Acquisition Layer
Access Abstraction Layer(MultiProvider)
Access Abstraction Layer(MultiProvider)
Multiple Source Systems (Layer)
Distribution actively designed:
Domains
Distribution inherited
LSA Domains distribute the transactional data for the entire BW EDW or certain areas (flows) in a disjunctive manner. The meta data definitions of domains are common.
The LSA addresses an evolutionary EDW approach introducing Data Domains to support ‘local’ BI services without neglecting the
broader EDW picture.
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 45
Criteria Defining LSA Domains
Golden rule defining domains:
as many domains as necessary – as less domains as possible
Defining Domains – rules of thumb
Often a geography driven domain concept works well Often one basic domain per continent makes sense ( rough time zone handling)e.g. APA, EMEA, Americas
time zone aspects may lead to e.g. 3 Asian domains East-Asia, Mid-Asia, West-AsiaE.g. for continental BW EDW a starting point could be East, Middle, West...
Expected Volume (realistic volume estimates are a key input defining domains )Large APA volume contribution & large (for your business important) countries may get an own domain e.g. China and US
Independency (special service level agreement) for certain countriesImportant countries/ markets get an own domain
Robustness: take Quality/ stability of different sources into consideration(potential) instable data offerings from sources may lead to an own domain
Each customer may have additional criteria, normally it is a mixture of multiple items
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 46
LSA Building Blocks Define a GridExample for Naming Conventions
Acquisition/PSA Layer
CorporateMemory
Layer
PropagationLayer
BusinessTransformation
Layer
FlexibleLayer
DimensionalLayer
VirtualLayer
(YADSS100)
YCDSS100
YPDSS1DX
YPDSS1GX
YPDSS1WX
YPDSS1UX
YBAPP1AX
YBAPP1DX
YBAPP1GX
YBAPP1WX
YBAPP1UX
YFAPP1AX
YFAPP1DX
YFAPP1GX
YFAPP1WX
YFAPP1UX
YDAPP1AX
YDAPP1UX
YVAPP1XX
YVAPP1XX
*
*
*
*
*
YDAPP1WX
YDAPP1DX
YDAPP1GX
Lookup-tables
Asia
Europe
Americas
Germany
US
YPDSS1AX
YPDSS1AX:Y : OwnerP : LAYERDSS : Area (DSS: DataSource/ APP: Application1 : Sequence-noA : DOMAINX : Further Partitioning
Controltables
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 47
Domains and Source Systems Defining Adequate Domains
...... ......
......
......Acquisition
Propagation
Acquisition
Propagation
DataSource from single corporate source-system
Same DataSource from multiple source-systems
?
??
Nodomains
Nodomains
adequatedomains
adequatedomains
Too manydomains
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 48
BW Customer LSA - Examples
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 49
1. BI Maturity and Data Logistics – Motivation2. BW Layered Scalable Architecture (LSA) –
The Reference Architecture for BW on Corporate/ Global scale2.1. LSA Building Blocks2.2. LSA Data Layers2.3. LSA Data Domains2.4. LSA & Master Data
3. LSA Implementation – Unified Data Warehousing4. BW LSA Assistent Building Blocks
1. Storage - RDBMS & Columnar DMS2. Landscape: BW – Centralization & Federation
5. Summary
Agenda
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 50
LSA & Master DataSituation
Master data have two main tasks to fulfill:– Being target of lookups during transactional data load– Being the shared dimensions of InfoCubes for reporting (MultiProvider
Today InfoObject hosted master data (P,Q,S,X,Y tables) serve for both purposes.
InfoObjectMaster data
Shared Dimension(Navigational Attributes)
Lookups
Transactional loads
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 51
Master Data From Data Mart to BW Level/ LSA Managed MD I
Process ChainData Mart A
Sub-Chain:Transactional
loads
Process ChainData Mart B
Sub-Chain:Master data
Load e.g.0CUST_SALES + Change Run
Sub-Chain:Transactional
loads
1:001:15
With non-architected BWs the master data loads are often managed on BI application/ Data Mart level what leads to redundant, uncontrolled master data processing.This is unacceptable for large scale BWs
Data Mart level managed master data:
Sub-Chain:Master data Load e.g.
0CUST_SALES + Change Run
2:00
Process ChainMaster Data
Master data Load e.g.
0CUST_SALES + Change Run
Sub-Chain:Transactional
loads
Process ChainData Mart A
Process ChainData Mart B
Sub-Chain:Transactional
loads
Step1: BW level managed master data:
Master data should be collected/ prioritized and loaded across the Data Marts thus become BW/ LSA managed
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 52
Master Data LSA Reference Layers & Master Data
LSA
Reporting Layer (Architected Data Marts)
Data Propagation/CM LayerData Propagation/CM Layer
Quality & Harmonisation LayerQuality & Harmonisation Layer
Data Acquisition LayerData Acquisition Layer
InfoObject tables
Master Data DSO
Shared Dimension(Navigational Attributes)
Lookups
Example shows master data with delta load, master data withfull loads need additional ‘assistant DSO’ to determine delta
Step3: BW level managed master data &LSA
Large scale BWs should layer the master data allowing
1. Separation of staging and reporting tasks
InfoObject tables for reportingPropagator DSO for staging
2. Storing master data history (introducing ‘active from’ ‘active-to’ time-slice in Propagator DSO)
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 53
Master Data From Data Mart to BW Level/ LSA Managed MD III
Process ChainsMaster Data
InfoObject Change Run
Propagator/Corporate Mem Process Chain
Data Mart AProcess ChainData Mart B
Data Mart LayerB Trans Layer
Step3: BW LSA managed master data
Master data become BW/ LSA managed (Strategic Approach)
EDW transactional & master data loads are as far as possible decoupled from Data Mart loads
Process ChainsEDW Master Data
Propagator for0CUST_SALES
Process ChainsEDW Transactional
Data
Data Mart LayerB Trans Layer
InfoObject Update*
InfoObject Update*
* both possible
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 54
1. BI Maturity and Data Logistics – Motivation2. BW Layered Scalable Architecture (LSA) –
The Reference Architecture for BW on Corporate/ Global scale2.1. LSA Building Blocks2.2. LSA Data Layers2.3. LSA Data Domains2.4. LSA & Master Data
3. LSA Implementation – Unified Data Warehousing4. BW LSA Assistent Building Blocks
1. Storage - RDBMS & Columnar DMS2. Landscape: BW – Centralization & Federation
5. Summary
Agenda
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 55
LSA/ EDW Implementation ReferenceLessons Learned
Standardize data management as much as possible regardless the origin of data
Observe 80:20 rule – first provide guidelines for core BI application requirementsImplementations standards are developed incrementallyExceptions to implementation guidelines must be approvedThe more exceptions the less robustness and the higher TCOThe bigger the expected EDW (meta data) will become the more generic the implementation must be
Anticipate growth – implementation standards must be able to manage growth Avoid serialization of data processing – parallelize data flows
Strategic – follow customer domain concept (general logical/ semantical partitioned implementation)Strategic + Tactical – expand domain concept by tactical parallelization to meet individual application requirementsTactical – no general domains chosen – use parallelization to meet individual application requirementsBranch out services – observe core services an put other services aside the main dataflow
Advertise & Train the idea of Customer LSA and implementation guidelines
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 56
LSA and BW DataWarehouse WorkbenchLayer & Domains Example
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 57
LSA EDW Layers & Data Unification
A main task of the LSA EDW-Layers is the unification of data. Unification means:
1. All records get the same kind of additional information making data more transparent (stamping)
2. Propagators should behave consistently
A consistent Propagator behaviour should be the target for largescale BWs for overall robustness and consistency reasons: consistency of data marts, ease of operations.
Unification should apply to transactional & master data
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 58
LSA EDW Layers & Data UnificationStamping Data
1. All records contain the same kind of additional information making data more transparent (stamping)1. Origin (0SOURSYSTEM)2. YBWORG (criterion that drives domain partitioning)3. YLPART (domain)4. YLDAT (load date)5. Others (e.g. Request-ID)
YLPARTDomaine.g. EU
YBWORGDomaindriving
BW criterionE.g. market
Source OrganizationalCriterion
company codesales organization cost centeremployee.....
1:n 1:n-------<< --------------<<
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 59
Split and Collect ControlData Unification & Domain Characteristics
BW EDWAll data
Domain ‘B’EMEA
Domain ‘A’APA
Domain ‘C’AMS
Country/ MarketE.g. ‘FR’
E.g. ‘UK’
,,,,,,,,
Organizational-unit0COMPCODE= FR01
0COMPCODE= FR02
In BW EDW all loaded records will be qualified/ ‘stamped’ assigning a stable ‘domain- driving’characteristic like market/ country to organizational criteria like in this example0COMPCODE coming fromsource system.
This allows easy redistribution of a domain data if service levelscannot be kept
assign
assignYBWORG
YIOBJECT
YLPART
YLPART 1:n YBWORG 1:n YIOBJECT
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 60
LSA EDW Layers & Data UnificationDifferent Load-Types
Data Propagation LayerData Propagation Layer
∆via
queue
∆via
generic
∆viafull
moving∆viafull
completefull
incompletefull
Unified BI application experience – additive delta
?
2. Propagators should behave consistently
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 61
LSA EDW Layers & Data UnificationIncomplete Full - Proposal
PSA
A-table Ch-Log
Acti-Queue
1. Transfer data to Propagator (1) DSO with added load timestamp and activate Propagator
2. Via generic extractor read all records where load timestamp is older than last load timestamp and load to Propagator DSO (2) • set all key-figures of selected records to zero
3. Activate Propagator DSO (3)• Propagator offers now complete additive delta
4. Transfer to application Layer (InfoCubes) (4)
Situation: Extractor offers full loads. A DSO cannot calculate a delta with respect to last load as the new full load does not contain records, which were deleted in the source or contain zero bookings
LSA Example
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 62
LSA Layer ImplementationPropagation Layer & Trimming Data
DataSource ASource 2
DataSource ASource 2
DataSource ASource 1
DataSource ASource 1
‘DataSource A’Propagator
‘DataSource A’Propagator
3.Collect
DataSource BDataSource BDataSource ADataSource A
‘DataSource A & B’Propagator
‘DataSource A & B’Propagator
2.Merge
DataSource ADataSource A
‘DataSource A +’Propagator
‘DataSource A +’Propagator
1.ExtendAdd data
DataSource ADataSource A
‘DataSource A’Propagator 1‘DataSource A’Propagator 1
4.Split
‘DataSource A’Propagator 2‘DataSource A’Propagator 2
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 63
Data Propagator & Digestable Data - ExampleMerge HR-InfoType DataSources II
Merge (Abap: PROVIDE)
Infotyp 0001 Infotyp 0002 Infotyp 0004
5000000231.12.9901.01.10
5000000131.12.0901.01.09
OrgUnittofrom
Meier31.12.9901.10.09
Müller30.09.0901.01.09
Nametofrom
50%31.12.9912.02.10
Handicaptofrom
Müller50000001 -30.09.0901.01.09Propagator Content:
Meier50000001 -31.12.0901.10.09
Meier50000002 -11.02.1001.01.10
Meier50000002 50%31.12.9912.02.10
Note: This is only an example: 0EMPLOYEE_ATTR DataSource merges InfoTypes: 0,1,7,8 alladditional InfoTypes like 4 must be merged in the Propagator Layer
LSA Example
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 64
LSA Layer ImplementationPropagation Layer & Trimming Data
DataSource ASource 2
DataSource ASource 2
DataSource ASource 1
DataSource ASource 1
‘DataSource A’Propagator
‘DataSource A’Propagator
3.Collect
DataSource BDataSource BDataSource ADataSource A
‘DataSource A & B’Propagator
‘DataSource A & B’Propagator
2.Merge
DataSource ADataSource A
‘DataSource A +’Propagator
‘DataSource A +’Propagator
1.ExtendAdd data
DataSource ADataSource A
‘DataSource A’Propagator 1‘DataSource A’Propagator 1
4.Split
‘DataSource A’Propagator 2‘DataSource A’Propagator 2
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 65
LSA Layer ImplementationShield Layers with InfoSources I
Layer in focus DSO
Inbound InfoSourceShield of Layer in focus-
unified view of layer as target
Outbound InfoSourceShield of Layer in focus –
unified view of layer as source
Central Transformationfrom Previous Layer to
Layer in Focus
Outbound InfoSource Shield of Previous Layer-
unified view of layer as source
Inbound InfoSourceShield of Subsequent Layer- -unified view of layer as target
Central Transformationfrom Layer in Focus to
Subsequent Layer
No Transformation
No Transformation
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 66
LSA Layer ImplementationShield Layers with InfoSources II
Central Transformationfrom Layer in Focus to
Subsequent Layer :No change
Central Transformationfrom Previous Layer to
Layer in Focus:No change
No Transformations
No Transformations
Layer in focus: Two new DSOs (logical
Partitions) of Domains introduced
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 67
Collecting DataMultiple Sources and Domains
PSA_DS1_B
PSA_DS1_C
PSA_DS1_D
PSA_DS1_E
YPDS11SX
YPDS11TX
YPDS11UX
PSA_DS1_A
Layer Flow-logic – Multiple Sources (A,B,C,D,E) to Propagator Domains (S,T,U)
DTPs
Data flowAcquisition
Unification/Harmonization Propagation
InfoSource: YPDS1100
InfoSource: YAD
S1100
InfoSource: YPDS1200
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 68
LSA Layer ImplementationPropagation Layer & Trimming Data
DataSource ASource 2
DataSource ASource 2
DataSource ASource 1
DataSource ASource 1
‘DataSource A’Propagator
‘DataSource A’Propagator
3.Collect
DataSource BDataSource BDataSource ADataSource A
‘DataSource A & B’Propagator
‘DataSource A & B’Propagator
2.Merge
DataSource ADataSource A
‘DataSource A +’Propagator
‘DataSource A +’Propagator
1.ExtendAdd data
DataSource ADataSource A
‘DataSource A’Propagator 1‘DataSource A’Propagator 1
4.Split
‘DataSource A’Propagator 2‘DataSource A’Propagator 2
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 69
Filling Domains Flow Split Implementation: Data Unification
The early PSA-based split
APA EMEA Americas
PSA
The Pass Thru DSO based split
APA EMEA Americas
PSA
Pass ThruWO-DSO
Unification InfoSource
Propagators InfoSource Propagators InfoSource
Unification of datadata managementadministration Preferred
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 70
LSA Implementation Reference Automated Development & Maintenance
Acquisition/PSA Layer
CorporateMemory
Layer
PropagationLayer
BusinessTransformation
Layer
FlexibleLayer
DimensionalLayer
VirtualLayer
(YADSS100)
YCDSS100
YBAPP1AX
YFAPP1AXYDAPP1AX
YVAPP1XX
YVAPP1XX
AsiaYPDSS1AX
YBAPP1GXYDAPP1GX
AmericasYPDSS1GX
New BW development features e.g.:Copy/ Merge Data Flow ____Semantical Partitioning ____
BW 7.20
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 71
Data Flow Copies
Data Flow Copy Wizard
BW 7.20
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 72
Semantical Partitioning and Domains
APA EMEA
InfoSource
Source
InfoSource
Source
AMERICAS
Acquisition Layer
Subsequent Layers
DomainsBW 7.20
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 73
LSA Administration & Operations Automated Alerting & BW DB Statistics
Acquisition/PSA Layer
CorporateMemory
Layer
PropagationLayer
BusinessTransformation
Layer
FlexibleLayer
DimensionalLayer
VirtualLayer
(YADSS100)
YCDSS100
YBAPP1AX
YFAPP1AXYDAPP1AX
YVAPP1XX
YVAPP1XX
AsiaYPDSS1AX
YBAPP1GXYDAPP1GX
AmericasYPDSS1GX
New BW admin & operation e.g.BW DB Statistics ____Automated Alerting for Process
Chains _____
BW 7.20
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 74
LSA Administration & Operations BW DB Statistics
BW 7.20
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 75
LSA Administration & Operations Automated Monitoring and Alerting
BW 7.20
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 76
SAP BW @Global CP CustomerLayered, Scalable Architecture & Business Value
Domains/ Logical BW Partitions:
Market reportingBW (Virtual) MultiProvider:
cross market reporting
EDW/ Corporate Memory:disjoint
Granular- Most atomicHistorical completeComprehensivecovers 95% of businessin total ~ 150 TB allocated
1 : 1, not all today
Global EDW/Corporate Memory
in total ~40 TBGlobal reporting/ dash boarding
on integrated data with drill thru to most granular level
The result:Better an intermediate state
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 76
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 77
1. BI Maturity and Data Logistics – Motivation2. BW Layered Scalable Architecture (LSA) –
The Reference Architecture for BW on Corporate/ Global scale2.1. LSA Building Blocks2.2. LSA Data Layers2.3. LSA Data Domains2.4. LSA & Master Data
3. LSA Implementation – Unified Data Warehousing4. BW LSA Assistent Building Blocks
1. Storage - RDBMS & Columnar DMS2. Landscape: BW – Centralization & Federation
5. Summary
Agenda
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 78
SAP BW EDW & RDBMS
BW 60 TB
Proof of Concept
onDB2
RDBMS & Space
-Compression
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 79
SAP BW LSA / RDBMS / Hosts Transparency of LSA Enables Embedding
SAP BW LSA Architecture
0
Dim
ensi
on
Tabl
es
Bas
is
Tabl
es
Mas
ter D
ata
6 7 8 9 10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
1 3 4 52
MidSize
Objects
(Fact)8
Partitions
DB2 Partitioning Layout
LPAR 0 - 0 sysXdb00pDB2 Partition 0
LPAR 1 - 0sysXdb10pDB2 Partition 6 … 13
LPAR 2 – 0sysXdb20pDB2 Partition 14 … 21
LPAR 3- 0 sysXdb30pDB2 Partition 12 … 29
LPAR 4 - 0 sysXdb40pDB2 Partition 30 … 37
LPAR 0 - 1 sysXdb01pStorage Agent
LPAR 1 - 1 sysXdb11pStorage Agent
LPAR 2 - 1 sysXdb21pStorage Agent
LPAR 3 - 1 sysXdb31pStorage Agent
LPAR 4 - 1 sysXdb41pStorage Agent
4 FC(tape)
8 FC(tape)
8 FC(tape)
8 FC(tape)
8 FC(tape)
4 FC(tape)
8 FC(tape)
8 FC(tape)
8 FC(tape)
8 FC(tape)
4 FC(FC disk)
8 FC(FC disk)
8 FC(FC disk)
8 FC(FC disk)
8 FC(FC disk)
LPAR 0 - 2 sysXdb02pApp Server
LPAR 1 - 2sysXdb12pApp Server
LPAR 2 - 2 sysXdb22pApp Server
LPAR 3 - 2 sysXdb32pApp Server
LPAR 4 - 2 sysXdb42pApp Server
4 FC(FC disk)
8 FC(FC disk)
8 FC(FC disk)
8 FC(FC disk)
8 FC(FC disk)
pSeries Architecture
BW-LSA-Cell BW-DataClass(es)Tablespace(s) DB2 Partition(s) virtual machines
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 80
Renewing BW based on In-Memory TechnologiesContinuous Improvement and Accelerated Innovation
Enterprise Data WarehouseCorporate memory layer (detailed data)and analysis marts synchronized on common metadata (x00 TB)
Agile WarehouseOperational data and dimensional analysis models (marts) available directlyfrom memory (10th of TB)
MartStand-alone accelerator for data marts(low data volumes)
Data ServicesEasy integration w/ 3rd party data
Data quality for load processes
Agile modelingVisual modeling environment to support
modelers with different levels of sophistication.Reduce the Total cost of development (TCD)
High data volumesSupport handling of very large data
volumes by pushing operations tounderlying databases (IBM DB6)
Continuous ImprovementImproving today’s DWH experience
Accelerated InnovationNext generation DWH technology
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 81
Faster DataStore Activation with 7.20
Optimization FocusStandard DataStore Objects for building EDW-layers, i.e. optimize for fast updates BEx flagis unchecked, no SIDs are created
BW 7.0 DataStore Object Activation is based on single lookups of active table.
BW 7.2 DataStore Object - ImprovementsActivation is based on package fetch of active tableRuntime option “new, unique data records only” prevents lookups during activation e.g. in case of initial loadsPartitioning support by time characteristics
Performance gain for package fetch (lab results)Avg. 20 – 40% improvement Max. improvement measured – 2.5x fasterVaries by data profile (#inserts/updates/deletes) and DB platform
BW 7.20
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 82
SAP BW Layered, Scalable Architecture-Use Adequate Storage and Access
Data AcquisitionLayer
InIn--memorymemorycolumnar columnar
Data ManagementData ManagementDiscDisc--centriccentric
RDBMSRDBMS
Analytical Reporting Layer
Data Warehouse Layer
Memory-centric (Ram-based) data management:By most measures, computing power doubles every couple of years.(Moore’s Law)Exception is disk access speed it has grown only 12.5-fold in a half a centuryConventional DBMS are designed to get data on and off disk quicklyMemory-centric products assume all the data is in RAM in the first placeRAM access speeds are up to 1,000,000 times faster than random reads on disk.
SAP NetWeaver BWSAP NetWeaver BW
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 83
SAP BW EDW & Transparent Data ManagementUse Adequate Storage
BW &
BWA/In
Memory,columnar
DataManagement
BW &
NearLine
Storage(NLS)
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 84
RDBMSSmart data warehousing
SAP BW EDW & LSA –Consistent Architecture, Consistent Data, Consistent BI
SAP BW Transparent
Data WarehouseManagement
BW AcceleratorSmart data aggregation
& retrieval
Near-Line StorageSmart data volume
management
BW LSA
Analytic Engine
Data ManagementCross Media
Manager
Staging
Extraction
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 85
Renewing BW based on In-Memory TechnologiesContinuous Improvement and Accelerated Innovation
Enterprise Data WarehouseCorporate memory layer (detailed data)and analysis marts synchronized on common metadata (x00 TB)
Agile WarehouseOperational data and dimensional analysis models (marts) available directlyfrom memory (10th of TB)
MartStand-alone accelerator for data marts(low data volumes)
Data ServicesEasy integration w/ 3rd party data
Data quality for load processes
Agile modelingVisual modeling environment to support
modelers with different levels of sophistication.Reduce the Total cost of development (TCD)
High data volumesSupport handling of very large data
volumes by pushing operations tounderlying databases (IBM DB6)
Continuous ImprovementImproving today’s DWH experience
Accelerated InnovationNext generation DWH technology
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 86
SAP BW Layered, Scalable Architecture-Use Adequate Storage and Access (cont.)
Data AcquisitionLayer
In-MemoryColumnar
Data Management
DiscDisc--centriccentricRDBMSRDBMS
Analytical Reporting Layer
Data Warehouse Layer
Leverage Memory-centric (Ram-based) data management in SAP BWRam-based SAP BW Accelerator offers:
Performance speedup factor between 10 and 100Compression by factor 10Easy migration, fully transparent
Near Line Storage Interface (e.g. SAND Dynamic Near-Line Access)Fully integration to SAP BW (Query access & DataTransferProcess)
Vertical Decom
positionC
ompression ~ 90%
SAP NetWeaver BWSAP NetWeaver BWTransparent
BW AcceleratorBW Accelerator
‘‘NearNear--LineLineStorageStorage’’
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 87
1. BI Maturity and Data Logistics – Motivation2. BW Layered Scalable Architecture (LSA) –
The Reference Architecture for BW on Corporate/ Global scale2.1. LSA Building Blocks2.2. LSA Data Layers2.3. LSA Data Domains2.4. LSA & Master Data
3. LSA Implementation – Unified Data Warehousing4. BW LSA Assistent Building Blocks
1. Storage - RDBMS & Columnar DMS2. Landscape: BW – Centralization & Federation
5. Summary
Agenda
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 88
EDW Incremental Implementation
2. It may be incremental in terms of landscape consolidation
An EDW implementation is always an incremental one, never a big-bang.
1. Incremental in terms of functional and organizational coverage:
Dem
and Planning
Generating D
emand
Procure 2 P
ay
CO
PA
..... .....
EDW
nBI Application Coverage & EDW
Org U
nit A
Org U
nit B
Org U
nit C
Org U
nit N
..... .....EDW
nOrganizational Coverage & EDW
SourceSource
SourceLocal
SAP BI LocalSAP BI
Source
Source
SourceLocal
SAP BI Unit 2SAP BI
Stream CSAP BI Group
SAP BI
Unit 1DWH
EMEA:
Global BI
AMSAPA…
12
3
EMEA :
Global BI
AMSAPA…
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 89
Step 1 – BI/ DWH Consolidation - CentralizationTools, Architecture, Development, Organization, Processes
As a result of BI & DWH Consolidation we find a new corporate Information culturenew corporate Information culture:Awareness about the value of standards for consistency, flexibility & TCO
Organization: BI CCProcesses
Tools: SAP BW, BIA
Architecture & Landscape:
BW LSA/ BW EDW
Development: central BI applications template
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 90
Step 1: BI/ DWH CentralizationValue of LSA & Central Template
BW EDW Customer LSA
Central BI Applications Template Covering standard BI needs
Central template comprises:Complete dwh managementComplete front-end designComplete operational setting
Centralize BI/ DWH:Corporate-wide standardized reporting & analytics as core BI offering based on Customer LSA.
Value:consistent data & applicationsscalable applications on EDWflexibility caused by granular EDWdriving organizational & process
alignmentreduced TCO & TCD
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 91
Step 1: BI/ DWH CentralizationLimits of Centralized BI
BW EDW Customer LSA
Central BI Applications Template Covering standard BI needs
Central template comprises:Complete dwh managementComplete front-end designComplete operational setting
BI template coverage of end-user needs What is the coverage of the template-based core BI offering with respect to end-user needs?
Depending on central governance, functional-area and customer industry we could expect 60-100%
There is a ‘BI gap’ that cannot be closed by centralization!
How to close this gap?
Centralization with Federation gives the holistic BI picture!What kind of Federation is reasonable depends on the gap-size.
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 92
Step 2: BI Centralization & Federation A) Data Federation – Small Gap
Coverage of end-user needsE.g. CP-Customer Sales-force & local market data
Solution:Standard template + Data Federation
Query/ Semantical Layer
Local
BW EDW
Central BI
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 93
BW EDW
Central BI
Step 2: BI Centralization & Federation B) Newton Federation – Medium Gap
Coverage of end-user needs
Query/ Semantical Layer
Country1Source
Country 1NewtonServer
Country2NewtonServer
Country3 NewtonServer
E.g. pharma industry: considerable amount of individual country reporting
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 94
BW EDW
Central BI
Step 2: BI Centralization & Federation c) BW Federation – Large Gap – Hub & Spoke
Coverage of end-user needs
Query/ Semantical Layer
DivisionalSources
Divisional BI
BW
Divisional BI
BW
Divisional BI
BW
E.g. mining industry divisional reporting (Aluminum, petrol..) Prerequisite:
educated IT staff at divisional side
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 95
1. BI Maturity and Data Logistics – Motivation2. BW Layered Scalable Architecture (LSA) –
The Reference Architecture for BW on Corporate/ Global scale2.1. LSA Building Blocks2.2. LSA Data Layers2.3. LSA Data Domains2.4. LSA & Master Data
3. LSA Implementation – Unified Data Warehousing4. BW LSA Assistent Building Blocks
1. Storage - RDBMS & Columnar DMS2. Landscape: BW – Centralization & Federation
5. Summary
Agenda
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 96
BI-Project-Design
Life Cycle of The Customer LSA
BW LSA: The Reference Architecture
Customer LSA : Standards - Handbook
BI-Project-DesignBI Project Design
Step 3:PerfectPerfect
Customer LSA
Step 4:UpdateUpdate
Customer LSA
Step 1:DesignDesign
Customer LSA
Step 2:ApplyApply
Customer LSA
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 97
Customer LSA ‘Handbook’Shell
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 97
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 98
Customer LSA ‘Handbook’Shell
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 98
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 99
Customer LSA ‘Handbook’Land Hessen
© SAP 2009 / Daimler - BW Layered, Scalable Architecture (LSA) Intro, Juergen Haupt /200903/ Page 99
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 100
BI-Project-Design
Life Cycle of The Customer LSA
SAP LSA: The Reference Architecture
Customer LSA : Standards - Handbook
BI-Project-DesignBI Project Design
Step 3:PerfectPerfect
Customer LSA
Step 4:UpdateUpdate
Customer LSA
Step 1:DesignDesign
Customer LSA
Step 2:ApplyApply
Customer LSA
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 101
New Planned Features with BW 7.20 for BW EDW
Description Area FunctionHybrid Provider EDW InfoProvidersTransientProvider EDW InfoProvidersEnhancements Real Time Data Acquisition EDW InfoProviders
Semantic Partitioning EDW InfoProviders
Monitoring: New Technical Content EDW Operations
CTC templates for setting up BW systems EDW Operations
Search Enhancements in DWB EDW UsabilitySoft shutdown for BW systems EDW Operations
Process chain job overview (RSM37) EDW OperationsEnhanced process chain monitoring (RSPCM) EDW OperationsReset interface for process chains EDW OperationsRestarting loading processes in background EDW OperationsDelete content of table RSDDSTAT_WHM EDW Operations
Data request archiving for DTP requests EDW OperationsInfoPackages that switch automatically from Init to Delta EDW Operations
Delta INIT without data transfer for DTPs EDW ImplementationCopy data Flow EDW ImplementationMigration Tools 3.x -> 7 EDW Implementation
Before Export Check EDW
Generic Delta functionalities for further kinds of DataSources EDW
Multi Source System InfoPackage transport EDWWeb Service (Pull!) EDWDataStore Object: Partitioning by time EDWDataStore Object: Improved Activation, Initial Load EDWDataStore Object: Remodeling EDWHierarchies: Integration into data flow via DTP, Transformation EDWHierarchies: remote Hierarchies EDWLoading hierarchies through tRFC EDWOpen Hub Destination - XML as format EDWOpen Hub Destination - Export of field values in external format EDWTransformation Field selection in start/end routine EDWTransformation - Currency conversion for DataStore Objects EDWMapping Content Technology EDWNLS_Archiving: Support of MultiProviders EDWNLS_Archiving: Transparent embedding of NLS into Dataflow (depending on selection System uses appropriate source) EDW
Archiving of uncompressed InfoCube data (relevant for all DBs!!!) EDW
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 102
Defining a BW / BI Excellence Framework*Reminder
BI Business Value Drivers
SAP ERP SAP CRM Other Legacy
InfrastructureEnterprise Layer Concept, Data Marts, ODS, ETL,
BI-Topology, Data Quality, Data Model
Applications and FunctionalityBI Flavors ; Reporting;
Strategic, Operational and Analytical Applications
Organization Process
StrategyBusiness Objectives, Transparency
Performance Management, Methodology
Skills, BI CompetencyCentre
BI Framework*BI Framework*
• Consistency• Flexibility
• Efficiency• Suitability
• Realization
• Alignment
* BI Framework introduced by Gartner
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 103
Layered Scalable Architecture (LSA) asBest Practice Modeling of High-End BWs
The LSA is a Best Practice modeling for large SAP BW Data Warehouses.The LSA structures and standardizes a BW DWH in a transparent, service-level oriented, scalable manner. Transparency is achieved introducing a grid on a BW DWH defined by Data Layers and Data Domains
Services are modeled by Data LayersScalability and Manageability is guaranteed by Data Domains
The broader the organizational and value-chain coverage of a BW DWH is, the more necessary is a design-standardization like propagated by the LSA. The most comprehensive approach of a BW data logistic is an Enterprise Data Warehouse (EDW)
LSA Architectured
D a
t a
f l o
w
A transparent data-logistic like a Layered, Scalable Architecture is a ‘key success factor’
of BW ‘High-End’-/ EDW-implementations!Basic concepts can very well applied to smaller implementations!
∑
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 104
Further Info
Blogs:
SAP NetWeaver BW: BW Layered Scalable Architecture (LSA) / Blog Series
https://www.sdn.sap.com/irj/scn/weblogs?blog=/pub/wlg/14313
Workshop – SAP Education Germany:
PDEBW1 - BW-Blueprinting an Enterprise Data Warehouse: Architecture and Implementation Best Practices
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 105
Q&A
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 106
Thank you!
© SAP 2009 / SAP Skills 2009 Conference / B1 / Page 107
Copyright 2009 SAP AGAll Rights Reserved
No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP AG. The information contained herein may be changed without prior notice.
Some software products marketed by SAP AG and its distributors contain proprietary software components of other software vendors.
Microsoft, Windows, Excel, Outlook, and PowerPoint are registered trademarks of Microsoft Corporation.
IBM, DB2, DB2 Universal Database, System i, System i5, System p, System p5, System x, System z, System z10, System z9, z10, z9, iSeries, pSeries, xSeries, zSeries, eServer, z/VM, z/OS, i5/OS, S/390, OS/390, OS/400, AS/400, S/390 Parallel Enterprise Server, PowerVM, Power Architecture, POWER6+, POWER6, POWER5+, POWER5, POWER, OpenPower, PowerPC, BatchPipes, BladeCenter, System Storage, GPFS, HACMP, RETAIN, DB2 Connect, RACF, Redbooks, OS/2, Parallel Sysplex, MVS/ESA, AIX, Intelligent Miner, WebSphere, Netfinity, Tivoli and Informix are trademarks or registered trademarks of IBM Corporation.
Linux is the registered trademark of Linus Torvalds in the U.S. and other countries.
Adobe, the Adobe logo, Acrobat, PostScript, and Reader are either trademarks or registered trademarks of Adobe Systems Incorporated in the United States and/or other countries.
Oracle is a registered trademark of Oracle Corporation.
UNIX, X/Open, OSF/1, and Motif are registered trademarks of the Open Group.
Citrix, ICA, Program Neighborhood, MetaFrame, WinFrame, VideoFrame, and MultiWin are trademarks or registered trademarks of Citrix Systems, Inc.
HTML, XML, XHTML and W3C are trademarks or registered trademarks of W3C®, World Wide Web Consortium, Massachusetts Institute of Technology.
Java is a registered trademark of Sun Microsystems, Inc.
JavaScript is a registered trademark of Sun Microsystems, Inc., used under license for technology invented and implemented by Netscape.
SAP, R/3, SAP NetWeaver, Duet, PartnerEdge, ByDesign, SAP Business ByDesign, and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany and other countries.
Business Objects and the Business Objects logo, BusinessObjects, Crystal Reports, Crystal Decisions, Web Intelligence, Xcelsius, and other Business Objects products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of Business Objects S.A. in the United States and in other countries. Business Objects is an SAP company.
All other product and service names mentioned are the trademarks of their respective companies. Data contained in this document serves informational purposes only. National product specifications may vary.
These materials are subject to change without notice. These materials are provided by SAP AG and its affiliated companies ("SAP Group") for informational purposes only, without representation or warranty of any kind, and SAP Group shall not be liable for errors or omissions with respect to the materials. The only warranties for SAP Group products and services are those that are set forth in the express warranty statements accompanying such products and services, if any. Nothing herein should be construedas constituting an additional warrant.