Real Time Data Acquisition RDA - Powerpoint Presentation
-
Upload
abhisheksapbi -
Category
Documents
-
view
222 -
download
0
Transcript of Real Time Data Acquisition RDA - Powerpoint Presentation
-
8/12/2019 Real Time Data Acquisition RDA - Powerpoint Presentation
1/34
Andreas Keppler RIG EMEA BI,
SAP AG
SAP NetWeaver BIReal Time Data Acquisition
-
8/12/2019 Real Time Data Acquisition RDA - Powerpoint Presentation
2/34
SAP AG 2005, Title of Presentation, Speaker Name /Version 1 2
Enterprise Data Warehousing Variants & Processes
Modeling the EDW
Data ModelingMeta Data
&
Document
Manage-
ment
Data Acquisition
Transformation
Data Distribution
Running the EDW
Administration & Monitoring
Data Flow Control
Performance Optimization
Information Lifecycle Management
User
Manage-
ment
-
8/12/2019 Real Time Data Acquisition RDA - Powerpoint Presentation
3/34
Overview
Real-time Data Acquisition via the Service API
Real-time Data Acquisition via Web Services
Real-time Data Acquisition Daemon Monitor
-
8/12/2019 Real Time Data Acquisition RDA - Powerpoint Presentation
4/34 SAP AG 2005, Title of Presentation, Speaker Name /Version 1 4
Motivation for real-time
Upload frequency for regular staging not sufficient Number of requests can not be handled by BI system
Reporting requirements of data with sub 1 hours actuality
Operational reporting
The clear distinct ion between tools for analytical applications and tools foroperational applications is more and more difficult.
See transactional data in reporting as it is created in source system (post & see )
Availability of data in source system not known Use Push mechanism (e.g. SAP NW XI)
Polling or Trigger data load process via event from source system
Remote access of data not feasible due to resource consumption insource system (and cross application reporting requirements)
Splitting of staging processes (EDW) General business rulesare processed during the day whereas scenario specific rules (definedby application, region, time zone, etc.) are processed at customerdefined times
-
8/12/2019 Real Time Data Acquisition RDA - Powerpoint Presentation
5/34 SAP AG 2005, Title of Presentation, Speaker Name /Version 1 5
What is real-time?
Real-time data warehousing is a framework for deriving
information from data as the data becomes available:
Lower time scale than for scheduled/batch data acquisit ion
Stream oriented
Near immediate availabili ty for reporting
In general, Real-time data warehousing supports tactical decision-making
Simon Terr: Real-time Data Warehousing 101, DM Review Online October 2003
-
8/12/2019 Real Time Data Acquisition RDA - Powerpoint Presentation
6/34 SAP AG 2005, Title of Presentation, Speaker Name /Version 1 6
External
System
Web ServicePush
Real-time Data Acquisition (RDA)
SAP Source System
Service API
ApplicationDelta Queue
Real-time
Update
SAP NetWeaver Business Intelligence
DataSource
Data Warehouse Layer
Operational Data
Store
Architected Data Marts
Daemon
Pull
~ 1/min
InfoPackage for Real-
Time Data Acquisit ion
DataTransfer Process for
Real-time Data Acquisi tionScheduled Data
Transfer Process
Scheduled Data
Transfer Process
Real-time
Update
PSA
DataStore Objects
-
8/12/2019 Real Time Data Acquisition RDA - Powerpoint Presentation
7/34 SAP AG 2005, Title of Presentation, Speaker Name /Version 1 7
RDA Scenarios
Real-time data acquisition can be used in two primary scenarios:
via the Service API (SAPI)
Incorporates usage of InfoPackage for Real-time Data Acquisition
(source to PSA)
Then leverages Data Transfer Process for Real-time Data Acquisition
(PSA to DataStore Object)
via a Web Service
Incorporates usage of Web Services to populate the PSA
Then leverages the Real-time DTP to transfer data to the DataStore
Object
A system Daemon is used to initiate and control data transfer in
both scenarios.
-
8/12/2019 Real Time Data Acquisition RDA - Powerpoint Presentation
8/34 SAP AG 2005, Title of Presentation, Speaker Name /Version 1 8
What is a Daemon ?
Daemon
System process fulfils a specific task at regular intervals
The SAP NetWeaver BI Daemon performs three steps (depending on
the scenario chosen):
Initiate Service-API (SAPI) data pull via InfoPackage for Real-t ime Data Acquisition into PSA
Track status of data transfer from source system
Init iate update of DataStore Object via DTP
Successful execution of each step is tracked in a control table
Allows restarting if necessary
Restart can be initiated so it starts at the next step after the last
successfully executed step.
-
8/12/2019 Real Time Data Acquisition RDA - Powerpoint Presentation
9/34 SAP AG 2005, Title of Presentation, Speaker Name /Version 1 9
Daemon Monitoring
Maintenance screen for demon control can be called from the Data
Warehousing Workbench
Transaction RSRDA
-
8/12/2019 Real Time Data Acquisition RDA - Powerpoint Presentation
10/34
Overview
Real-time Data Acquisition via the Service API
Real-time Data Acquisition via Web Services
Real-time Data Acquisition Daemon Monitor
-
8/12/2019 Real Time Data Acquisition RDA - Powerpoint Presentation
11/34
SAP AG 2005, Title of Presentation, Speaker Name /Version 1 11
Scenario I: RDA via SAP applications
SAP Source System
Service API
ApplicationDelta Queue
Real-time
Update
SAP NetWeaver BI
DataSource
Operational Data
Store
Daemon
Pull
~ 1/min
InfoPackage for Real-time
Data Acquisition
Data Transfer Process for
Real-time Data Acquisi tion
PSA
DataStore Objects
Service API-based real-time dataacquisit ion is a two-stage
process:
Data is pulled into PSA by Real-
time InfoPackage
Data is transferred to DataStore
Object via Real-time DTP
In both cases the process of datamovement is initiated by the
system daemon
-
8/12/2019 Real Time Data Acquisition RDA - Powerpoint Presentation
12/34
SAP AG 2005, Title of Presentation, Speaker Name /Version 1 12
Application Activities on the Source System Side
Source system
(ERP, CRM ...) Application
Service API
1 2 3
Time 03:10:09
Delta Queue
D T i RDA P i
-
8/12/2019 Real Time Data Acquisition RDA - Powerpoint Presentation
13/34
SAP AG 2005, Title of Presentation, Speaker Name /Version 1 13
Daemon Triggers RDA Processing
Source system
(ERP, CRM ...) Application
Service API
BI
1 2 3
Daemon
pull ~ 1/min
PSA
Time 03:11:00
Confirmation Table
DataStore Object
Delta Queue
Have a look Data
transfer
TID 1
TID 2
TID 3
Log entries
Update PSA
1
2
3
TID 1 ok
TID 2 ok
TID 3 ok
CommitCommit
1 2 3
ata tore
DTP
Initiate
Step 1 completed
Step 2 compl.
R ti St I
-
8/12/2019 Real Time Data Acquisition RDA - Powerpoint Presentation
14/34
SAP AG 2005, Title of Presentation, Speaker Name /Version 1 14
Runtime Steps I
Step 1
Call source system for new records
Status update of transferred records in confirmation table
Update PSA
Commit
Update confirmation table and update PSA in one step Guarantees synchronization
R ti St II
-
8/12/2019 Real Time Data Acquisition RDA - Powerpoint Presentation
15/34
SAP AG 2005, Title of Presentation, Speaker Name /Version 1 15
Runtime Steps II
Step 2
Check for records in confirmation table
If records are available in confirmation table the corresponding records
exist as well in PSA
Daemon flags records in source system
Reply confirmation sent to SAP NetWeaver BI
Entries are flagged as processed in confirmation table
Initiate DTP after the records are confirmed in confirmation table
Commit
If not successful in BI, records will stay in confirmation table
Guarantees restart of step 2 even when the update in the source systemwas successful
Daemon Triggers RDA Processing every 3 minutes
-
8/12/2019 Real Time Data Acquisition RDA - Powerpoint Presentation
16/34
SAP AG 2005, Title of Presentation, Speaker Name /Version 1 16
Daemon Triggers RDA Processing every 3 minutes
Source system
(ERP, CRM ...) Application
Service API
BI
1 2 3
Daemon
pull ~ 1/min
PSA
Time 03:14:00
Confirmation Table
DataStore Object
Delta Queue
TID 1
TID 2TID 3
TID 1 ok
TID 2 ok
TID 3 ok
1 2 3
ata tore
DTP
Initiate
4 5
Next
data load cycle
1
2
34
5
1
2
3
4 5
TID 1 ok
TID 2 ok
TID 3 okTID 4 ok
TID 5 ok
TID 1 ok
TID 2 ok
TID 3 ok
Closure of open request
-
8/12/2019 Real Time Data Acquisition RDA - Powerpoint Presentation
17/34
SAP AG 2005, Title of Presentation, Speaker Name /Version 1 17
Closure of open request
Not executed on every data load cycle
After completion of data upload into PSA:
Entries in confirmation table are deleted once the request is closed
(threshold for number of records per request / time exceeded or daemon
stopped).
Before the next data load cycle is started:
The next request is opened for the next data load from source system
(determined by update period of the daemon)
Requests of InfoPackage and DTPs are synchronized via daemon
(i.e. request in PSA and DataStore Object are aligned).
Daemon closes request
-
8/12/2019 Real Time Data Acquisition RDA - Powerpoint Presentation
18/34
SAP AG 2005, Title of Presentation, Speaker Name /Version 1 18
Daemon closes request
BI
Daemon
pull ~ 1/min
PSA
Time 03:17:00
DataStore Objectata tore
DTP
Initiate
Step 0 completed
cont. with load
1
2
3
4
5
1
2
3Confirmation Table
TID 1 ok
TID 2 ok
TID 3 okTID 4 ok
TID 5 ok
Request closed due to
time or n of records
Request 1
Source system
(ERP, CRM ...) Application
Service API
1 2 3
Delta Queue
1 2 3 4 54 5
InfoPackage for Real-time Data Acquisition Creation
-
8/12/2019 Real Time Data Acquisition RDA - Powerpoint Presentation
19/34
SAP AG 2005, Title of Presentation, Speaker Name /Version 1 19
InfoPackage for Real-time Data Acquisition Creation
Real-time flag must be selected for the InfoPackage which the
daemon will monitor.
This option is only available for one InfoPackage attached to theDataSource
Note: If you want to use a DataSource with real-time capabilities,you have to install the PI_BASIS 2005.1 in the source system
InfoPackage for Real-time Data Acquisition Definition
-
8/12/2019 Real Time Data Acquisition RDA - Powerpoint Presentation
20/34
SAP AG 2005, Title of Presentation, Speaker Name /Version 1 20
InfoPackage for Real-time Data Acquisition Definition
Real-time extraction must be selected in the Adapter section of
the InfoPackage settings
InfoPackage for Real-time Data Acquisition DaemonS tti
-
8/12/2019 Real Time Data Acquisition RDA - Powerpoint Presentation
21/34
SAP AG 2005, Title of Presentation, Speaker Name /Version 1 21
Settings
Automatic Request Closure by # records
Periodicity is determined by Update Periods setting via Real Time Monitor (RSRDA).
Data Package Size: No. of Rows Allows control of number of records per data package (multiple data packages per request possible)
The deamon uploads at least one data package (even if number of actual records is smaller than
setting)
Request Size: Number of Rows Allows control of packets for downstream processing based on time
Automatic c losure of Request by number of records (1,000,000 shown here)
Max number of failed attempts
Max number of times Daemon attempts to request/commit data before throwing error condition
InfoPackage for Real-time Data Acquisition Update:I h tl D lt
-
8/12/2019 Real Time Data Acquisition RDA - Powerpoint Presentation
22/34
SAP AG 2005, Title of Presentation, Speaker Name /Version 1 22
Inherently Delta
Only Delta Processing is
available for Real-timeupdates (because the deltaqueue is leveraged for datastaging)
On the Schedule tab you can jumpvia Assign to the RDA Monitor.There, you can assign theInfoPackage to a deamon.
-
8/12/2019 Real Time Data Acquisition RDA - Powerpoint Presentation
23/34
Overview
Real-time Data Acquisition via the Service API
Real-time Data Acquisition via Web Services
Real-time Data Acquisition Daemon Monitor
Scenario II: RDA via Web Services
-
8/12/2019 Real Time Data Acquisition RDA - Powerpoint Presentation
24/34
SAP AG 2005, Title of Presentation, Speaker Name /Version 1 24
External
System
Web Service
Push
Sce a o a eb Se ces
SAP NetWeaver Business Intelligence
DataSource
Operational Data
Store
Daemon
Pull
~ 1/min
Real-time
Update
PSA
DataStore Objects
InfoPackage for Real-time
Data Acquisition
Data Transfer Process for
Real-time Data Acquisi tion
Web Service-based Real-time data acquisition is also a two-stageprocess:
Data is pushed into the PSA via a Web Service
Data is transferred to DataStore Object via Real-time DTP
The daemon monitors the PSA and initiates the DTP on a regularbasis (via settings in Real-time InfoPackage)
Web Service Data Source
-
8/12/2019 Real Time Data Acquisition RDA - Powerpoint Presentation
25/34
SAP AG 2005, Title of Presentation, Speaker Name /Version 1 25
InfoPackage for Real-time Data Acquisition via Web Service
-
8/12/2019 Real Time Data Acquisition RDA - Powerpoint Presentation
26/34
SAP AG 2005, Title of Presentation, Speaker Name /Version 1 26
g q
An InfoPackage is needed for the Web Service to defineparameters such as t ime / record thresholds for request closure
and to allow the assignment of a daemon via the Real-time
Monitor.
Real-time requests in DataStore Object
-
8/12/2019 Real Time Data Acquisition RDA - Powerpoint Presentation
27/34
SAP AG 2005, Title of Presentation, Speaker Name /Version 1 27
q j
DataSource / PSA
ActivationQueue
ActiveData
ChangeLog
Per.1 Per. 2 Per.3
Req.ID, Pack ID, Rec. No.
Req.SID, Pack ID, Rec. No.
Document No.
Latency between source system transaction and availabil ity in DataStore Object isminimal due to Lean Staging .
Lean Staging means that the DataStore Object handles the data differentlyinternally less logging and activation activity is allowed when the DTP for Real-time Data Acquisit ion is connected to the DataStore Object
From an administrative point of view, the DataStore Object is a standard one.
The DataStore Object also handles the availability of this real-time data (e.g. datawhose request has not been closed) for reporting.
How to embed RDA in your data flow
-
8/12/2019 Real Time Data Acquisition RDA - Powerpoint Presentation
28/34
SAP AG 2005, Title of Presentation, Speaker Name /Version 1 28
DataSource
PSA
RegularDataStore Object
OLTP
BI
Delta Queue
2LIS_02_VAITM Appl ication
real-time
Update
Using one DataSource
DataSource
PSA
RDADataStore Object
RegularDataStore Object
OLTP
BI
Daemon
Pull~5/min
Delta Queue
2LIS_02_VAHDR2LIS_02_VAITM
Periodic
Scheduling
2LIS_02_VAITM
Appl ication
real-time
Update
Using two DataSources
Daemon
Pull~5/min
Well established Data Flow can
can be used
Separate (potentially) leaner
DataStore Object can be used
Reporting has to be enabled via
MultiProvider
Data has to be deleted frequently
from RDA DataStore Object to
avoid redundant data
Old data flow has to be replaced
by new real-time data flow
No issues with data alignment
from different DataSources
Can facilitate upload of huge
data volumes by spreading
data load over many smaller
data loads
Periodic
Scheduling
2LIS_02_VAITM
Extraction done via RDA and InfoPackage scheduling
( Normal Delta ) can currently not be done in parallelDataSource can only use a single extraction mechanism
-
8/12/2019 Real Time Data Acquisition RDA - Powerpoint Presentation
29/34
Overview
Real-time Data Acquisition via the Service API
Real-time Data Acquisition via Web Services
Real-time Data Acquisition Daemon Monitor
Daemon Monitor
-
8/12/2019 Real Time Data Acquisition RDA - Powerpoint Presentation
30/34
SAP AG 2005, Title of Presentation, Speaker Name /Version 1 30
The RDA Daemon monitor provides an overview on the status of each daemonand attached
InfoPackage(s) for RDA Data Transfer Process(es) for RDA
Important to note that both loading to the PSA and the DataStore Object aremonitored within the Daemon Monitor
Transaction RSRDA
Real-time
DTP
Real-time
InfoPackage
DataSource
Daemon Status
-
8/12/2019 Real Time Data Acquisition RDA - Powerpoint Presentation
31/34
SAP AG 2005, Title of Presentation, Speaker Name /Version 1 31
The Daemons can have the following statuses (identified by the icon in front):
- Daemon active and running
- Daemon not active
- Daemon has an error
- Daemon is stopped (changes its status to not active or error)
Standard BI Monitor for InfoPackages
-
8/12/2019 Real Time Data Acquisition RDA - Powerpoint Presentation
32/34
SAP AG 2005, Title of Presentation, Speaker Name /Version 1 32
Granular details
of datamovement look
similar to Data
Load Monitor in
prior releases
DTP for RDA Monitor
-
8/12/2019 Real Time Data Acquisition RDA - Powerpoint Presentation
33/34
SAP AG 2005, Title of Presentation, Speaker Name /Version 1 33
Detailed data transfer process steps in DTP Monitor
Similar to details provided by the Data Load Monitor
Copyright 2004 SAP AG. All Rights Reserved
-
8/12/2019 Real Time Data Acquisition RDA - Powerpoint Presentation
34/34
SAP AG 2005, Title of Presentation, Speaker Name /Version 1 34
No part of this publication may be reproduced or transmitted in any form or for any purpose without the express
permission of SAP AG. The information contained herein may be changed without prior notice.
Some software products marketed by SAP AG and its distributors contain proprietary software components of othersoftware vendors.
Microsoft, Windows, Outlook, and PowerPoint are registered trademarks of Microsoft Corporation.
IBM, DB2, DB2 Universal Database, OS/2, Parallel Sysplex, MVS/ESA, AIX, S/390, AS/400, OS/390, OS/400, iSeries,
pSeries, xSeries, zSeries, z/OS, AFP, Intelligent Miner, WebSphere, Netfinity, Tivoli, and Informix are trademarks or
registered trademarks of IBM Corporation in the United States and/or other countries.
Oracle is a registered trademark of Oracle Corporation.UNIX, X/Open, OSF/1, and Motif are registered trademarks of the Open Group.
Citrix, ICA, Program Neighborhood, MetaFrame, WinFrame, VideoFrame, and MultiWin are trademarks or registered
trademarks of Citrix Systems, Inc.
HTML, XML, XHTML and W3C are trademarks or registered trademarks of W3C, World Wide Web Consortium,
Massachusetts Institute of Technology.
Java is a registered trademark of Sun Microsystems, Inc.
JavaScript is a registered trademark of Sun Microsystems, Inc., used under license for technology invented and
implemented by Netscape.
MaxDB is a trademark of MySQL AB, Sweden.
SAP, R/3, mySAP, mySAP.com, xApps, xApp, SAP NetWeaver and other SAP products and services mentioned herein
as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany and in several othercountries all over the world. All other product and service names mentioned are the trademarks of their respective
companies. Data contained in this document serves informational purposes only. National product specifications may vary.
These materials are subject to change without notice. These materials are provided by SAP AG and its affiliated
companies ("SAP Group") for informational purposes only, without representation or warranty of any kind, and SAP Group
shall not be liable for errors or omissions with respect to the materials. The only warranties for SAP Group products and
services are those that are set forth in the express warranty statements accompanying such products and services, if any.Nothing herein should be construed as constituting an additional warranty.