Post on 22-Nov-2015
Backup, Recovery and High Availability
for SAP APO (3.x) / mySAP SCM (4.x) Best Practice for Solution Management
Version Date: February 2004
This version is valid for SAP APO 3.0, 3.1 and mySAP SCM 4.0, 4.1. Not valid for ICH and EM in mySAP SCM 4.x architecture.
The newest version of this Best Practice can always be obtained through SAP Solution Manager
Contents Applicability, Goals, and Requirements....................................................................................................2 Best Practice Procedure and Verification .................................................................................................4
Procedure...........................................................................................................................................4 Backup Strategy ...........................................................................................................................4 Restore and Recovery ...............................................................................................................10 High Availability ..........................................................................................................................14 Incomplete Recovery .................................................................................................................20
Verification........................................................................................................................................20 Further Information .................................................................................................................................21
Best Practice: APO Backup and Availability
2004 SAP AG
2
Applicability, Goals, and Requirements To ensure that this Best Practice is the one you need, consider the following goals and requirements.
Goal of Using this Service Use this service in the context of your mySAP SCM system landscape to guide you in creating:
An operational concept for Backup and Recovery (B&R) A suitable strategy for High Availability (HA)
Alternative Practices Your consulting partner can set up your B&R concept according to this Best Practice.
You should discuss HA options with your hardware partners.
Staff and Skills Requirements To apply this Best Practice, you need a team consisting of:
System administrators, consultants, or technical persons tasked with planning, setting up, or administering your SCM system landscape
Application developers or consultants with knowledge of your applications and business processes
SAP Active Global Support, APO CoE
APO 3.0 System
APO
liveCacheliveCache
APOApplication
Optimizer
App
Serv
er
Presentation Client
Database
Dat
abas
e Se
rver
APOApplication
BW Layer
APO DB
System Requirements The simplest example of a mySAP SCM landscape consists of one OLTP system and one APO system. This Best Practice deals with a:
mySAP SCM system landscape comprising: o An SAP R/3 System o An SAP APO server, SAP liveCache, and optionally an Optimizer
Best Practice: APO Backup and Availability
2004 SAP AG
3
Duration and Timing You should define the initial B&R and HA concept during the implementation of your mySAP SCM solution. You must finalize it before the start of production. Be sure to start early enough to leave time for procuring the required software and hardware.
Depending on the complexity of your solution landscape and business scenarios, you may need about a week to define your B&R and HA concept. You may need an additional week to verify and test it.
How to Use this Best Practice Complete the analysis described in this section. Design your operational concept using the information in section Procedure. Verify your success in implementing this Best Practice as described in section Verification.
Case Studies / Sample Scenarios For a sample backup schedule for a mySAP SCM landscape, see:
Case Study Backup Scheduling For a sample HA concept for a mySAP SCM landscape, see:
Case Study High Availability Options For a sample outline of an operations manual for B&R and HA, see:
Case Study Outline of an Operations Manual
Best Practice: APO Backup and Availability
2004 SAP AG
4
Best Practice Procedure and Verification
Procedure
Backup Strategy To determine a B&R strategy for an APO system, decide how often you need to make backups of the entire system and of the components. The two factors are data security and cost. The data security and thus your B&R strategy should be comparable for APO DB and liveCache.
Recovery is the process of bringing a database back to the point in time of a crash.
Restore is the process of retrieving data files from a backup medium.
Introduction You must create backups for the individual systems. Follow SAP standard recommendations for backup cycles and storage media.
How often should I make complete backups of the database?
The more often you make complete backups, the better. It is always faster to restore a recent backup than to restore an old backup and play back numerous logs. The backup frequency you need is directly proportional to the frequency of changes. But you should also keep in mind that creating online backups affects system performance.
Do I need to stop APO DB and liveCache to make a backup?
If you stop APO DB and liveCache to perform a backup, you get a consistent backup of the two components with a definite timestamp. In this case, no further log backups are necessary.
If instead you perform the backups during normal operation of the APO system, in a recovery you need to perform the restore and play back the logs. Only when you have completed the entire recovery procedure do you achieve a fully consistent state between APO DB and liveCache.
However, you may not need a consistent and complete backup. In any recovery, to prevent loss of data you must play back the log backups up to the last change. You only need a consistent complete backup of the APO system in offline mode if you plan to copy the APO system. Alternatively, you may wish to make such a backup to save a state of the APO system for fallback use in emergency. For example, you may wish to do this after the initial data load or before an upgrade.
In most situations, offline backups are an option only for customers who can afford increased planned downtime.
Tip If you make offline backups, always use the liveCache and APO DB tools. Never make backups by copying files, since then you cannot guarantee the consistency of the files or of the data they contain.
SAP APO server and SAP R/3 Follow the standard procedures for backup and recovery of SAP systems.
Make backups of the installation using operating system tools. You can make backups of the database using database tools, standard interfaces or the SAP
Computing Center Management System (CCMS).
The following graphic shows recommendations for an Oracle database.
Best Practice: APO Backup and Availability
2004 SAP AG
5
SAP AG 2001
28 days28 days28 days
AdditionalAdditional
Offline redo log Offline redo log file backup (x2)file backup (x2)
Offline redo log Offline redo log file backup (x2)file backup (x2)
Verify the databaseVerify the databaseVerify the backupVerify the backup
OnlineOnline
OnlineOnline
Verify the backupVerify the backupOfflineOffline
Oracle: Minimum Backup Cycle Recommendations
For details about the backup cycles, see the SAP Help Portal
or take database administration via SAP Training:
BC505 Oracle BC511 Informix ADM515 SAP Database BC520 MS SQL Server BC525 DB2 on OS/400 BC530 DB2 on OS390 BC535 DB2 on UDB
LiveCache
The following dependencies of APO releases and liveCache versions exist:
SAP APO 3.0 is delivered with liveCache 7.2 or liveCache 7.4.2, SAP APO 3.1 is delivered with liveCache 7.4.2, mySAP SCM 4.0 is delivered with liveCache 7.4.3, mySAP SCM 4.1 is delivered with liveCache 7.5,
The liveCache architecture has been improved from 7.2 to 7.4. The changes have an impact on B&R and HA as procedures become much simpler with liveCache 7.4. Therefore, this Best Practice treats the two versions separately. Operators of liveCache 7.2 should also read the 7.4 sections to facilitate later work with 7.4.
The backup & recovery procedures of liveCache versions 7.4.2, 7.4.3 and 7.5 do not differ so they are all covered by the SAP liveCache 7.4-section of this document.
Best Practice: APO Backup and Availability
2004 SAP AG
6
For either liveCache version, you need to make backups using operating system tools after installation and upgrades.
For more details, see SAP liveCache 7.4: Backup and Recovery & Administration News (E)
For liveCache, the following administration tools are available:
liveCache Administration Tools Several options are available for liveCache backup and recovery. The use of the tools or the line commands is as for SAP DB - the backup procedure for liveCache is integrated into the Computer Center Management System (CCMS) as of mySAP SCM 4.0 ( with SAP Basis Release 6.20 ).
Database Manager Graphical User Interface (DBMGUI) The Database Manager GUI is a liveCache / SAP DB tool for administration of LiveCache / SAP DB that runs on Windows platforms. It is the recommended tool for performing backups and recoveries from liveCache 7.4. onwards.
The procedure for backup administration is described in the document Database manager GUI.
For liveCache 7.2 backup and recovery other tools apply (see below).
Web Database Manager (WebDBM) The Web-based database manager (WebDBM) has the same look and feel as DBMGUI and can be used also on non-Windows clients. No local client installation is required. Access is provided over a web server. Find more about the WebDBM under SAP DB Web Tools.
Database Manager Command Line Interface (DBMCLI) The Database Manager CLI is a liveCache / SAP DB administration tool that works at command line level on the liveCache platform. For liveCache 7.4, it can be used to perform script-based backups of the database and thus also supports the backup of liveCache from an SAP system as a background job.
The backup administration procedure is described in the document:
Database Manager CLI
For liveCache 7.2 backup and recovery, other tools apply (see below).
LiveCache 7.4.x / 7.5 Performing a backup for liveCache 7.4 is basically the same as backing up the database of an APO system (APO DB).
To schedule backups and other SAP R/3 database maintenance tasks as background jobs, you can use the CCMS Database Planning Calendar. In APO 3.1, such weekly scheduling is only supported for APO DB. From mySAP SCM 4.0 (SAP Basis Release 6.20) onwards you can schedule liveCache backups from the CCMS Database Planning Calendar. You can schedule backups of liveCache in background jobs or by executing external commands.
Logging
LiveCache 7.4 features complete physical logging for all changes on persistent objects. The logging is similar to logging in RDBMS. The differences have only a minor effect on administration.
In liveCache, changed data is written to disk at regular intervals during checkpoints. This normally occurs every 10 minutes. Depending on the processed data volume, there is high I/O throughput on the log devspaces. At times of especially intensive data changes, for example during planning runs, writing the log areas can have a negative impact on performance.
The liveCache log devspaces are the disk areas with the greatest write activity. Since the data throughput for the log devspaces is so critical, disk layout should be planned thoroughly with the SAP hardware partner. For general recommendations about disk layout see the Installation Guides.
Best Practice: APO Backup and Availability
2004 SAP AG
7
Backup Cycle
A typical backup cycle for liveCache is shown in the following graphic. The term Gen specifies the generation of tapes used in the backup cycle.
SAP AG 2001, Title of Presentation, Speaker Name 26
liveCache Backup Cycle
LOG_00005
SAVEDATA (Full backup)SAVEDATA (Full backup)
SAVE PAGES (DATA incr.)SAVE PAGES (DATA incr.)
DAT_00014
AUTOLOGAUTOLOG
0101 0202 03030404
05050606
0707
28282727......
08081. Gen.1. Gen.
2. Gen.2. Gen.3. Gen.3. Gen.
4. Gen.4. Gen.
DAT_00001
DAT_000
DAT_00010
LOG_00002
PAG_00004LOG _00003
PAG _00005
060708
...
LOG_00006
LOG_00008
0909
VERIFYVERIFYLOG _00003 Check BACKUPCheck BACKUP
.. 2..3
1. Perform a backup
Backups are typically created online either as incremental backups (daily) or as full backups (weekly).
Perform backups using
liveCache administration tools
CCMS Database Planning Calendar ( as of SAP Basis Release 6.20 )
A job from within the SAP APO system that
o Starts report RSLVCBACKUP - see SAP Note 455154 o Starts an external command using DBMCLI
Third party tools using backup interfaces
Log backups can be made in parallel to online liveCache backups either
Manually by an administrator or
Periodically by CCMS Database Planning Calendar ( as of SAP Basis Release 6.20 ) or
Automatically in liveCache mode Auto Log
Make sure log backups are performed regularly to prevent liveCache standstills due to log overflow.
For details on how to perform backups, see SAP liveCache 7.4: Backup and Recovery & Administration News (E).
2. Verify the database
Verification of the database is required to detect corrupt data structures before you enter a new backup cycle. Therefore, perform verification at least once per backup cycle. Verification runs may have an impact on system performance and lock situations. For details see SAP Note 521870.
Use liveCache administration tools or schedule a job in the SAP APO system using the applicable DBMCLI command as external job step. See SAP Note 506981.
Best Practice: APO Backup and Availability
2004 SAP AG
8
3. Check the backup
Checking a backup ensures that it can be read from the backup medium. This enables you to detect hardware failures. You should perform a backup check at least once per backup cycle.
To perform backup checks, use liveCache administration tools or schedule a job in the SAP APO system using the applicable DBMCLI command as external job step.
LiveCache 7.2 In liveCache 7.2, there is no database logging, but logical logging is realized on application level. This means that changes to data in liveCache are logged at logical level in APO database. The logging does not include all the objects in the APO application module. The logging does not include changes to data in the area Demand Planning (DP), changes to inactive plan versions, or changes to some Automotive-specific data.
Checkpoint During a checkpoint, all the changes that were initially made only in the cache area of liveCache are saved to the data devspaces of liveCache in accordance with a previously defined schedule. The checkpoint is triggered by the execution of a background job on the APO system. The checkpoint ensures that data is consistent between the data cache and the data devspaces on hard disk. Also, liveCache itself needs to be in a consistent state, so write transactions are disabled for the time of the checkpoint.
How often should checkpoints be written?
Writing checkpoints can hinder work in the APO system, since no new transactions are started until all running transactions are ended. On the other hand, a checkpoint substantially increases data security. So for data security it is advantageous to trigger checkpoints as often as possible. Especially if you use DP, which is not covered by the database logging, you should write checkpoints at intervals of 2-3 hours. If you run critical long-running transactions such as planning runs, the checkpoint report can also be scheduled as directly following job steps or be triggered by those events.
The time it takes to perform a checkpoint increases when the log area is activated, since at each checkpoint data is copied from the deactivated log area into the log area. How long this takes depends mainly on how much data is copied.
Backup of liveCache 7.2
Backup of liveCache 7.2 uses the checkpoint report. Whereas in liveCache 7.4 the DBMGUI suffices for a backup, in liveCache 7.2 first a checkpoint must be written and logging switched. Therefore, when performing a backup you must always use the checkpoint report. Do not perform backups using only DBMGUI, as this falsifies the logging in the APO system and leads to inconsistencies.
For details, see SAP APO, SAP liveCache 7.2 Backup and Recovery, or SAP APO System Administration.
Optimizer The Optimizer does not contain a database. It stores data in memory only. Therefore only an operating system backup is required to secure the installation. Perform backups after any major software changes.
To enhance operational security, save optimizer logfiles regularly, preferably at least once a week.
Scheduling Online backups
Online backups affect overall system performance and should therefore be scheduled to run at times of low system workload.
Tip APO DB and liveCache can be backed up online at the same time. Simultaneous backups are easier to administer and low system workload affects both systems.
Best Practice: APO Backup and Availability
2004 SAP AG
9
In general, there is no dependency between the backups taken of the different databases forming the SCM landscape. Logs are used to recover any of the databases to the point in time of the last committed transaction. Except in the case of liveCache 7.2, database backup scheduling does not depend on the application scenario.
Offline Backups
SAP R/3 and APO offline backups may be performed at different times and independently of each other. The core interface (CIF) ensures that any requests sent are stored in one system while the partner system is down. No data gets lost.
Tip Perform offline backups of the APO DB and liveCache at the same time in order to reduce planned downtime. LiveCache is not accessed if the APO system is down.
Perform simultaneous offline backups of liveCache, APO DB, and R/3 DB if you want to create a consistent image of the SCM landscape, for example for system copies.
Monitoring of Backups Currently, backups need to be checked for each system component. It is planned to offer backup monitoring support via the SAP Solution Manager. Your operations manual has to include company-specific procedures about backup monitoring and reaction methods on backup failures.
SAP APO and SAP R/3 Use the CCMS Database Planning Calendar and the central CCMS monitoring or external backup tools.
LiveCache Use the CCMS Database Planning Calendar for SAP Basis Release >= 6.20 or the SAPDB tools and the logs of scheduled background jobs or external backup tools.
Other Components and Operating System Backups Define your own procedures in your operations manual for monitoring backups.
Consistent Landscape Backups A backup of the entire system landscape is performed component by component, for each database. If the backup is performed online, a consistent backup of the entire system landscape is not ensured.
Simultaneous Offline Backups A consistent backup of the entire system landscape is only necessary in exceptional cases, such as before a system upgrade or to create a regular system copy. The simplest way to create a consistent backup is to make simultaneous offline backups of all components during a sufficiently large maintenance window, such as over a weekend.
Technical Realization There are various methods that are recommended by storage system manufacturers. These methods allow you to make a consistent backup of several databases without shutting down the databases or scheduling a downtime.
One such method is to make a split mirror backup together with suspend write.
For details, see the Best Practice Backup and Restore for mySAP.com.
Best Practice: APO Backup and Availability
2004 SAP AG
10
Restore and Recovery Because of the architecture of an APO system, a crash can affect various system components. The following cases can be distinguished:
Crash of APO DB
In this case, a recovery with the help of the relevant RDBMS is required. The procedure is familiar from the SAP R/3 System.
Crash of liveCache
APO DB is not affected, but there is little or no availability of the APO system as a whole. In this case, liveCache needs to be recovered. Note that the recovery procedure is different for liveCache 7.2 and liveCache 7.4.
Crash of APO DB and liveCache
The procedure depends on the liveCache release.
For liveCache 7.4, the recoveries of APO DB and liveCache can run in parallel.
For liveCache 7.2, the recovery of APO DB can run in parallel with the restore of liveCache. The liveCache recovery can only be performed after successful recovery of APO DB.
Preparation In an APO system crash, the connected OLTP systems are also indirectly affected. Therefore, it is advisable first to interrupt the data transfer from these systems. Find out which SAP systems are transferring data to the APO system through the CIF interface. Stop the outbound queues going to the unavailable APO system. If the APO system is unavailable, the queues will stop anyway with an error message. However, the source systems will make multiple attempts to send the data. Prevent this if possible, as it puts an extra load on the source systems.
Analyze the crash situation. If an APO DB recovery is required, the technical procedure corresponds to the DB recovery of SAP R/3 Systems. Use the relevant database tools to perform a complete recovery. The APO system can then be reconnected for data transfer by restarting reports to reactivate the inbound und outbound queues. This can also be scheduled to occur periodically. See SAP Note 442478.
If a liveCache recovery is required, the situation is different. In this case, the APO instance remains available, so user actions can lead to runtime errors. Therefore, it is advisable to lock the APO system against further activities. These include:
User dialog activities
Background jobs
Processing of inbound queues or incoming requests from the outbound queues of dedicated systems
SAP APO and SAP R/3 If an SAP R/3 or APO system crashes, first stop the outbound queues from the APO system to the crashed R/3 System or the outbound queues from the R/3 System to the crashed APO system. The recovery procedure itself is as usual.
LiveCache
LiveCache 7.4 Since liveCache 7.4 uses full database logging, recovery is performed in a similar way as for SAP DB using SAP DB tools.
The following graphic shows what is performed during liveCache instance recovery in a case where there is hardware failure where no datafiles are lost.
Best Practice: APO Backup and Availability
2004 SAP AG
11
The technical terms of data and log devspaces, which are included in this document, correspond to the terms data and log volumes and will be replaced by them in the future.
SAP AG 2001, Global Support 1
Recovery Process
Restart to latest consistent data state data and log are still available on liveCache devices
No action: for transactions T1 and T5Redo: apply after images stored in the archive log for T4 starting at S3
and for T6 completelyUndo: apply before images stored in the undo files for T3 and T2,
both actions starting at S3
T1 Commit
T2
t
T3 Rollback
T4 Commit
T6 Commit
Checkpoint S1 CrashCheckpoint S2 Checkpoint S3
T5 Rollback
Instance recovery during restart of liveCache:
The restart ensures that transactions that were open at the time of the last checkpoint and committed at the time of the crash are redone. Transactions that were open at the time of the crash are only rolled back if they were open at the last checkpoint.
The starting point for the redo/undo operations is always the last checkpoint. Any data that was written later to data devspaces is ignored.
In the above graphic: Transactions 1 and 5 are irrelevant for redo/undo operations. Transaction 1 was committed at
the last checkpoint. The changes were written to the data devspaces. Changes from transaction 5 are not in the data area of the last checkpoint.
Transactions 2, 3, and 4 were not completed at the last checkpoint. Restart of liveCache redoes transaction 4.
Transactions 2 and 3 are undone in a rollback to the last checkpoint. Transaction 6 is completely redone during the restart. Changes from transaction 6 are not in
the data area of the last checkpoint.
Procedure in case data files need to be restored
1. Restore liveCache data files
This occurs in liveCache mode Cold. As of liveCache version 7.5 this mode is called Admin.
2. Recover the logs
If all required log data is still available in the log devspaces, liveCache can be started (instance recovery). LiveCache is automatically rolled forward with the data from the log. If you need to play back log data, do not start liveCache yet, and do not set it to mode Warm (or Online as of liveCache version 7.5). First, play back the log backups and/or restore the incremental backups.
3. Restart liveCache
Once the required data and log backups are played back, liveCache automatically restarts into Warm (Online as of liveCache version 7.5) mode. Please consider that it is recommended that the liveCache
Best Practice: APO Backup and Availability
2004 SAP AG
12
be restarted with the APO-liveCache administration transaction LC10 to ensure that post processing procedures are scheduled.
4. Resuming work in the APO system
The APO system can now be used again. Users can work and background jobs can run. Since many items may have accumulated in the queues, they should be reactivated in stages if possible.
Several options are available for the backup and recovery of a liveCache instance. The use of the tools or the line commands is as for SAP DB - the backup procedure for liveCache is integrated into the Computer Center Management System (CCMS) as of mySAP SCM 4.0 (with SAP Basis Release 6.20).
Factors influencing the time needed for liveCache restore:
Case 1. Data files not damaged
Amount of changed data
Number and length of transactions that must be rolled back
Hardware (CPUs)
Case 2. Data files must be restored
Backup infrastructure used
How much can be done in parallel
Whether logs must be played back
Plus case 1 factors
For more information about B&R, see:
SAP Training ADM55
www.sapdb.org
LiveCache 7.2 Since liveCache 7.2 has no physical logging, the recovery scenario is different from liveCache 7.4.
Do not use DBMGUI to set liveCache in mode Warm (Online as of liveCache version 7.5). To continue a recovery, use the APO system.
Logging with Checkpoints
There is no database log written but checkpoints are written at discrete intervals. If a recovery is necessary, data loss can be expected. In this case, a roll forward of liveCache is impossible. After the restore of a backup, the next step is to restore consistency between liveCache and APO DB and between the APO system and the dedicated R/3 Systems. The duration of this process depends on the data volume involved.
Alternatively, reinitialize liveCache and make an initial data transfer from the connected R/3 Systems. This limits the data loss to APO-specific data such as planning versions. To decide which procedure to use, consider the time required and the possible data loss. If you wish to offer the initialization strategy as an alternative, it must be developed and tested and then documented in the operations handbook. Key factors are the duration of the initial data transfer and the resulting data quality. Your choice of strategy also depends on whether your business processes use data that cannot be restored from an initial data transfer, such as Automotive data or orders and requests created only in APO.
Recovery with Synchronous Logging
In synchronous logging, besides checkpoints, each change between two checkpoints is recorded at object level in APO DB. Recovery of liveCache is performed using a report at APO system level. You can choose to use the logs in the recovery process. On completion of the recovery, liveCache is automatically restarted. The administrator is informed of completion by e-mail. The duration of the process depends mainly on the data volume to be restored.
Best Practice: APO Backup and Availability
2004 SAP AG
13
Tip To start liveCache, always use the APO-liveCache administration transaction LC10. Otherwise there is no synchronization between APO system and liveCache. Use other tools (such as DBMGUI) only to start liveCache in write-protected mode.
When the recovery is completed, reschedule checkpoints.
In APO 3.0 with liveCache 7.2, there is no logging for the component DP. In the case of a failure, data loss is to be expected. Therefore, if you use DP, a full recovery must include treatment of the DP data. For this purpose, you can do the following.
If a recent backup of the DP data is available in InfoCubes, you can reload the data. This process is described in detail in the APO documentation. The process can take a long time, at least as long as the creation of the DP data in the InfoCubes.
For further details, see the document SAP APO, SAP liveCache 7.2 Backup and Recovery.
Optimizer To restore the Optimizer, restore the latest operating system backup.
Test Cases When you set up a company-specific operations manual, you must test backup and recovery of the whole system landscape. It is recommended that test procedures be established for each database involved.
SAP APO and SAP R/3 Depending on the database type and release, the test procedure should include the following:
Hardware crash (instance recovery)
Crash of a disk with data files/devices
Test example for Oracle database:
Simulate a disk crash by deleting directory sapdata1
Recover your database completely using one of the backups you performed. Use the SAPDBA function Partial restore and complete recovery. After you have eliminated the error, again choose Partial restore and complete recovery.
Core Interface between SAP APO and SAP R/3 Test procedures should include volume tests to verify that during planned or unplanned downtime, CIF queues
Do not increase in size beyond the limitations of the database (to avoid tablespace overflow). See also SAP Note 505304.
Can be restarted correctly after recovery
Can be processed fully and with acceptable system performance even if queues are long
The tests presuppose that downtimes are not so long that too many items accumulate in the queues. If the queues are too long, it may be difficult to process them in the receiving system.
LiveCache 7.4 The test procedures from SAP DB apply. Test
Hardware crash (instance recovery)
Crash of a disk with data devspaces (restore and recovery)
Crash of a disk with log devspace. Note that loosing log information must be avoided by means of hardware redundancy and always causes data loss in the system. For more information see section Incomplete Recovery.
Best Practice: APO Backup and Availability
2004 SAP AG
14
LiveCache 7.2 Test restore and application recovery using the document SAP APO, SAP liveCache 7.2 Backup and Recovery.
High Availability This section lists HA options and considers their impact on level of availability and cost of ownership.
If you need an HA solution for your mySAP SCM landscape, you must consider:
The SAP R/3 System
APO DB
liveCache
APO optimizer
Since APO DB is based on the RDBMS familiar from R/3 Systems, you have the same HA options.
As for liveCache:
LiveCache 7.2 supports cluster solutions. However a cluster solution should only be used in case used applications are included in synchronous logging. In particular, when using the DP component you cannot use automatic switching but need manual action.
LiveCache as of 7.4 has familiar RDBMS behavior. Cluster solutions are fully supported.
One of the most important aspects of HA is the backup and recovery strategy for the APO system. This involves:
B&R of the SAP R/3 DB
B&R of APO DB
B&R of liveCache
Backup and restore of APO optimizer
Main Factors In a simple environment without an HA solution, unplanned downtime depends on the following main factors:
Detection of system crash
Replacement of failed hardware (service level contract with hardware partner)
Restoration of operating system
Restoration of database (and thus speed of tapes and degree of parallelization)
Recovery of database (number of transactions to recover, CPUs, )
The time it takes to restore an SCM landscape is set by the longest component restore time.
For information about disk technology, see the SAP documentation on Disk Technology and the Installation Guides.
Best Practice: APO Backup and Availability
2004 SAP AG
15
Clustering
Hardware clusters protect your system against hardware failure but not against disk failure.
SAP R/3 and SAP APO System The standard HA solutions are in place. See http://service.sap.com/ha
Clusters protect against hardware failure but not against disk failure.
LiveCache 7.4 A cluster solution is provided for liveCache 7.4 by SAP hardware partners.
For more information contact your SAP hardware partner.
Cluster support is based on the standard recovery process with the prerequisite that data and logs are stored on secure disk systems. At switchover there are two liveCache instances that access the same data resource (generally a disk subsystem). Of the two instances, only one is active at any time. If the active instance fails, for example due to a hardware failure, the second instance is activated on a different machine and receives authorization to write to the same data resource. The switchover is performed by an SAP partner product.
- Recovery time in case of a server failure includes the time normally required for the restart of a liveCache instance.
- Logical errors cannot be corrected or prevented by cluster support.
- Standard cluster solutions are offered by SAP hardware partners.
The following graphic shows liveCache in a cluster environment during normal operation.
SAP AG 2002
Cluster
Cluster Support
ArchiveLogData
l iv e C a c h el iv e C a c h el iv e C a c h el iv e C a c h e l iv e C a c h el iv e C a c h el iv e C a c h el iv e C a c h e
APO ABAP Application
primary backup
Best Practice: APO Backup and Availability
2004 SAP AG
16
The following graphic shows a liveCache cluster in which the primary server fails and a backup server takes over.
SAP AG 2001, Global Support 16
Cluster
Cluster Support
ArchiveLogData
l iv e C a c h el iv e C a c h el iv e C a c h el iv e C a c h e l iv e C a c h el iv e C a c h el iv e C a c h el iv e C a c h e
APO ABAP Application
IP SWITCH
AUTO RESTART
primary
RECONNECT
backup
A liveCache cluster scenario
When liveCache is switched over after hardware failure, it is started by means of cluster scripts. Once liveCache has reached status Cold (as of liveCache version 7.5: Admin), instance recovery begins.
If the instance recovery was performed successfully, liveCache is switched to status Warm (as of liveCache version 7.5: Online). At this point in time, liveCache can be accessed by application programs but the data is not yet in the data cache. In prefetching, data pages are read by liveCache processes into main memory. After prefetching, all data from the data cache is in main memory. This improves performance.
The following graphic shows the typical switchover process for liveCache.
The following liveCache statuses are displayed:
LC red circle: liveCache is in status Offline
LC yellow circle: liveCache is in status Cold (as of liveCache version 7.5: Admin) and performs instance recovery
LC green circle: liveCache is in status Warm (as of liveCache version 7.5: Online) and starts with prefetching
Best Practice: APO Backup and Availability
2004 SAP AG
17
SAP AG 2001, Title of Presentation, Speaker Name 19
Measurements on liveCache 7.4.2.003
TimeFailover InstanceRecovery Production Operation
liveCacheliveCache
CrashliveCacheliveCache
Ava
ilabi
lity
1
0LC
LCLC
Pref
etch
Fill
grad
e
Timing of liveCache cluster switchover
The following graphic shows what happens over time when liveCache is switched to the backup node. At the time of switchover, no transactions are running.
After a few minutes, liveCache is in status Warm (as of liveCache version 7.5: Online). Once liveCache is in status Warm data can be accessed. However, performance is restricted until all data pages are read into main memory.
The graphic shows a switchover process with subsequent prefetching. The times are only typical. Actual times depend strongly on the installed hardware and system configuration.
SAP AG 2001, Title of Presentation, Speaker Name 20
Switchover from Node D1 to Node D2: Timeline
Data 4 GB
Data cache 5 GB
Switch liveCache at simulated hardware failure
0
100000
200000
300000
400000
500000
600000
1 4 7 10 13 16 19 22
time [mins]
8K b
lock
s re
ad
D1offline
D2primary
LC8 s 60 s
LC57 s
~ 20 min
LC LCliveCache COLD liveCache WARM
Best Practice: APO Backup and Availability
2004 SAP AG
18
LiveCache 7.2 For most applications (except e.g. Demand Planning), liveCache 7.2 uses synchronous logging.
Clustering makes sense if all APO applications used by the customer are fully included in synchronous logging. However, expectations as to unplanned downtime must be limited by the time needed for application recovery and checkpoint intervals.
Clustering makes no sense if a core business process of the customer is not included in synchronous logging, since manual action is required after a failure.
Optimizer High availability of the SAP Optimizer can be achieved through Installation of the APO optimizer programs on several servers. An Optimizer availability check can be activated in APO which is executed before each Optimization run. If one Server fails it is being recognized by the Application and the Optimization run is started on the alternative Host. Optimizer Runs can also be distributed to both machines during normal productive use to provide load balancing.
The high availability concept can not rescue the actual state of the aborted optimizer run the optimizer run has to be repeated completely but it can ensure immediate availability of the optimizer software on a backup location.
For details see the Installation Guides
Note that there is no need to use cluster software.
SAP AG 20012
Optimizer High Availability
APO Application
APO Application
Optimizer 1Optimizer 1
APO-DB
liveCacheliveCacheliveCache
Optimizer 2(standby)Optimizer 2(standby)
Fast Restores Instead of backing up the database to tape, backups can be performed to fast disks or using a split mirror backup method. Compared to tapes, this allows much faster restoration times.
SAP R/3 and SAP APO System Depending on the size of the database, use of fast backup media can ensure that restoration times meet your maximum unplanned downtime requirements.
Best Practice: APO Backup and Availability
2004 SAP AG
19
LiveCache The database size of liveCache is typically much smaller than that of R/3 and APO systems. Therefore, restoring liveCache from disk or a similar medium may be fast enough.
Standby Database Standby databases are replicated databases. They offer protection against:
Disk failures (the shadow database uses its own dataset)
Errors in the restore/recovery process (the shadow database is always in recovery mode)
Logical errors (if the shadow database is run with a delay)
The process of shipping logs from one database to the standby database is performed by SAP partner products.
SAP R/3 and SAP APO System The standard HA solutions are in place. See also http://service.sap.com/ha
LiveCache 7.4 The Standby Database is available with liveCache 7.4. For details see http://help.sap.com -> SAP Netweaver -> SAP Web Application Server -> SAP Web Application Server 6.30 -> English -> SAP Netweaver Components -> SAP DB -> Basic Information -> Background Knowledge SAP DB 7.4 -> Standby Databases with SAP DB -> Setting up a Standby Database.
Hot Standby Database In a hot standby system, data is replicated on a transactional basis. Intelligent storage systems are required to support this solution. Main features:
Only committed transactions are replicated.
There is minimal delay for replication: in the worst case only some transactions has to be redone upon switchover.
The replicated database is already in memory, so startup time is minimized and the data cache is already filled.
LiveCache The hot standby database scenario is available as of mySAP SCM 4.1 with liveCache 7.5. For details see http://service.sap.com/scm >> Technology.
Use of liveCache hot standby is appropriate if extremely high availability is required and extreme high data volume needs to be processed. Since the hot standby is in memory, it offers good performance immediately after switchover. In the hot standby, there are no uncommitted transactions to be undone.
Twin Sites The above concepts may be applied to twin sites to protect system operation against the breakdown of an IT center. If you are interested, talk to your hardware partners.
Best Practice: APO Backup and Availability
2004 SAP AG
20
Incomplete Recovery In an incomplete recovery, the system landscape or one of its databases is reset to an earlier point in time before a crash. An incomplete recovery may be caused by logical errors or loss of log data. To prevent loss of log data, use redundant disks.
Logical Errors In most cases, logical errors are the reported reasons for resetting a database. In such situations, there is usually an alternative. For a detailed discussion, see SAP Note 434645.
Symptom Corrective measure
Table entries or entire tables corrupt or deleted
Recreate data using redundant data from other tables or recreate field values partly from redundant table entries in other tables of the same system
Table indexes corrupt Delete index on DB level and recreate using table data.
Corrupt blocks on file system level prevents successful database recovery
If corrupt block belongs to table index, SAP or database vendor support may be able to restore database
The index must be recreated afterwards
Objects destroyed by accidental import
Create and apply correction request
Other user errors resulting in data inconsistencies
Use specific report to repair inconsistencies in the application
Client deleted accidentally See SAP Note 31496
Consistency Once a database of a mySAP SCM landscape is recovered incompletely, data inconsistencies or losses may rise between the SAP APO and SAP R/3 and/or the SAP APO system and liveCache.
This applies specifically to data imported by external interfaces.
SAP provides tools to detect and repair inconsistencies. However, repairing inconsistencies can lead to data loss.
Tip The SAP tools for checking and repairing data inconsistencies are not a required part of backup and recovery procedures. Consistency checks might be performed after a recovery if the planned downtime allows execution and evaluation by qualified application staff. It is recommended to perform consistency checks periodically to improve overall system operation stability.
For information about consistency check tools, see the document Consistency Checks.
Verification After you executed this Best Practice, check whether you produced the desired results as follows:
1. Perform B&R tests with your operations team using your operations manual.
2. Perform tests to check that your HA environment and configuration work and meet your time constraints.
Best Practice: APO Backup and Availability
2004 SAP AG
21
Further Information
Case Study Backup Scheduling The following graphic shows an example of a backup schedule for liveCache 7.2. A daily planning process starts with the inventory load in an SAP R/3 System (available stock). After this background step is finished in the R/3 System, the results are sent to the APO via CIF. All irrelevant orders (such as orders from the previous day that were not fulfilled) are deleted at this time. The operation planning scheme is described below.
SAP AG 2001, Title of Presentation, Speaker Name 6
APO
Supply Network Planning
R/3 System
SNP Interactive Planning
Purchasing
Create/Update purchase requisition
Daily Operation: Production Planning Example in APO
Load inventory
Deployment / TLB
Deployment run (stock, released process orders)
Create/change Truck orders
Release to PP/DS
PP/DS
Detailed scheduling
Adjust/Change Final plan
Convert planned orders in process orders
Release process orders (Manufacturing)
Transfer orders
Final Deployment run (stock, confirmed
process orders)
TLB run
The first deployment run has to be finished by 11 a.m. A checkpoint is scheduled explicitly (using synchronous logging) to run at that time. This checkpoint can either be scheduled for that time or triggered by the event of completion of the application job. Depending on the data volume load, further checkpoints with intervals varying from 4 to 6 hours may be set. The deployment plan is based on daily demand and available-to-deploy (ATD) quantity. With the results of this run, the trucks are ordered via telephone for the afternoon (4 p.m. the same day and 5 a.m. the next day) to deploy the ATD quantity from the production plants to the distribution centers.
These daily requests are illustrated below.
Best Practice: APO Backup and Availability
2004 SAP AG
22
SAP AG 2001, Title of Presentation, Speaker Name 2
Daily Requests
1. Deployment run for each of 2 plants
1 Day
Truck Ordering
Planned transport Orders into Confirmed Transport Order
Final Deployment run
TLB for filling trucks
11 a.m. 4 p.m.
Checkpoint Complete online backup
(incl. Checkpoint)
The final deployment run takes the confirmed production orders of the current day into account. After the deployment run, the system automatically converts planned transport orders into confirmed transport orders, which are used for transportation planning in the transport load builder (TLB). The TLB is executed to ensure that trucks are filled at least to minimum capacity and if possible to maximum capacity. Both steps must be finished at 4 p.m. to ensure that the trucks can be loaded in time. A complete daily backup is then scheduled, for example as a background job.
Case Study High Availability Options The following customer example illustrates availability and cost considerations. The options are based on liveCache 7.4.
Simple Environment A simple environment does not have any redundancy in place. Therefore, recovering from the crash of a single component may take up to one day.
Type of fault: Disk crash or server failure
Best Practice: APO Backup and Availability
2004 SAP AG
23
SAP R/3R/3
Database
SAP APOAPO
Database
LivecacheLivecacheDatabaseCOM
Routines
covers 1 faultwithin next day
Spare Server Environment In this environment, a spare server is added. The spare server includes a cluster installation for one or more of the following:
liveCache
SAP APO central instance
SAP APO database
SAP R/3 central instance
SAP R/3 database
Type of fault: disk failure or server failure
Impact on availability
A downtime of 4 hours is expected for disk failure. Downtimes for server failures depend on the cluster product and database recovery.
Impact on costs of operation
Assumption: no performance loss after switchover
Additional server is required. Sizing of this backup server is the same as the sizing for the biggest of all the primary servers.
License fees for cluster software
Best Practice: APO Backup and Availability
2004 SAP AG
24
Cluster
SAP R/3R/3
Database
SAP APOAPO
Database
LivecacheLivecacheDatabase
Failover
COMRoutines
covers 1 faultwithin 4 hours
Spare Server and Fast Recovery Environment In this scenario, the Spare Server Environment is extended by fast disks to allow a restore from disk instead of tape.
Type of fault: disk failure or server failure
Impact on availability
This scenario reduces the recovery process after a disk failure to 2 hours. Downtimes for hardware failures depend on the cluster product and database recovery.
Impact on costs of operation
Additional disk space is required. Disk space is sum of space required by:
SAP APO DB
SAP R/3 DB
liveCache
SAP R/3R/3
Database
SAP APOAPO
Database
LivecacheLivecacheDatabase
Failover
COMRoutines
3rdmirror
3rdmirror
3rdmirror
covers 1 faultwithin 2 hours
Cluster
Best Practice: APO Backup and Availability
2004 SAP AG
25
Standby Databases In this scenario, the Spare Server and Fast Recovery Environment is extended by shadow databases for:
SAP APO DB
SAP R/3 DB
liveCache
Type of fault: disk crash or server failure
Impact on availability
This scenario accelerates the recovery process after a disk failure to 1 hour, since no restore is required. Downtimes for server failures depend on the cluster product and database recovery.
Impact on costs of operation
Additional disk space is required. Disk space is sum of space required by:
SAP APO DB
SAP R/3 DB
liveCache
License fees for shadow database software (automatic log shipping)
SAP R/3OLTP
Database
SAP APOOLAP
Database
LivecacheLivecacheDatabase
Cluster
COMRoutines
3rdmirror
ShadowLiveCacheDatabase
ShadowOLAP
Database
ShadowOLTP
Database
3rdmirror
3rdmirror
Extended Service Environment In this scenario, theStandby Databases scenario is extended by 0.5 FTE of service staff for permanent monitoring of the landscape.
Type of fault: disk failure or server failure
Impact on availability
With permanent monitoring, tasks that cannot be automated can be performed by service personnel and therefore after a disk failure an expected downtime of 30 minutes can be achieved. Downtimes for server failures depend on the cluster product and database recovery.
Impact on costs of operation
0.5 FTE of service personnel
Best Practice: APO Backup and Availability
2004 SAP AG
26
Cluster
SAP R/3R/3
Database
SAP APOAPO
Database
LivecacheLivecacheDatabase
Failover
COMRoutines
3rdmirror
ShadowLiveCacheDatabase
ShadowAPO
Database
ShadowR/3
Database
3rdmirror
3rdmirror
covers 1 faultwithin 0,5 hours
0,5 x
Second Spare Server In this scenario, the Extended Service Environment is extended by another backup server.
Type of fault: Disk crash or server failure
Impact on availability
Now two failures can be tolerated at the same time. A server failure is covered by a cluster and a disk failure by a shadow database.
Impact on costs of operation
Assumption: no performance loss after switchover
Additional server is required. Sizing of this backup server is the same as the sizing for the biggest of all the primary servers.
Cluster
SAP R/3R/3
Database
SAP APOAPO
Database
LivecacheLivecacheDatabase
Failover
Failover
COMRoutines
3rdmirror
ShadowLiveCacheDatabase
ShadowAPO
Database
ShadowR/3
Database
3rdmirror
3rdmirror
covers 2 faultswithin 0,5 hours
0,5 x
Best Practice: APO Backup and Availability
2004 SAP AG
27
Twin Site Environment In this scenario, the Second Spare Server scenario is extended by an additional remote IT center.
Impact on availability
Failure of an IT center can be tolerated.
Impact on costs of operation
Setup of an additional IT center is required.
Case Study Outline of an Operations Manual The following is a sample outline of an operations manual: General
Object, Purpose, Scope Technical Prerequisites Roles & Responsibilities System Monitoring
System Landscape Backup Strategy
R/3 APO DB liveCache Optimizer Flow Chart
High Availability Concept Detailed Procedures
Problem Discovery Problem Identification and Assessment Actions in the Application / Information Issued Problem Resolution
Failure of R/3 Server Failure Disk Failure
Failure of APO Server Failure Disk Failure
Failure of liveCache Server Failure Disk Failure
Failure of Optimizer Server Failure Disk Failure
Logical System Error Validation and Communication Lessons Learned: Failure Mitigation & Adjustment of Procedures
Appendix Process Flows Checklists Organization Charts Glossary
Best Practice: APO Backup and Availability
2004 SAP AG
28
Background Information and References
Database Manager GUI www.sapdb.org >> Documentation http://www.sapdb.org/pdf/dbmgui_73eng.pdf
Database Manager CLI www.sapdb.org >> Documentation http://www.sapdb.org/pdf/dbmcli_73eng.pdf
SAP DB Web Tools www.sapdb.org >> Software >> Choose Platform >> SAP DB Web Tools
External Backup Tools www.sapdb.org >> Documentation >> External Backup Tools http://www.sapdb.org/pdf/extsich_73eng.pdf
SAP APO, SAP liveCache 7.2 Backup and Recovery SAP Service Marketplace alias scm >> Technology http://service.sap.com/~form/sapnet?_FRAME=CONTAINER&_OBJECT=011000358700006126142001
SAP APO System Administration Author: Will, Liane, ISBN 3-89842-248-8 currently available only in German
Backup and Restore for mySAP.com SAP Service Marketplace alias bestpractices >> Solution Management http://service.sap.com/~sapidb/011000358700004447112001E
SAP Training SAP Service Marketplace alias education http://service.sap.com/education
Disk technology Help Portal http://help.sap.com/saphelp_46c/helpdata/en/08/57437e4ae611d1894f0000e829fbbd/frameset.htm
Installation Guides SAP Service Marketplace alias instguides http://service.sap.com/instguides
Consistency Checks Service Marketplace alias scm >> Technology http://service.sap.com/~form/sapnet?_SHORTKEY=01100035870000331500&
SAP liveCache 7.4: Backup and Recovery & Administration News (E) SAP Service Marketplace alias scm >> Technology
Best Practice: APO Backup and Availability
2004 SAP AG
29
http://service.sap.com/~form/sapnet?_FRAME=CONTAINER&_OBJECT=011000358700001330932002
SAP Help Portal http://help.sap.com
Feedback and Questions Send any feedback by formulating an SAP customer message to component SV-GST-SMC. You can do this at: http://service.sap.com/message
Best Practice: APO Backup and Availability
2004 SAP AG
30
Copyright 2004 SAP AG. All rights reserved. No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP AG. The information contained herein may be changed without prior notice. Some software products marketed by SAP AG and its distributors contain proprietary software components of other software vendors. Microsoft, Windows, Outlook, and PowerPoint are registered trademarks of Microsoft Corporation. IBM, DB2, DB2 Universal Database, OS/2, Parallel Sysplex, MVS/ESA, AIX, S/390, AS/400, OS/390, OS/400, iSeries, pSeries, xSeries, zSeries, z/OS, AFP, Intelligent Miner, WebSphere, Netfinity, Tivoli, and Informix are trademarks or registered trademarks of IBM Corporation in the United States and/or other countries. Oracle is a registered trademark of Oracle Corporation. UNIX, X/Open, OSF/1, and Motif are registered trademarks of the Open Group. Citrix, ICA, Program Neighborhood, MetaFrame, WinFrame, VideoFrame, and MultiWin are trademarks or registered trademarks of Citrix Systems, Inc. HTML, XML, XHTML and W3C are trademarks or registered trademarks of W3C, World Wide Web Consortium, Massachusetts Institute of Technology. Java is a registered trademark of Sun Microsystems, Inc. JavaScript is a registered trademark of Sun Microsystems, Inc., used under license for technology invented and implemented by Netscape. MaxDB is a trademark of MySQL AB, Sweden. SAP, R/3, mySAP, mySAP.com, xApps, xApp, and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany and in several other countries all over the world. All other product and service names mentioned are the trademarks of their respective companies. Data contained in this document serves informational purposes only. National product specifications may vary. These materials are subject to change without notice. These materials are provided by SAP AG and its affiliated companies ("SAP Group") for informational purposes only, without representation or warranty of any kind, and SAP Group shall not be liable for errors or omissions with respect to the materials. The only warranties for SAP Group products and services are those that are set forth in the express warranty statements accompanying such products and services, if any. Nothing herein should be construed as constituting an additional warranty. Disclaimer: SAP AG assumes no responsibility for errors or omissions in these materials. These materials are provided as is without a warranty of any kind, either express or implied, including but not limited to, the implied warranties of merchantability, fitness for a particular purpose, or non-infringement. SAP shall not be liable for damages of any kind including without limitation direct, special, indirect, or consequential damages that may result from the use of these materials. SAP does not warrant the accuracy or completeness of the information, text, graphics, links or other items contained within these materials. SAP has no control over the information that you may access through the use of hot links contained in these materials and does not endorse your use of third party Web pages nor provide any warranty whatsoever relating to third party Web pages.