DS Training
-
Upload
abdulshafique -
Category
Documents
-
view
225 -
download
0
Transcript of DS Training
-
7/30/2019 DS Training
1/14
DATASTAGE TRAINING
Date:-03/09/2012
Objective:-DataStage Backup is weekly activity tasks which will be perform every Monday after
Production Batch run. Two types of Backup we are taking.
. Configuration Backup
. Code Backup1. Configuration Backup: -
Configuration backup needs to be taken from 4 servers. They are:-
1. Production server (10.144.223.210)2. QA server (10.147.109.205)3. AT/TEST server (10.147.109.202)4. DEV server (10.147.109.201)1. Backup from Production server (10.144.223.210)
Step1:- Logon to code backup server (10.144.223.40) as a datastage user.
Then with the help of winscp toolusing dsadm user logon Production server (10.144.223.210)
Step2:- Go to/root/datastage/projects/
These are the four main projects inside the projects folder.
a.
DWH_EU_prdb. DW_INV_PRDc. EDI_prdd. SHAREreporting_prd
From each projects we will copy two files for configuration backup.
1. uvodbc.config2. DSparams
So from here we will take 8 files, 2 files from each project.
Step3:-We will put these files in the code backup server (10.144.223.40).
Create a folder like Backup_YYYYMMDD inside D:\DataStage_configfiles\Prodand put all 8
files inside this Backup_YYYYMMDD with the following nomenclature :-
1. DWH_EU_ prd _ YYYYMMDD_ uvodbc.config2. DWH_EU_ prd _ YYYYMMDD_ DSparams3. DW_INV_PRD_ YYYYMMDD_ uvodbc.config
-
7/30/2019 DS Training
2/14
4. DW_INV_PRD_ YYYYMMDD_ DSparams5. EDI_prd_ YYYYMMDD_ uvodbc.config6. EDI_prd_ YYYYMMDD_ DSparams7. SHAREreporting_prd_ YYYYMMDD_ uvodbc.config8. SHAREreporting_prd_ YYYYMMDD_ DSparams
Step4:- Go to/root/software/datastage/Ascential/datastage/DSEngine
Copy 3 files for backup
1. .odbc.ini2. dsenv3. uvconfig
Step5:- Go to D:\DataStage_configfiles\Prod on the code backup server (10.144.223.40).
Create Backup_YYYYMMDD folder.
Paste
1. .odbc_ YYYYMMDD.ini2. dsenv_ YYYYMMDD3. uvconfig_ YYYYMMDD
Step6:- Go to/root/software/oracle/product/10.2.0/network/admin
Copy
1. tnsnames.ora
Step7:- Go to D:\DataStage_configfiles\Prod on the code backup server (10.144.223.40).
Create Backup_YYYYMMDD folder.
Paste
1. tnsnames_YYYYMMDD.ora
2. Backup from QA server (10.147.109.205)Step1:- Logon to code backup server (10.144.223.40) as a datastage user.
-
7/30/2019 DS Training
3/14
Logon QA server (10.147.109.205) using winscp tool as dsadm user.
Step2:- Go to/root/datastage/projects/
These are the four main projects inside the projects folder.
a. DWH_EU_prdb. DW_INV_PRDc. EDI_prdd. SHAREreporting_prd
From each projects we will copy two files for configuration backup.
1. uvodbc.config2. DSparams
So from here we will take 8 files, 2 files from each project.
Step3:-We will put these files in the code backup server (10.144.223.40).
Create a folder like Backup_YYYYMMDD inside D:\DataStage_configfiles\Prodand put all 8
files inside this Backup_YYYYMMDD with the following nomenclature :-
1. DWH_EU_ prd _ YYYYMMDD_ uvodbc.config2. DWH_EU_ prd _ YYYYMMDD_ DSparams3. DW_INV_PRD_ YYYYMMDD_ uvodbc.config4. DW_INV_PRD_ YYYYMMDD_ DSparams5. EDI_prd_ YYYYMMDD_ uvodbc.config6. EDI_prd_ YYYYMMDD_ DSparams7. SHAREreporting_prd_ YYYYMMDD_ uvodbc.config8. SHAREreporting_prd_ YYYYMMDD_ DSparams
Step4:- Go to/root/software/datastage/Ascential/datastage/DSEngine
Copy 3 files for backup
1. .odbc.ini2. dsenv3. uvconfig
Step5:- Go to D:\DataStage_configfiles\Prod on the code backup server (10.144.223.40).Create Backup_YYYYMMDD folder.
Paste
1. .odbc_ YYYYMMDD.ini2. dsenv_ YYYYMMDD3. uvconfig_ YYYYMMDD
-
7/30/2019 DS Training
4/14
Step6:- Go to/root/software/oracle/product/10.2.0/network/admin
Copy
1. tnsnames.ora
Step7:- Go to D:\DataStage_configfiles\Prod on the code backup server (10.144.223.40).
Create Backup_YYYYMMDD folder.
Paste
1. tnsnames_YYYYMMDD.ora3. Backup from AT/TEST server (10.147.109.202)
Step1:- Logon to code backup server (10.144.223.40) as a datastage user.
Use winscp tool to log on to AT/TEST server (10.147.109.202) as dsadm user.
Step2:- Go to /root/software/datastage/Ascential/datastage/DSEngine
Copy
1) .odbc.ini2) dsenv3) uvconfig4) uvodbc.config
Step3:-Create a folder named Backup_YYYYMMDD and paste the copied files on server (10.144.223.40)
at D:\DataStage_configfiles\AT/TEST
Put the files inside this folder Backup_YYYYMMDD with the nomenclature as:-.
1) .odbc_ YYYYMMDD.ini2) dsenv_ YYYYMMDD3) uvconfig_ YYYYMMDD4) uvodbc.config _ YYYYMMDD
Step4:- Go to/root/software/oracleadm/product/10.2.0/network/admin
Copy
1. tnsnames.ora
-
7/30/2019 DS Training
5/14
Step5:- Create a folder named Backup_YYYYMMDD at D:\DataStage_configfiles\AT/TESTon the code
backup server (10.144.223.40).
Paste the file inside this folder Backup_YYYYMMDD with the nomenclature :-
tnsnames_YYYYMMDD.ora
4. Backup from DEV server (10.147.109.201)
Step1:- Logon to code backup server (10.144.223.40) as a datastage user.
Use winscp tool to log on to AT/TEST server (10.147.109.201) as dsadm user.
Step2:- Go to /root/software/datastage/Ascential/datastage/DSEngine
Copy
1) .odbc.ini2) dsenv3) uvconfig4) uvodbc.config
Step3:-Create a folder named Backup_YYYYMMDD and paste the copied files on server (10.144.223.40)
at D:\DataStage_configfiles\AT
Put the files inside this folder Backup_YYYYMMDD with the nomenclature as:-.
1) .odbc_ YYYYMMDD.ini2) dsenv_ YYYYMMDD3) uvconfig_ YYYYMMDD4) uvodbc.config _ YYYYMMDD
Step4:- Go to/root/software/oracleadm/product/10.2.0/network/admin
Copy
1. tnsnames.ora
Step5:- Create a folder named Backup_YYYYMMDD at D:\DataStage_configfiles\AT on the code backup
server (10.144.223.40). Paste the file inside this folder Backup_YYYYMMDD with the nomenclature:-
tnsnames_YYYYMMDD.ora
-
7/30/2019 DS Training
6/14
Backup from Prod server in UNIX (10.147.109.210)
Logon through PUTTYinto 10.147.109.210using dsadm as a user.
Put all the files inside this location/software/configbkup/
Goto this path/root/software/oracleadm/product/10.2.0/network/admin
Type :- ls a
List of all the files will come.
Use this command to copy the file and put into desire location.
Type:-c p tnsnames.ora /software/configbkup/tnsnames/tnsnames_YYYYMMDD.ora
Goto this path:-/root/software/datastage/Ascential/datastage/DSEngine
Type :-lsa
List of all the files will come.
Type:-c p .odbc.ini /software/configbkup/odbc/.odbcYYYYMMDD.ini
Type:-c pdsenv /software/configbkup/ dsenv/dsenvYYYYMMDD
Type:-c puvodbc.config /software/configbkup/ uvodbc/ uvodbc YYYYMMDD.config
Type:-c puvconfig /software/configbkup/ uvconfig/uvconfigYYYYMMDD
1. Code Backup from Prod server (10.147.109.210)There are two types of code backup.
1. Manual code backup
2. Automated code backup
1. Manual code backup: - Logon to code backup server (10.144.223.40)
Step1:-Open Administrator client then click on Projects tab, select one project then click properties
tab then click on the unprotected tab then ok.
-
7/30/2019 DS Training
7/14
Step2:-Same way unprotect the all 4 projects one by one before code back.
Step3:-Open Manager Client, click on Exporttab then click on the Datastage componenttab, select
the path where you want put the backup file. Select the file
-
7/30/2019 DS Training
8/14
Required path is:-D: \Prod_Datastage_Code_Backups\ Backups_YYYYMMDD\
Here we are exporting only .dsx file one from each project.
SHAREreporting_prd_ YYYYMMDD.dsx
EDI_prd_ YYYYMMDD. dsx
DW_INV_PRD_ YYYYMMDD. dsx
DWH_EU_ prd _ YYYYMMDD. Dsx
-
7/30/2019 DS Training
9/14
Step4:-After code backup put the projects into protected mode.
-
7/30/2019 DS Training
10/14
1. Automated code backup:-Logon to code backup server (10.144.223.40)
Goto this path:- C:\Documents and setting\datastage\DS_Backup_xxx.bat
Double click on the backup bat file it will run and take the backup.If you want to verify this backup goto
C:\Prod_Datastage_Code_Backups\ExportedBackups\Prod_Backup.
Here we can check the above code backup has been taken correctly or not.
-
7/30/2019 DS Training
11/14
Date:-04/09/2012
ntroduction and Project Overview and server configuration
There are 2 DB servers.
SHARE(Hosted on Oracle)
SMART(Hosted on SIEBEL)
-
7/30/2019 DS Training
12/14
Mail Delivery List
Guy Allen- QA deployment manager Sarah Barker Prod deployment Manager Mark Woods & RosarioDatastage SME Hans & Robert Arnold - DBA
Stages in Batch run
Extraction -> 2. Staging -> 3. BIW -> 4. MV refresh -> 5. End ETL
All the sequences are linked with SeqDW_ETL_Main. We have to start this sequence first and it will automatically
rigger other sequence. If there is any abort during the batch run, the status of the job will be abort. The sequence
elated to concerned job will be in Abort/Restartable state (otherwise Running/Restartable). SeqDW_ETL_Main
will also be in Abort/Restartable state if the sequence inside it will be in Abort/Restartable state. Once ETL Main
s in Abort/Restartable state, Seqrecovery.InstanceName, which is DWH_ONE in production
Seqrecovery.DWH_ONE), will start automatically. Once Seqrecovery.DWH_ONE completed successfully, we can
estart the batch run from SeqDW_ETL_Main if it is an approved error. If Seqrecovery.DWH_ONE fails then we
ave to reset the sequence and manually submit the job. Once seq.recovery completed we can submit
eqDW_ETL_Main.
. Extraction-> Name ofthe sequence is SeqExtSrcToStg. It uploads the data from DB(Oracle and Seibel) to the AIX server
10.144.223.210
1st sequence to run. If there is any issue, then it should be looked into by E-Team(Oracle DB issue) and ES DBA team (Siebel
DBA Issue)[email protected] heading [email protected] the DL
Schemas present in this stage is Oracle_DWH and Siebel_DWH. It takes approximately 1 hr. for the extraction to complete.
. Staging -> Name of the sequence is SeqStgToBDW. It uploads data from AIX server (10.144.223.210) to BDW
database.
First the comparison takes place between data already present in CDC tables and the extracted data. CDC tables are in staging_once schema. DCD tables ID are in CDC_Lookup and are also is in staging_one schema. Schema names are same in QA. BDW_ONE(used for DW_EU_PRD project) and BDW_01(used for DW_ENV_PROD) schemas are also
present in BDW database.
It takes approximately 1:30 Hrs. to complete.
mailto:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected] -
7/30/2019 DS Training
13/14
. BIW -> Name of the sequence is SeqBDWtoBIW. There are 2 sequences that run inside this sequence.
o Load Dimension (seqLoadDimension). This sequence runs first as it holds the primary key to thefact tables. It takes 10-15 minutes to complete this sequence
o Load Facts (seqLoadFact). This will update the fact tables. It takes 4-5 hours to complete thesequence.
o This database is being looked in to by Oracle DBA.4. Refresh MV ->
Materialized views are being updated in this stage. Materialized views are stored physically on the server. It takes 30 minutes to complete this stage. Cube build starts after this stage.
. End_ETL -> Name of the sequence is SeqEndETL. It takes 5hours to complete this sequence. No SLA for this but we have to monitor this. This is a house-keeping job which clears dataset created in the scratch space of application server
(10.144.223.210).
Datasets contains temporary data created in different stages of batch run. AnalyzeDBSchema is a job which runs in SeqEndETL. It gathers the table data.
Projects in Datastage:-
DW_EU_PROD This is the only project with SLA. EDE_PRD
o 1st week of the month after batch run, mostly after 3:00PM BST.o Ticket needs to be assigned to DWH queue before starting with the project.
DW_INV_PRDo Order and stock reconciliation are part of this project.o Order Starts at around 5:10 AM BST and stock reconciliation starts at 5:15AM BST from Monday to
Friday.
Share_Reportingo RICOH CAPITAL starts at 12AM & SAR starts at 4AM BST.o Runs from Monday to Friday.o Tactical Projects are part of this.o SLAM and SLA_CHM batch runs on Monday.o SLA and SLA_CH batch runs on 2nd of the month. We will run the weekly batch if requested by the
client.
-
7/30/2019 DS Training
14/14
o Likewise PPGM runs on Monday and PPG runs on 1st day of every month.o If there is any there is clash between the dates of monthly and weekly run, then we have to de-
schedule the weekly run and only run the monthly run.
***Note :- FG500 is a sequence which runs once a month before 5 th of every month. Its a part of
DW_EU_PROD