Warehouse Replication Using Oracle Data Guard … · Eli Lilly and Company ... Warehouse...

Warehouse Replication Using Oracle Data Guard and Logical Standby

Eli Lilly and Company

• Fortune 500 Pharmaceutical and Bio-Tech Company• Headquartered in Indianapolis, IN• Over $14 Billion in Sales (2005)• 24 Manufacturing Sites World-wide excluding 3rd Party Manufacturers• Key products: Prozac, Zyprexa, Cialis, Insulins

12/11/2006 Copyright© 2006 Eli Lilly and CompanyWarehouse Replication Using Oracle Data Guard and Logical Standby

3

Agenda

• Overview of ISA S95 Standard• Review of Logical Standby• Facts & Figures• Lessons Learned• Conclusions


Overview of ISA S95 Standard


5

Process DeviceContinuous/ Batch Manufacturing

I/O andSensors

Discrete Manufacturing

High AlarmLow AlarmComm. Err.

Business Planning and

Logistics Systems

ManufacturingOperations and

Control Systems

AutomationControl Systems

Process ControlDedicated Systems

Business Process Information Network

Operations Information Network

Automation and Control Network

Discrete Device Process Device

Aseptic AreaLevel 0

BuildingLevel 2

PlantLevel 3

EnterpriseLevel 4

ISA S95 Physical Architecture

Aseptic AreaLevel 1


6

S95 Bottom Line

All systems at a given level (and all levels below it) should be able to operate autonomously and independently of the level above it for a given period of time (as determined by the implementer).


Review of Logical Standby


8

Benefits of Logical Standby

• Seamless replication of enterprise data warehouse (data, structures, accounts) to all sites

• 7 warehouses loaded at once! (near real-time replication of data)

• Low cost solution (no software cost, no ETL development cost, minimal maintenance cost)

• Low risk solution (i.e. high data quality – if it works in the primary, it will work in the standby)


9

Benefits of Logical Standby

• Easy to add additional sites (4-6 hours US, 12-18 hours OUS)

• No performance impact on the primary database during replication

• Allows sites to “extend” enterprise warehouse by adding local schemas and data as necessary to satisfy local needs

• Allows sites to easily suspend replication during local site processing


10

Implications of Logical Standby

• Requires homogeneous computing environment (i.e. chip set, bit size, oracle version)• Impacts if and how hardware, operating system,

and/or database upgrades may occur• Standby databases must be upgraded prior to the

primary (10.1.0.3 and up supports rolling upgrades)• Requires supplemental logging

• Increases the amount of redo generated by ~ 8% (storage issue, not performance)


11

Implications of Logical Standby

• Increases demand for archived redo log file system spacePrimary

• Automated gap resolution is cool, but … logs must remain on-line for gap resolution (possible change to existing archive logmaintenance strategies)

• Query V$ARCHIVED_LOG table to confirm deliveryStandby

• Logs from primary cannot be deleted until applied (automatic in v10.2)

• exec DBMS_LOGSTDBY.PURGE_SESSION• Query DBA_LOGMNR_PURGED_LOG

• SQL Apply transactions roll through the standby’s online redo logs

• 1 primary redo log yields ~ 1.6 archived redo logs on the standby• Improved/fixed in v10.2


12

Why Not Data Streams?

Our prototype comparison of Data Guard and Data Streams proved Data Guard to be the better solution for Eli Lilly• Simple replication of an entire database• Log mining may be done on primary db server

(performance impact) or downstream db server (homogeneous computing environment)

• More complex to set up• More complex to support and maintain• Data Guard proto-type worked “right out of the box”


Facts & Figures


14

Current State – Primary DB

P

• 90 Gigabyte Warehouse• Unix platform running Oracle v10.1.0.4• Located in Indianapolis, IN• Delta loads occur every other hour• Average load generates 100M of redo• Protection Mode: Maximum Performance• Redo Shipment: ARCH


15

Current State – Logical Standbys

s5s3s1

s4

s6

s2P

s7Standby sites:

• Indianapolis, IN (2)• Tippecanoe, IN • Carolina, Puerto Rico• Baskingstoke, England• Fegersheim, France• Kinsale, Ireland

Data Guard setting: Standby


16

Init.ora Parameters – Primary Database

• LOG_ARCHIVE_CONFIG='DG_CONFIG=(P,s1,s2,s3,s4,s5,s6,s7)'• LOG_ARCHIVE_DEST_1=

'LOCATION=/oracle/data/arch01/P/VALID_FOR=(ALL_LOGFILES,ALL_ROLES)DB_UNIQUE_NAME=P'

• LOG_ARCHIVE_DEST_2='SERVICE=s1VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE)DB_UNIQUE_NAME=s1‘

• LOG_ARCHIVE_DEST_STATE_1=ENABLE• …• LOG_ARCHIVE_DEST_STATE_8=ENABLE• LOG_ARCHIVE_MAX_PROCESSES=10


17

Init.ora Parameters – Logical Standby

• DB_NAME=s1• DB_UNIQUE_NAME=s1• SERVICE_NAMES=s1• LOG_ARCHIVE_CONFIG='DG_CONFIG=(P,s1)'• LOG_ARCHIVE_DEST_1=

'LOCATION=/oracle/data/arch01/s1/VALID_FOR=(ONLINE_LOGFILES,ALL_ROLES)DB_UNIQUE_NAME=s1'

• LOG_ARCHIVE_DEST_2='LOCATION=/oracle/data/arch01/s1_dest_2/VALID_FOR=(STANDBY_LOGFILES,STANDBY_ROLE)DB_UNIQUE_NAME=s1'


18

Init.ora Parameters – Logical Standby

• LOG_ARCHIVE_DEST_STATE_1=ENABLE• LOG_ARCHIVE_DEST_STATE_2=ENABLE• PARALLEL_MAX_SERVERS=9• REMOTE_LOGIN_PASSWORDFILE=EXCLUSIVE• INSTANCE_NAME=s1


19

Future State *

Ps2

s3 s5 s14

s13

s12

s1

s4s8

s7s10

s11s6s9

* proposed

• Up to 14 sites (8 OUS)• Cascaded standbys• Schema-based data filtering• Use of standby redo logs (LGWR ASYNC)


20

Cascaded Standby

• Having one logical standby serve as the primary database to another logical standby

• Capability documented in Oracle Data Guard manual beginning with version 9.2

• Provides the following significant benefits:• Overcomes 9 standby limit• Redo log shipped to zone 2 once regardless of the number

of zone 2 logical standbys• Unfortunately, a bug exists preventing the startup of the

SQL apply process at the cascaded (end point) site• Bug has been filed with Oracle Support and eagerly!

awaiting resolution


Lessons Learned


22

Lessons Learned

• Speed at which all standby databases receive redo is limited to the speed of the slowest network path to any one standby• Use standby redo logs• For high speed networks, use ASYNC

• Standard maintenance processes may have to be changed• Processes to compress archived redo• Process to delete archived redo


23

Lessons Learned

• Be aware of large loads and help Data Guard out when necessary• DEFER slower sites via LOG_ARCHIVE_DEST_STATE_#

(active sites will receive redo as normal)• Execute large load• Tar, compress, and manually transfer archived redo• Register archived redo in standby• When all log registered at slower site, re-ENABLE the site

• Set redo shipment for inaccessible standby sites to DEFER


24

Lessons Learned

• If redo shipment can’t occur, archive processes die (Bad! Very bad!) – bug being worked with Oracle

• Make sure your standbys are regularly applying redo

• Status and notification messages are recorded as “errors” in alert logs

• Password for SYS must be identical for all databases in the standby configuration (primary or standby) for standby to receive redo


25

Lessons Learned

• Backup/recovery of logical standby is not intuitive – awaiting official documentation from Oracle

• The Oracle Data Guard product team has been totally committed to our success as an early adopter of Data Guard technology


26

Conclusion

• Data Guard with Logical Standby database has proven to be extremely successful in allowing Eli Lilly to satisfy site reporting needs much quicker without incurring the costs typically associated with standard ETL delivery methods

Warehouse Replication Using Oracle Data Guard … · Eli Lilly and Company ... Warehouse...

Documents

Transcript of Warehouse Replication Using Oracle Data Guard … · Eli Lilly and Company ... Warehouse...