Streams IMplementation Plan

One-Way Schema Replication Using Streams

Abstract One way schema replication setup using Streams Technology.

Document Owner Pushyamitra Musunuru ([email protected])

Document Approver

Version 0.1

Oracle Recommended Implementation plan:

Setup streams pre-requisites as per the below document.

Master Note for Streams Performance Recommendations [ID 335516.1]

STEP 1: CHECK STREAMS UNSUPPORTED OBJECTS PRESENT WITH THE SCHEMA

Listing the Database Objects That Are Not Compatible With Capture Processes

export ORACLE_SID=$SOURCEDB

Spool Streams_Unsupported_Objects.sql COLUMN OWNER HEADING 'Object|Owner' FORMAT A8COLUMN TABLE_NAME HEADING 'Object Name' FORMAT A30COLUMN REASON HEADING 'Reason' FORMAT A30COLUMN AUTO_FILTERED HEADING 'Auto|Filtered?' FORMAT A9

SELECT OWNER, TABLE_NAME, REASON, AUTO_FILTERED FROM DBA_STREAMS_UNSUPPORTED where owner='&SCHEMANAME';

STEP 2: ADD SUPPLEMENT LOGIN TO ALL THE TABLES WHICH ARE PART OF STREAMS REPLICATION

export ORACLE_SID=$SOURCEDB

-- Add the supplement login for all the tables present in schema at the source side

CONN SYS@$SOURCEDB AS SYSDBA

set echo on

show user

set heading offset line 400

select 'alter table SCHEMANAME.' || table_name ||' ADD SUPPLEMENTAL LOG DATA (ALL) columns;' from dba_tables where owner=' SCHEMANAME ';

spool off

-- Check also that supplemental logging is set up for ALL replicated tables (check counts)-- The dba_capture_prepared_x views will show default supplemental logging (PK unconditional, UI/FK conditional)

-- list of tables

select distinct owner,table_name from dba_log_groups;

-- supplemental logging by tablecol object format a40 wrapcol column_name format a30 wrapcol log_group_name format a25 wrapselect owner||'.'||table_name OBJECT, log_group_name, log_group_type,decode(always,'ALWAYS','Unconditional','CONDITIONAL','Conditional',NULL,'Conditional') ALWAYS, generated from dba_log_groups;

-- supplemental logging by table / columncol logging_property heading 'Logging|Property' format a9select owner||'.'||table_name OBJECT, log_group_name, column_name,position,LOGGING_PROPERTY from dba_log_group_columns;

STEP 3: SETTING THE ENV VARIABLES AT SOURCE

The database must run in archive log modeset echo on

CONN SYS AS SYSDBA

SHOW USER

select * from global_name; –to see current global_name

alter system set global_names=true scope=both;

– Restart DB

spool off

STEP 4: SETTING THE ENV VARIABLES AT TARGET

— the database must run in archive log mode

set echo on

CONN SYS AS SYSDBA

SHOW USER

select * from global_name; –to see current global_name

alter system set global_names=false scope=both;

spool off

– Restart DB

STEP 5 : CREATING STREAMS ADMINISTRATOR USER AT SOURCE AND TARGET

CONN SYS AS SYSDBA

CREATE TABLESPACE streams_tbs DATAFILE '$PATH/streams_tbs.dbf'SIZE 25M REUSE AUTOEXTEND ON MAXSIZE UNLIMITED;

create user STRMADMIN identified by STRMADMIN default tablespace streams_tbs quota unlimited on streams_tbs ; grant connect,resource,dba to strmadmin;

BEGIN DBMS_STREAMS_AUTH.GRANT_ADMIN_PRIVILEGE( grantee => 'strmadmin', grant_privileges => true);END;/

spool off

STEP 6: CREATING DB LINK AT THE SOURCE

/* Connected as the Streams Administrator, create the streams queue and the database link that will be used for propagation at Source*/

conn STRMADMIN/STRMADMIN

show user--CREATE DATABASE LINK AT SOURCE as SYS

create public database link $dblinkname using '$TARGET';

---CREATE DATABASE LINK AT SOURCE as STRMADMIN USER

create database link $dblinkname connect to STRMADMIN identified by STRMADMIN using '$TAREGET';

----CREATE DATABASE LINK AT TARGET as SYS

create public database link $dblinkname using '$SOURCE';

---CREATE DATABASE LINK AT SOURCE as STRMADMIN USER

create database link $dblinkname connect to STRMADMIN identified by STRMADMIN using '$SOURCE';

spool off

STEP 7: IF SCHEMA IS NOT PRESENT IN THE TARGET PLEASE CREATE THE SAME.

Use the below script in source to take the schema definition and all roles.

STEP 8: CREATE QUEUE AND QUEUE TABLE AT THE SOURCE AND DESTINATION

/* Connected as the Streams Administrator, create the streams queue and the database link that will be used for propagation at DBSOURCE */


BEGIN DBMS_STREAMS_ADM.SET_UP_QUEUE( queue_table => 'STREAMS_QUEUE_TABLE', queue_name => 'STREAMS_QUEUE_Q', queue_user => 'STRMADMIN');END;/spool offexport ORACLE_SID=$TARGETDB

/* Connected as the Streams Administrator, create the streams queue and the database link that will be used for propagation at lmxena01*/conn STRMADMIN/STRMADMIN

BEGIN DBMS_STREAMS_ADM.SET_UP_QUEUE( queue_table => 'STREAMS_QUEUE_TABLE', queue_name => 'STREAMS_QUEUE_Q', queue_user => 'STRMADMIN');END;/

STEP 9: CREATE PROPAGATION PROCESS AT SOURCE

export ORACLE_SID=$SOURCEDBset echo on

conn STRMADMIN/STRMADMINSHOW USER

BEGINDBMS_STREAMS_ADM.ADD_SCHEMA_PROPAGATION_RULES( schema_name => '$SCHEMANAME', streams_name => 'STREAM_PROPAGATE_P1', source_queue_name => 'STRMADMIN.STREAMS_QUEUE_Q', destination_queue_name => 'STRMADMIN.STREAMS_QUEUE_Q@$DESTDBLINK', include_dml => true, include_ddl => true, source_database => '$SOURCEDB');END; /

STEP 10 : CREATE CAPTURE PROCESS AT SOURCE

conn STRMADMIN/STRMADMINshow userBEGIN DBMS_STREAMS_ADM.ADD_SCHEMA_RULES( schema_name => '$SCHEMANAME', streams_type => 'CAPTURE', streams_name => 'STREAM_CAPTURE_C1', queue_name => 'STRMADMIN.STREAMS_QUEUE_Q', include_dml => true, include_ddl => true, source_database => '$SOURCEDB');END;/

STEP 11: CREATE APPLY PROCESS AT TARGET


BEGIN DBMS_STREAMS_ADM.ADD_SCHEMA_RULES( schema_name => '$SCHEMANAME', streams_type => 'APPLY', streams_name => 'STREAM_APPLY_A1', queue_name => 'STRMADMIN.STREAMS_QUEUE_Q', include_dml => true, include_ddl => true, source_database => '$SOURCEDB');END;/

STEP 12: CREATE NEGATIVE RULE AT SOURCE FOR UNSUPPORTED TABLES

APPLY FOR ALL OBJECTS SELECTED IN STEP 1.


BEGINDBMS_STREAMS_ADM.ADD_TABLE_RULES(table_name => ‘$SCHEMA.OBJECTNAME’,streams_type => 'capture',streams_name => 'STREAM_CAPTURE_C1',queue_name => 'strmadmin.STREAMS_QUEUE_Q',include_dml => true,include_ddl => true,inclusion_rule => false);END;/

STEP 13: STREAMS OBJECT INSTANTATION

Instantiating Objects Using Original Export/Import and Data Pump Export/Import - Example [ID 550955.1]

SQL> SELECT DBMS_FLASHBACK.GET_SYSTEM_CHANGE_NUMBER FROM DUAL;

expdp strmadmin/strmadmin SCHEMAS=$SCHEMANAME DIRECTORY=STREAMS_DP_DIR DUMPFILE=hr_schema_dp.dmp FLASHBACK_SCN=

nohup impdp strmadmin/strmadmin@$DESTDB SCHEMAS=$SCHEMANAME DIRECTORY=STREAMS_DP_DIR DUMPFILE=hr_schema_dp.dmp TABLE_EXISTS_ACTION=REPLACE parallel=4 &

Capture should have been created prior to export. Confirm capture prepared SCNs are prior to flashback SCN.

select * from dba_capture_prepared_schemas;select * from dba_capture_prepared_tables;

Check instantiation SCNs at target -after- import. Instantiation SCNs should match flashback SCN.

select * from dba_apply_instantiated_schemas;select * from dba_apply_instantiated_objects;

STEP 14: ESTABLISH SUBSTITUTE KEYS WITH APPLY

-- Reference: Master Note for Troubleshooting Streams Apply Errors ORA-1403, ORA-26787 or ORA-26786,Conflict Resolution (Doc ID 265201.1)

The following is from the manual - it applies to both 10g and 11g:

Setting Substitute Key Columns for a Table

When an apply process applies changes to a table, substitute key columns can either replace the primary key columns for a table that has a primary key or act as the primary key columns for a table that does not have a primary key. Set the substitute key columns for a table using the SET_KEY_COLUMNS procedure in the DBMS_APPLY_ADM package. This setting applies to all of the apply processes that apply local changes to the database.

For example, to set the substitute key columns for the hr.employees table to the first_name, last_name, and hire_date columns, replacing the employee_id column, run the following procedure:

BEGINDBMS_APPLY_ADM.SET_KEY_COLUMNS(object_name => 'hr.employees',column_list => 'first_name,last_name,hire_date');END;/

Note: You must specify an unconditional supplemental log group at the source database for all of the columns specified as substitute key columns in the column_list or column_table parameter at the destination database. In this example, you would specify an unconditional supplemental log group including the first_name, last_name, and hire_date columns in the hr.employees table.

If an apply process applies changes to a remote non-Oracle database, then it can use different substitute key columns for the same table. You can run the SET_KEY_COLUMNS procedure in the DBMS_APPLY_ADM package to specify substitute key columns for changes that will be applied to a remote non-Oracle database by setting the apply_database_link parameter to a non-NULL value.

Finally, confirm correct designation of substitute key columns:

Select * from dba_apply_key_columns;

If Source site is having primary keys enabled at table level we can set unconditional supplemental logging using the below.To specify an unconditional supplemental log group for PRIMARY KEY column(s):

SQL > ALTER TABLE SCOTT. EMP ADD SUPPLEMENTAL LOG DATA (PRIMARY KEY) COLUMNS;

If the tables are not having primary columns enabled then we need to create separate log groups for each column for the key values defined in capture rule.

To specify an unconditional supplemental log group that includes SELECTED columns:

SQL> ALTER TABLE SCOTT.EMP ADD SUPPLEMENTAL LOG GROUP t1_g1 (C1,C2) ALWAYS;

STEP 15: IMPLEMENT RECOMMENDATIONS TO TUNE THE APPLY PROCESS TO ENSURE MINIMAL APPLY LATENCY DURING REPLICATION

Apply is set up to process in parallel, so definitely review the recommendations for configuring apply with parallelism > 1

Master Note for Streams Performance Recommendations (Doc ID 335516.1)

STEP 16: START THE APPLY PROCESS AT TARGET

connect STRMADMIN/STRMADMIN

- Set stop_on_error to false so apply does not abort for every error; then, start the Apply process on the destination

BEGINDBMS_APPLY_ADM.ALTER_APPLY(apply_name => 'STREAM_APPLY_A1',apply_user => '$SCHEMANAME');END;/

BEGINDBMS_APPLY_ADM.SET_PARAMETER( apply_name => 'STREAM_APPLY_A1', parameter => 'disable_on_error', value =>'n');END;/

begindbms_apply_adm.alter_apply(apply_name => 'STREAM_APPLY_A1',rule_set_name => NULL,remove_rule_set => TRUE);end;/

BEGIN DBMS_APPLY_ADM.SET_PARAMETER ( apply_name => 'STREAM_APPLY_A1', parameter => 'PARALLELISM', value => 4);END;/

BEGINDBMS_APPLY_ADM.SET_PARAMETER( apply_name => 'STREAM_APPLY_A1', parameter => '_HASH_TABLE_SIZE', value =>'1000000');END;/

BEGINDBMS_APPLY_ADM.SET_PARAMETER( apply_name => 'STREAM_APPLY_A1', parameter => '_DYNAMIC_STMTS', value =>'Y');END;/— START APPLY

DECLAREv_started number;BEGINSELECT decode(status, 'ENABLED', 1, 0) INTO v_startedFROM DBA_APPLY WHERE APPLY_NAME = 'STREAM_APPLY_A1';if (v_started = 0) thenDBMS_APPLY_ADM.START_APPLY(apply_name => 'STREAM_APPLY_A1');end if;END;/

STEP 16: START THE CAPTURE PROCESS AT SOURCE

set echo on

connect STRMADMIN/STRMADMIN

show user

BEGIN

DBMS_CAPTURE_ADM.ALTER_CAPTURE(capture_name => 'STREAM_CAPTURE_C1',checkpoint_retention_time => 7);END;/

beginDBMS_CAPTURE_ADM.START_CAPTURE(capture_name =>'STREAM_CAPTURE_C1');end;/

BEGIN DBMS_PROPAGATION_ADM.START_PROPAGATION( propagation_name => 'STREAM_PROPAGATE_P1');END;/

Streams Recommended Documentation:

Master Note for Streams Performance Recommendations [ID 335516.1]

Script to Prevent Excessive Spill of Message From the Streams Buffer Queue To Disk [ID 259609.1]

Master Note for Troubleshooting Streams Apply Errors ORA-1403, ORA-26787 or ORA-26786,Conflict Resolution [ID 265201.1]

Troubleshooting Streams Capture when status is Paused For Flow Control [ID 746247.1]

How To Setup One-Way SCHEMA Level Streams Replication [ID 301431.1]

Troubleshooting Oracle Streams Performance Issues [ID 730036.1]

Troubleshooting Long-Running Transactions in Oracle Streams [ID 783927.1]

Streams IMplementation Plan

Documents

Transcript of Streams IMplementation Plan