Logrep White

7/31/2019 Logrep White

1/18

Oracle LogMiner Based Replication(A detailed discussion on how to use the LogMiner utility to replicate data)


2/18

________________________________________________Table of ContentsTable of Contents

LogMiner Overview ..............................................................................................................3

SCN ..................................................................................................................................3Oracles Redo Log........................................................................................................... 4LogMiner API................................................................................................................... 6

Add Logfile .................................................................................................................6Start Logmnr ................................................................................................................6LogMiner View .............................................................................................................7

Using LogMiner for replication:............................................................................................ 8Extraction Program ..........................................................................................................8

The Dynamic table X$KTUXE .....................................................................................8Calling LogMiner to Extract Data .................................................................................9Cleaning Up ...............................................................................................................11Mapping of Object Identifiers to known names .........................................................12

Notes on Restoring via an import.............................................................................. 14Extraction Trick Number 1 .........................................................................................15

Why you should drop logs ....................................................................................15How to drop unused logs .......................................................................................15

Extraction Trick Number 2 .........................................................................................16Keep all activity out of the database ......................................................................16Extracted Data .......................................................................................................16

Export Process ...............................................................................................................16Import Process ...............................................................................................................16Posting Process .............................................................................................................16

Appendix A - LogMiner Replication Flow ..........................................................................17Appendix B - Definitions ....................................................................................................18

Redo Logs ......................................................................................................................18SCN ...............................................................................................................................18


3/18

______________________________________________________________________

LogMiner OverviewLogMiner Overview

LogMiner is a utility provided by Oracle Corporation to read the contents ofthe Oracle redo logs in order to recreate SQL that was applied on the system.LogMiner is a set of API programs that allow a user to query the contents of theonline redo logs. The redo logs are presented as a table in the database, and auser has the ability to issue standard select statements to retrieve the contentsof the redo log.

SCN

All changes made to all objects in the database are kept track via the SCN. The SCN (orsystem change number) is a sequential number generator that stamps all physicalobjects (control files, data files, and redo logs) with this number. This is the mechanismOracle has chosen to enforce database integrity. Not only are the physical objectsstamped with the SCN, but transactions that are recorded in the redo log are as well.The other function of the SCN is for backup and recovery, which is outside the scope ofthis document.

DatabaseInstance Data file

Data file

Data File

The 3 physical components of an oracle database are:: redo logs control files data files

The SCN (System control number) is a counter thatis applied to all the physical components. It's mainpurpose is to enforce the integrity of the database

10001

10001

10001

10001

10001

10001

In this example, all physical components are stamped with the SCN value 10001.


4/18

Oracles Redo Log.

The function of the Oracle online redo log is to record ALL changes to the database.This includes data, indexes, the Oracle catalog, rollback segments, etc.. Any and allchanges to any data block in the database are recorded in the redo log. The redo logs

are setup as a circular queue, so that as one queue (log) fills up, it will switch to anotherlog. Once the final log is filled, it starts to write to the first log.

Each recording has an associated SCN value. The SCN is basically a sequential numberthat is stamped with the transaction, in order to keep the database consistent.

Database Instance

Log 6

Log 4

Log 5

Log 1

Log 8

Log 7

Log 2

Log 3

LGWR

log buffer

Overview of the

redo logs

SCN 10000

SCN 15000

1

2

3

(1) As changes to the database occur, they are recorded in the redo log

buffers The log-writer process (LGWR) writes to the redo log (2)

As a log fill up , it moves to the next log (3)

Each log (4) has a starting SCN value that can be queried from the oracle

catalog

4

In the figure above, redo log 2 ends before SCN 9999, redo log 3 starts at 10,000 and ends at 15000.


5/18

The Oracle database catalog records the starting SCN value for each redo log. Thesevalues are easily retrieved with the following query:(NOTE: The SCN value is specified in the column FIRST_CHANGE#)

select group#, sequence#, status, first_change#, first_time from

v$log;

The system will return rows similar to the results below. In this example, there are a total of nine(9)rows returned. In the results returned below, group #5 is the current redo log being written to, andthe value 48138is the first SCN value associated with the redo log.

GROUP# SEQUENCE# STATUS FIRST_CHANGE# FIRST_TIME---------- ---------- ---------------- ------------- -------------------

1 1 INACTIVE 1 2004-10-29:11:36:412 2 INACTIVE 21236 2004-10-29:11:40:223 3 INACTIVE 30786 2004-10-29:11:43:334 4 ACTIVE 39148 2004-10-29:11:50:115 5 CURRENT 48138 2004-10-29:11:54:10

6 0 UNUSED 0

7 0 UNUSED 08 0 UNUSED 09 0 UNUSED 0

In order to query the catalog to see the actual name of the file being written to, issue the followingquery from sql*plus.

select * from v$logfile

returns the rows

GROUP# STATUS TYPE MEMBER---------- ------- ------- ----------------------------------------

1 ONLINE /u01/oradata/DB01/redo01a.log1 ONLINE /u02/oradata/DB01/redo01b.log2 ONLINE /u02/oradata/DB01/redo02a.log2 ONLINE /u03/oradata/DB01/redo02b.log3 ONLINE /u03/oradata/DB01/redo03a.log3 ONLINE /u01/oradata/DB01/redo03b.log4 ONLINE /u01/oradata/DB01/redo04a.log4 ONLINE /u02/oradata/DB01/redo04b.log5 ONLINE /u02/oradata/DB01/redo05a.log5 ONLINE /u03/oradata/DB01/redo05b.log6 ONLINE /u03/oradata/DB01/redo06a.log6 ONLINE /u01/oradata/DB01/redo06b.log7 ONLINE /u01/oradata/DB01/redo07a.log7 ONLINE /u02/oradata/DB01/redo07b.log8 ONLINE /u02/oradata/DB01/redo08a.log

8 ONLINE /u03/oradata/DB01/redo08b.log9 ONLINE /u03/oradata/DB01/redo09a.log9 ONLINE /u01/oradata/DB01/redo09b.log

NOTE: Oracle needs to write to at least two logs at the same time. In the example above the two filesbelonging to group 5 are mirrored.

For more information on redo logs, please go to:http://download-west.Oracle.com/docs/cd/B10501_01/server.920/a96521/onlineredo.htm - 3848
http://download-west.oracle.com/docs/cd/B10501_01/server.920/a96521/onlineredo.htm#3848http://download-west.oracle.com/docs/cd/B10501_01/server.920/a96521/onlineredo.htm#3848


6/18


7/18

LogMiner View

After issuing the START_LOGMNR API, the utility creates a view into the redologs: This view (or logical table) allows standard SQL statements to be used to query the

data.

SELECT a.scn, a.cscn, to_char(sysdate,'DD-MON-YYYY HH24:MI:SS'),substr(a.sql_redo,1, length(sql_redo)-1)

FROM v$logmnr_contents aWHERE a.operation in ('INSERT', 'UPDATE','DELETE','COMMIT')AND (a.username != 'SYS' OR a.username is NULL)AND ( a.seg_owner = 'seg_owner' or a.seg_owner is null)AND a.serial# not in (select serial# from post_stats)AND cscn > :hv_WorkingSCNAND cscn :hv_WorkingSCNAND cscn


8/18

______________________________________________________________________

Using LogMiner for replication:Using LogMiner for replication:

Extraction Program

The first part of the replication process involves wrapping around theabove LogMiner APIs within a Pro*C++ program. The program continuouslymonitors the database for activity, and then calls LogMiner to scan the redo logsfor committed transactions.

The Dynamic table X$KTUXE

The first thing the replication program needs to know if there has been any type ofactivity in the system. Whenever there is activity, Oracle stores the system wide SCNvalue in a dynamic table (non-physical) called X$KTUXE. The extract program storesthis information in a variable, and compares values from the last time it ran. If there is adifference, then some kind of database activity has occurred.

sys.x$ktuxe;

lastSYSSCN

The variable lastSYSSCN stores the SCN value of when the last databaseactivity took place

If any activity occurs, the value in the X$KTUXE table increases and isstored in a variaible

If the two values are different, then some database activity occured, whichneeds to be investigated.


9/18

The process is as follows:

Step 1. select max(ktuxescnw * power(2, 32) + ktuxescnb) from sys.x$ktuxe;

Step 2. Save value of Step 1 into variable.Step 3. Copy variable from step 2 into a variable labeled old_variable_name.

Step 4. Re-issue above query (after sleeping for n seconds)Step 5. Save query into variable.Step 6. Compare old value to new.Step 7. If different, then call LogMiner (something has changed), else sleep n seconds

and repeat.

Calling LogMiner to Extract Data

The extract program (lg_ext) senses that some sort of database activity has occurred bycomparing values from the current x$ktuxe table and the last time it ran.

Before calling LogMiner, the extract program checks the value of the current system

wide SCN value (from x$ktuxe), against the Oracle catalog to see if this new informationmight be on another (new) redo log (from v$log). If so, it will call the LogMiner attach

API to attach the new redo log, and then issue the SELECT statement against theLogMiner table.

SYSSCN(from x$ktuxe)

I need all the way to 65,000

Log 5Log 6

48138

55123

64000

In this example, some activity is now occurring on redo log 6, which we will need to attach to theLogMiner utility.


10/18

After all logs have been attached, the extract program calls LogMiner to start scanningfor data, starting from the last SCN we wrote out, to the current system wide SCN.

SYSSCN(from x$ktuxe)

LastWorkingSCN

Starting value of anycurrently runningtransaction (fromv$transaction)

Range to Scan

LastWorkingSCN - or thelast time we printed outany committed SQL

Possible range to scan ifno active transactions.

Some things to note:

The name of the LogMiner table is called : V$LOGMNR_CONTENTS

When issuing a query against this table, its possible to filter data via thepredicate. In terms of replication, we are only interested in objects that are ownedby the schema owner


11/18

The call to LogMiner is as follows:

Step 1. Call LogMiner with the SCN range of the last time LogMiner extractedinformation, and the current system wide SCN value.

Step 2. If LogMiner returns rows, print those rows to a queue file, else do nothing andagain wait for system activity.

Step 3. If rows were printed, record the last SCN value that was printed to a variable inthe program and to disk (every committed transaction will have an SCNassociated with it)

Step 4. The lg_exp program knows how far into the file it has processed. If the file hasgrown, then continue to ship the SQL to the target system.

Cleaning Up

After rows have been printed, the program then compares the last SCN value itprocessed versus the SCN values for the online redo logs. If the redo logs are no longerneeded, the program calls the LogMiner API to detach the redo logs from the session.

This is done to minimize the amount of I/O LogMiner will do when extracting data, asLogMiner will use attached redo logs to scan for data.

The last work I did was at64,300. I no longer need log

5

Log 5Log 6

48138

55123

64000


12/18

Mapping of Object Identifiers to known names

In the internal workings of Oracle, Oracle uses identifiers (numbers) to work with objects.Whenever an object is created, its assigned a number (sequentially assigned) that will

be used internally by the database. In order to have readable SQL extracted from thedatabase, LogMiner needs to map the object ID to the name of the object.

The mapping is done when LogMiner is first started.

To following illustrates this concept

Create a tablecreate table foobar (foo number(1));

Run This queryselect object_id, object_name, object_type from sys.dba_objects

where object_name = FOOBAR'

Returns the resultsOBJECT_ID OBJECT_NAME OBJECT_TYPE---------- ------------------------------ ------------------ 6658 FOOBAR TABLE

Drop and re-create the tabledrop table foobar;create table foobar (foo number(1));

Re-Run this queryselect object_id, object_name, object_type from sys.dba_objectswhere object_name = FOOBAR'

Note the change in the object IDOBJECT_ID OBJECT_NAME OBJECT_TYPE---------- ------------------------------ ------------------ 6659 FOOBAR TABLE


13/18

When an object is dropped and recreated, it is assigned a

new object ID.

On startup, LogMiner maps the object ID to the known

name via the oracle catalog

TB_TABLENAME=ID12345

Object ID 12345 = DS_ASX_INFO

TB_TABLENAME=ID 43210

Drop and Recreate of object

(via restore)

In this example, LogMiner isnt stopped. A restore is done, which drops and recreates the objects(with a new object_ID). LogMiner only knows that object_ID 12345 = TB_TABLENAME, therefore anychanges after the restore are lost.


14/18

Notes on Restoring via an import

The LogMiner replication program remembers the last SCN of when it did any work. Thisvalue is stored in a file, so when the program starts, it knows where in the redo log to start

scanning. Stopping and starting isnt enough. If you should restore from an export and dontre-initialize (re-install) the replication program, all data imported via the restore will beextracted and replicated.

TB_TABLENAME=ID12345

Object ID 43210 = TB_TABLENAMEand the last time I did work was at 55123

TB_TABLENAME=ID 43210

Drop and Recreate of object(via restore)

All changes in this rangeare replicated

SCN: 56000 SCN 58000

In this example, the last time replication did any work was at SCN 55123. On startup, any changesfrom 55123 to 56000 will be ignored (since the object_id in the redo log dosent match any object inthe current catalog), and all changes from 56,000 will be replicated.

If the export file is also used on the target system, in this scenario, data will be applied twice to thetarget: Once from the export and again from replicated data originating from the primary.


15/18

Extraction Trick Number 1

With LogMiner, its possible to sustain heavy I/O if you have too many redo logsattached. If possible, only use the current active redo log for processing. When no longer

needed, drop the redo log.

Why you should drop logs

When calling the START API, it appears that the LogMiner utility will use all redo logsthat is currently attached, which creates more I/O and longer scan times.

How to drop unused logs

Query the v$log catalog table, which tells you the first SCN value of each redo log.

SELECT a.first_change#, a.group#, b.member

FROM SYS.V_$LOG a, SYS.V_$LOGFILE bWHERE a.first_change# >=

(

SELECT nvl(max(first_change#),0)

FROM sys.v_$log

WHERE first_change# < (THE LAST SCN YOU PROCESSED))

AND b.group# = a.group#;


16/18

Extraction Trick Number 2

Keep all activity out of the database

Extracted Data

In the examples that Oracle provides for LogMiner, when extracting SQL from theredo logs, they usually stage them into another table. The problem with this, is

a. You have to make sure that the table does not generate redo information.b. Even if you create the table with no logging, inserts into any table does

increment the SCN.

Therefore, you should always extract data to a flat file.

Export Process

The export process runs on all replicated nodes and can start independentlyof all other replication processes.

The extractor process, when extracting, writes data to a queue file. When thequeue has grown to a certain size, it closes the file and start writing to a different logfile. The export process continuously looks at the size of the queue file. If changed, itreads the contents of the file in the data directory, and via a TCP socket connection,transfers the contents over.

After receiving a confirmation that the information has been received, itincrements its recorded offset into the queue.

Import Process

The import process is a simple TCP Socket server that accepts connectionsvia a specified port. When a connection is made, it waits for data to be shipped over.When it receives data, it writes the content to a queue file on the target system.

The import process can start independently from all other replication processes.

Posting Process

The posting process is a simple C++ program that scans the queue file (files that

has been modified by the import process). The process behaves similarly to the exportprocess, waiting for the data queue file to grow.Once its been determined that the queue has changed, the poster prepares (fordynamic SQL you must prepare an SQL statement) the SQL and executes the DML.

The post process can start independently from all other replication processes.

____________________________________________________________________


17/18

Appendix A - LogMiner Replication FlowAppendix A - LogMiner Replication Flow

lg_ext(extractor)

lg_exp(exporter)

lg_imp(importer)

lg_post(poster)

Target DBSource DB

Temp

Queue

Temp

Queue

IP sockets

Data

Queue

Data

Queue


18/18

_____________________________________________________________________________________

Appendix B - DefinitionsAppendix B - Definitions

Redo LogsEvery Oracle database has a set of two or more redo log files. The set of redo log filesis collectively known as the redo log for the database. A redo log is made up of redoentries (also called redo records).The primary function of the redo log is to record all changes made to data. If a failureprevents modified data from being permanently written to the datafiles, then the changescan be obtained from the redo log, so work is never lost.To protect against a failure involving the redo log itself, Oracle allows a multiplexedredo log so that two or more copies of the redo log can be maintained on different disks.The information in a redo log file is used only to recover the database from a system ormedia failure that prevents database data from being written to the datafiles. Forexample, if an unexpected power outage terminates database operation, then data inmemory cannot be written to the datafiles, and the data is lost. However, lost data canbe recovered when the database is opened, after power is restored. By applying theinformation in the most recent redo log files to the database datafiles, Oracle restoresthe database to the time at which the power failure occurred.

SCN

The system SCN (System Change Number) is a system wide sequential counter that isused to synchronize all physical components of the Oracle Database. Each physicalcomponent (Datafile Headers, Control Files, Redo Logs, Redo Log entries) record theSCN. By making sure all physical components have the same SCN, ensures the integrityof the physical structure of the database, in which insures the integrity of the data.

For example, the database compares the SCN of the control files to the SCN of thedatafiles. If the SCNs arent the same, then the database assumes that the datafileswere restored from a previous backup and prompts that the database needs furtherrecovery.

Oracle 9i Database Concepts Chapter 1

Logrep White

Documents

Transcript of Logrep White