Running Oracle’s Global Single Instanceon E-Business Suite R12

Donna Buhlig Nigel CooperDirector, AIT Program Mgmt Director, AIT Technical Svcs

The following is intended to outline our generalproduct direction. It is intended for informationpurposes only, and may not be incorporated into anycontract. It is not a commitment to deliver anymaterial, code, or functionality, and should not berelied upon in making purchasing decisions.The development, release, and timing of anyfeatures or functionality described for Oracle’sproducts remains at the sole discretion of Oracle.

<Insert Picture Here>

Agenda

• Running Oracle’s Global Single Instanceon E-Business Suite Release 12

• Technical Overview of Oracle’s GSI

• Upgrading to R12

• Q&A

<Insert Picture Here>

Oracle Applications IT (AIT)

• AIT supports all of Oracle’s lines of business andevery Oracle employee

• AIT is responsible for evaluating, implementing, andsupporting application solutions that allow Oracle torun its business

• Early adoption of Oracle products….Developmentpartner

Applications Supported by AITE-Business Suite Products Implemented

FinancialsGeneral Ledger

Fixed Assets

Treasury

Cash Management

Subledger Accounting

iAssets

E Business Tax

Response Capture

Manufacturing & Distribution,Advanced Product Catalog andPricing

Materials Requirement Planning

Bill of Materials

Inventory Management

Warehouse Management System

Work in Progress

Engineering

Shipping Delivery Based

Advanced Pricing

External Order Capturing SystemsSupply Chain Trading Connector to

RosettaNet

XML Gateway

iStore

ProcurementiProcurement

Purchasing

Accounts Payable

Internet Expenses

Payments

Property Manager

Product Lifecycle ManagementAdvanced Product Catalog

Human Resources & PayrollAdvanced Benefits

Human Resources

Human Resources Intelligence

Payroll

Time and Labor

Global Payroll Interface

Appraisals

Talent Management

Compensation Workbench

iRecruitment

Self Service Human Resources

Learning Management

Internal Order Capturing SystemsIncentive Compensation

Quoting

Order Management

Contracts Core

ProjectsProjects

Project Resource Management

Projects Intelligence

Support Sales SystemsService Contracts

Contracts Intelligence

Install Base

Internal Support SystemsTeleservice

Knowledge Management

iSupport

Customer Support

DBI for Customer Support

TechnologyAlert

Application Object Library

e-Commerce Gateway

Report Manager

Self Service Web Applications

System Administration

Web Applications Desktop Integrator

XML Gateway

XML Publisher

Workflow Cartridge

User Management

CRM Foundation Applications

Trading Community

Oracle Customers Online

Internal Controls Manager

Trade Management

Applications Supported by AITE-Business Suite Products Implemented

SalesCampaign Member / Lead Management

Opportunity Management

Sales Forecasting

Quote (for integration)

Project Management (for References)

Territory Management (Skills)

Call CenterAccount Management

Contact Management

Activity Management

Calendar Management

Product & Catalog Management

Position Management

MarketingMarketing Resource Management (MRM)

Campaign Management

List Import

Events Management

PRMPartner Portal (eChannel)

Partner Manager

Partner Programs

Customer Data IntegrationUniversal Customer Master (UCM)

Siebel Data Quality

MarketPlace (D&B, 3rd party data)

Customer Sync

Siebel ServerAssignment Manager

EIM

EAI

Workflow Policy Manager

Testing Automation (using Mercury QTP)

OBIEE Applications – CRM AnalyticsSales Analytics

Partner Analytics

Marketing Analytics

List & Segmentation

Territory Analytics (custom)

Usage Accelerator

Applications Supported by AITSiebel Products Implemented

Oracle Sales & Marketing AnalyticsDQM Metrics & Measurements

Territory Analysis

Territory Validation

Marketing Budget Reconciliation & Analysis

List Building & Segmentation Analysis

Campaign Member Lead Analysis

Campaign to Deal Analysis

Campaign & Event Analysis

Pipeline Analysis & Trending

Opportunity & Forecast Reporting

Partner Marketing Analysis

Channel Manager Dashboard

Partner Sales Pipeline

Partner Segmentation

Oracle Financial AnalyticsReceivables

General Ledger

Profitability

Applications Supported by AITBI Products Implemented

Global Single InstanceCurrent Data Statistics

14.6 terabytes 703,000 customers

32.3 billion rows of data 421,000 vendors

104 operating units 326,000 projects

675 Ledgers 47.6 million sales order lines

10 languages 68 million AR invoice lines

20 years of SLA data (8 yearsconverted during R12 upgrade)

736 million GL lines

Global Single InstanceSystems Metrics

Database6,604 database sessions

13,833 peak

Forms Users32,000 named users

1,100 concurrent users / 5,995 peak

Self-Service Users118,000 named users

2,700 concurrent users / 10,000 peak

Discoverer3,500 named users

50 concurrent users / 315 peak

Concurrent Manager187 concurrent jobs / 2,800 jobs complete per hour

500,000 jobs completed May 2008 close week

GSI Production Architecture

Application Server Configuration

Mid-tier function

DMZ External apps 8

Self-service apps 18

Forms Server 8

Concurrent Manager 4 + 1 standby

Mid-tier server

Server / operating system Dell 1950 OEL 4U4

Memory 16 GB

CPU speed / # 2 x 2.33 GHz (dual-core)

# of servers 39

search.oracle.com GSI Linux mid tier

Database Server ConfigurationDatabase Servers

• Sun F25Ks – 44 Dual Core CPUs

• 176GB Memory each

OS / Clusterware

• Solaris 9

• Sun Cluster 3.1

• Veritas: VxVM/VxFS 4.0* mp02

Database / SGA

• RDBMS 10.2.0.3

• 34GB SGA Target

• 20GB PGA Aggregate Target

Cluster Interconnects

• 3 GigE cards per nodeDB Storage

•EMC DMX3000

•Groups of 8x146GB drives

(software striping)

• 1MB stripe width

• 8 way HW striping

Managing PerformanceAutomated Features

• Ad-hoc sessions• Database Resource Profiles

• Forms, self service, reporting sessions• Automated Kills terminate:

• Long running queries

• Lengthy idle sessions with uncommittedtransactions

• High temp space consumers

• Promotes the logging of performance bugs

• Concurrent Manager• Active Request Limits

Managing PerformanceConcurrent Manager Workload

• Goal: Maximize overallthroughput

• Strategy: Fast runningjobs don’t wait for slow jobs

Managing PerformanceConcurrent Program Request Types

Overnight

Run times

> 10 hours

Immediate

Run times

less than 5

minutes

Priority

Run times

between 5

and 10

minutes

Standard

Run times

between 10

minutes and

10 hours

Critical

Business

Critical

See Metalink note 285267.1 for more information

System Monitoring

• Enterprise Manager 10g (EM 10.2.0.4):• Replaced custom alerting and reporting solutions

• Monitoring and Alerts (standard metrics & UDMs)

• Smart Monitors (fix it jobs)

• Metrics collection for capacity planning

• Automated Service Level Management (ASLM)

• Oracle Applications Manager (OAM)

• EM eBiz plug-in (2.1) testing in progress

• Ad hoc monitoring:• EM, AWR, sar, vmstat, iostat, mrtg

System MonitoringConcurrent Manager User Defined Monitors

• Conflict Resolution Manager blocked byenqueues

• Concurrent Manager queue backlogs

• Queue process level changes

• Long running jobs

• Controlled by threshold table

Managing System Change

• Three key considerations:• Changes are made in a measured fashion

• Changes are tested for functionality and performance impactbefore deployment

• Changes are recorded – audit trail

• Defined and publicized patching windows

• Use “Quarterly Release Bundles” to minimize outages

Managing System ChangeEnvironment Progression

TechnicalTest

(RAC)

The Technical Testenvironment is a one-off testenvironment and has nodesignated change flow toother test environments. It isused for initial upgrade testingand performance testing.

Reference

(Non-RAC)

A copy of production is createdjust before go-live as areference for issues which mayarise.

Build Test(RAC)

UAT

(RAC)

Staging

(RAC)

Quarterly Release BundleChange Path

Path used for small-to-mediumproject changes.

DRStandby

(RAC)Production

(RAC)

BuildTest(RAC)

DRStandby

(RAC)

UAT

RACBuild Test

(RAC)

UAT

(RAC)

Large Project Change Path

Production

(RAC)

Staging

(RAC)

VerificationTest

(Non-RAC)

Upgrade changes will beintroduced into normalproduction support path viarefresh after project go-live

Managing System ChangePerformance Testing

• New functionality (eBiz module) rollout• Design testing specific to the deployment

• Major RDBMS / eBiz releases

• Load Runner stress tests

• Batch load tests

• User participation tests

• Applications patches• WinRunner to regression test key flows

Test and Tune before deployment on production!

<Insert Picture Here>

Agenda

• Running Oracle’s Global Single Instanceon E-Business Suite Release 12

• Technical Overview of Oracle’s GSI

• Upgrading to R12

• Q&A

Scoping and Planning the ProjectProject Teams

Oracle R12Development

AIT ApplicationTeams

Oracle BusinessTeams

AIT SteeringCommittee

AIT Technical Teams

R12 Upgrade Team

Apps (mid-tier) Services

DBA Services

Performance Services

AIT Project Mgmt

Scoping and Planning the Project

• Assist R12 Development with testing by using GSI copies

• Upgrade database and mid-tier OS (pre-req)

• Refresh mid-tier server technology to new hardware standard

• Implement R12 upgrade in smaller volume production environmentprior to GSI go-live

• Utilize “Upgrade by Request” to upgrade subset of SLA data• 8 out of 20 years of SLA data migrated, no specific plans to migrate remaining 12

years of data

• reduced downtime and provided first stage of future project to archive transactionaldata

• Clean-up APPLTOP of obsolete customizations

• Convert custom reports to XML Publisher

System Architecture Projects

R12 Application Upgrade

Scoping and Planning the ProjectKey AIT Project Phases

Dev Testing

• xBuild 7, 9, 13, 19

10.2.0.3 database upgrade

Technical Testing

• Upgrade andperformancetesting

Build Test

• AITmoduletesting

UAT

• End-to-endtesting

• Load testing

ProductionLive

Mid-tier OS upgrade & server refresh

Technical Testing

• Configuration andperformance testing

Build Test

• AIT module testing

UAT

• AIT regressiontesting

ProductionLive

Build Test

• Module testing

• Extension / customdev

UAT

• AIT – 12.0, RUP1; Business – RUP2, RUP3

• Performance testing

Technical Upgrade Timing TestsProduction

Live

Preparing the System ArchitectureDatabase Tier Pre-Requisites

• RDBMS upgrade project• Role includes contributing to e-Business Suite certifications

• Goal to keep as current as possible

• Upgrade production every 12 – 18 months

• 10.2.0.3 available on Solaris - November 2006

• GSI live - March 2007

• Storage project• Pre-upgrade GSI size (11.5.10): 12.2 TB

• Upgrade requirements: 1.3 TB

Preparing the System ArchitectureMid-tier OS Upgrade and Server Refresh

Mid-tier function # Before # After

DMZ External apps 14 8

Self-service apps 25 18

Forms Server 15 8

Concurrent Manager 8 + 2 standby 4 + 1 standby

Mid-tier server Before After

Server / operating system Dell 2650 Linux AS 2.1 Dell 1950 OEL 4U4

Memory 6 GB 16 GB

CPU speed / # 2 x 2.80 GHz (hyper-thread enabled)

2 x 2.33 GHz (dual-core)

# of servers 64 39

Verifying PerformanceR12 Performance Testing

• All significant architecture and technicalchanges undergo performanceevaluation

• Performance compared:Database upgrade

Mid-tier OS upgrade to OEL 4.0

Mid-tier server capacity analysis

XML Publisher report throughput

R12 upgrade timings

R12 run-time analyses

Planning the Outage TransitionAdvance Steps

• Weeks+ ahead:• Complete required architecture projects

• Build the “Master” APPLTOP using most recent test upgrade

• Refine the transition plan

• Up to a week ahead:• Apply pre-upgrade application patches

• Complete application preparation steps, cancel obsoletescheduled requests

• Technical teams’ tasks:

• Setup dedicated listener and automated monitoring

• Pre-allocate storage

• Prepare R12 APPLTOP (copy Master) and code tree

• Analyze all objects with stale CBO statistics

• Hours ahead - sync the BCV backup disks

Planning the Outage TransitionPre-Upgrade Steps

• Transition pre-upgrade tasks• Turn off archive log

• Disable flashback DB

• Use “upgrade” listener and shutdown production listener

• Disable runtime production monitoring (EM monitors, kill scriptsfor “runaway” processes and long running sessions)

• Modify init.ora parameters

Planning the Outage TransitionPre-Upgrade Steps (continued)

Init.ora parameter changes for R12 upgrade value

parallel_max_servers 80

log_buffer 32010240

job_queue_processes 40

db_block_checking False

Recyclebin Off

_db_handles_cached (for ar120trxl.sql) 16

log_archive_max_processes 8

pga_aggregate_target 16 gbytes

_db_file_optimiser_read_count REMOVE

db_file_multiblock_read_count REMOVE

Planning the Outage TransitionPre-Upgrade Steps (continued)

• Transition pre-upgrade tasks• Put scheduled concurrent jobs on hold

• Run any datafixes required based on test runs

• Add “extension plsql_no compile yes” line in upgrade driverfile to enable PL/SQL no compile option

• Create separate concurrent manager queue to processconcurrent programs submitted by upgrade (See Metalinknote 399362.1 for more information)

Planning the Outage TransitionUpgrade Steps

• Number of workers – 60 workers across 4 mid tiers

• adpatch options used on main mid tieroptions=nocompilejsp,noautoconfig,nocopyportion,nogenerateportion

batchsize=10000, workers=60, localworkers=15

adctrl distributed =y on other 3 mid tiers

• Monitor space in new R12 tablespaces, session waitsand alert log

Planning the Outage TransitionPost-Upgrade Steps

• Run hrglobal before NLS (See Metalink notes Note:145837.1 &414434.1 for more information)

• Merge standard patches to save time

• Reset init.ora parameters to original pre-upgrade values except fordb_file_multiblock_read_count and _db_file_optimizer_read_count

• DBA steps (start up 4 nodes,FNDCPASS, enable flashback) beforetesting

• Apply merged NLS D driver in parallel with US language testing

• Migrate CUSTOM library (custom.pll)

• Perform functional post-upgrade steps/verification

• Take concurrent scheduled jobs off-hold and ensure productionmonitoring is re-enabled.

Planning the Outage TransitionOutage Steps Summary

Place pending concurrent requests on hold,cold backup (split BCV),

startup one database instance

Run the R12 upgradeand apply RUP

Perform R12 setups onthe intranet mid-tiers,

reregister SSO

Apply standard patches,run hrglobal

Apply custom patches

Planning the Outage TransitionOutage Steps Summary (continued)

Start concurrent managers

Verify each application module

Restart normal monitoring and operations,remove held scheduled requests,

release system

Analyze stale objects andprepare concurrent managers

Complete mid-tierconfiguration

Backup (split BCV) andstartup all database instances

Apply merged NLS patchesPerform application specific

post-upgrade steps

Issues and LessonsIdentified During Testing

• FNDLOAD• Needed methodology to allow some concurrent programs with

customer extensions to be overwritten by new R12 programdefinitions.

• Customer extensions to seeded program definitions then overwrittenby these new program definitions (incompatibilities, request typesand program priorities)

• “Create Accounting” and “Submit Accounting” design changesimpacted concurrent manager throughput

• Using “stale” analyze method for large objects:• New columns on large tables which don’t initially get populated by

the upgrade, the tables are so large the stats are not consideredstale, so one-off analyzes are required

• Some new seeded histograms not analyzed

• Use testing phases to identify additional CBO stats gatheringrequirements

Issues and LessonsActions to Reduce Outage Downtime

• Separate database upgrade

• Separate mid-tier OS upgrade

• Complete Functional tasks listed in Reducing downtime section(Appendix E) in R12 Upgrade manual

• Make use of “Upgrade by Request” option (Appendix G) in R12upgrade manual

• Create a “Master” R12 APPLTOP

• Restart all database instances for CBO analyze

• Parallelize pre- and post-upgrade technical activities wherefeasible

• Prioritize concurrent programs / requests which must completeprior to system release

Issues and LessonsKeys to a Successful Upgrade

• Build step-by-step transition plans foreach test round

• Isolate performance testing of eacharchitectural change

• Collaborative planning across all projectteams

• Complete architecture changes inadvance

For More Information

or oracle.com

search.oracle.com

GSI Lessons

GSI Architecture

GSI Data Guard

“Oracle Running Oracle” (OOW S299990)

“Lessons Learned in Upgrading to Financials Release 12: Oracle’sInternal Upgrade” (OOW S299147)

“How Oracle DBAs Control the Data Center Explosion” (S298471)