Ravi Ammamuthu Sarayu Damodaran Virendra Galotra Sep 25, 2013 · •Sarayu Damodaran •Virendra...

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 1

• Ravi Ammamuthu

• Sarayu Damodaran

• Virendra Galotra Sep 25, 2013


Integrated Global Monitoring

Database Provisioning

Database Patching

Q & A


IT manages hundreds of databases if not thousands

Global Data Centers

Heterogeneous Systems

Complexity

Lack of Agility

Flat/Shrinking IT budget


Integrated Global

Monitoring

Database Provisioning

Database Patching

Database Upgrade


Prod

293 *

Standalone

86

RAC

207

Standby

96

Non-Prod

1011 *

Standalone

486

RAC

525

Standby

40 * Total excluding Standby DBs


Cloud Control & Targets

Targets Only

Legend

Total targets - ~22000

Targets by category are: • Prod - 488 hosts, 293 DBs • NonProd – 1216 hosts,

1011 DBs • DG – 69 hosts, 96 DBs

Targets on UCS, Sun-OS

DB Repository Size

• 600 GB for PROD1

• 1500 GB for PROD2

112 Metrics

32 Reports & 20 Jobs

Global OEM users

STG &

DEV

PROD2 NonProd

Targets

PROD1 Prod

Targets

DR


DB Repos

Passive

OMS

DB Repos

Active

OMS

Agents

Global Site Selector (GSS)

ACE ACE

RTP RCDN

Redo Shipping

S/W Library Replication

Snapmirror-Async

DR Primary

Richardson RTP Amsterdam


DB Repos

Active

OMS

Active

OMS

Passive

OMS

DB Repos

DB Repos

Agents

Global Site Selector (GSS)

ACE ACE ACE

RTP Allen RCDN

Redo Shipping Redo Shipping

S/W Library

Snapmirror- Async

MVDC DR

S/W Library

Metrocluster

Richardson RTP Amsterdam


• Self Healing Through Corrective Actions

Feature

Restart Database, Listener

• Tablespace Space

Addition

• File System Cleanup


• Cluster Database Performance


• Historical Information

• Ability to drill down and narrow down

• Graphical Representation

• Comparison of configurations across

environments

• Agile report generation


• Segregation of Prod and Non-Prod Targets

• Apply Standard Templates Periodically

• Adopt Metric Extensions

• Avoid Storing Job Outputs in Repository

• Always be on Latest Patches

• Eliminate Customization as much as Possible


• OEM 10.2.0.5

• Centralized Monitoring of Prod and Non-Prod DBs

• Data Masking

• User Defined Metrics (UDM)

• OEM Infrastructure Migration to new Data Center

• 4 OEM Cloud Control LC Envs

• Enterprise wide Patching of DBs using OEM

• OEM 12.1.0.1 Upgrade

• Monitoring ASM DBs

• Self Healing

• Automatic feed of OEM data to Portal

• Provision Grid Infrastructure

• Provision RAC & Standalone DBs

• OEM 12.1.0.2 Upgrade

• Metric Extensions

2012 2009-2011 2013


Situation

• IT Vision : Provision Infrastructure Stack in 5 Days

Complication

• Complexity : Virtualization, Cloud, Inter-connected Services

Implication

• Scalability : Enhancement to Custom Tool for each new release

Position

• Identify Tool to scale horizontally & vertically to meet demands - OEM

Action

• Use Out-of-Box Solution with little or no Customizations

Benefits

• Faster Provisioning

• Optimization

• Quality

• Adhere Standards

• Less OPEX


Software Deployment

Validate Server, Network, Storage, Network, Storage

Set up Oracle Home from existing Environment

Standalone Database

Validate Server, Network, Storage

Install Oracle Software

Create Standalone Database

Setup/Validate Cisco Standards

RAC Database

Validate Server, Network, Storage

Install Grid Infrastructure

Configure Grid Infrastructure

Install Oracle Software

Create RAC Database

Setup/Validate Cisco Standards


• Manual Installation Process scripts, silent mode pre-requisite checks

• Lack of Standardization

• Long and Error Prone Manual Process

• Increased Resource Requirement

• Maintain Up-to-date Releases

Solution:

Enterprise

Manager (OEM) to

Provision

Databases


Clone Oracle Home

Clone, Configure

GI

Create Database

Create RAC

Database

Clone Oracle Home

DB

Type

?

Deployment Procedures

A. Cisco Provision Oracle

Database 1. Deploy Database Software

2. Create a New Database

Non-RAC

RAC

A.1 A.2

B.1, B.3 B.2 B.4

B. Cisco Provision Oracle

RAC Database 1. Deploy Grid Infrastructure

2. Deploy Database Software

3. Configure Grid

Infrastructure

4. Create a New Database


• Launch Cisco Procedure and provide inputs


• Real Time monitoring of step status • Drill down to log levels directly

Deployment procedures

extended to include CISCO

Customizations


Prod

293 *

Standalone 86

RAC

207

Standby

96

Non-Prod

1011 *

Standalone

486

RAC

525

Standby

40 * Total excluding Standby DBs

* 60% software only deployments

OEM Provision

222

Standalone DP

• DB 43

• w/o DB 65 *

RAC DP

• DB 48

• w/o DB 69 *

From

Feb’13


0

2

4

6

8

10

12

14

16

18

Manual OEM

8

2

3

2

6

0.5

H

o

u

r

s

RAC DB

Create RAC DB

Clone Oracle Home

Install & Configure GI

0

1

2

3

4

5

6

7

Manual OEM

3

2

4

0.5

H

o

u

r

s

Standalone DB

Create non-RAC DB

Clone Oracle Home

65%

Savings

74%

Savings

* Grid and Oracle Home includes patch storage

Multiple Environments are also provisioned at the same time

Environments Provisioned using EM12c = 222 (from Feb 2013)

Avg. No. of Concurrent Envs. Provisioned = 6


Value Metrics

Faster Provisioning of DBs

• 4x reduction for RAC cluster

Increased Agility • 75% SLA Improvement

Reduced OPEX • Scaled down resources from 4

to 1 for 222 Envs

Increased Scalability • Multiple clusters in a single go

• No Extra Efforts for Scale-Ins

Increased Quality • Standards Maintained

• Incident Prevention

• Minimal Errors


• Apply patch for DBCA in Cisco’s Gold Code Tree

DB Templates not supporting multiple file locations

• Fix manually; Proceed with ignore option

Sudo privileges prevent running Auto-Fix automatically

• Use reference target from same DC as source

Longer time to copy software across WAN

• Plug-in scripts to create database listeners

Database listeners are not registered on RAC DB


Standardization using Gold Image

Leverage hooks for any customizations

Use Referenced Components for Remote DBs; saves bandwidth and time

Define Templates; RAC, Non-RAC, ASM, dNFS


Enhance Standard Deployments

• Provisioning Profile with Locked inputs to

Deployment Procedures

• Segregation of Designer and Operator roles


• Quarterly Patching

• PSU, CPUs, One-off Patches

• 3200 DB Instances

Audit Requirement

• Extended Outages

• Resource Intensive

• Not Scalable Challenges

• Leveraged OEM to Patch DBs Solutions


STEPS Manual Mode EM Mode

Login Login to individual

Boxes

Centralized login

to EM

Schedule Patch N/A

Automated using

Enterprise

Manager

Download Patch

Run Pre-Patch

Shutdown DB

Apply Patch

Startup DB

Post Patch

Health Checks

Manual process by every single

DBA on every Database

Multiple patches can be

scheduled by one DBA or by

multiple DBAs


• Patch RAC & Standalone DBs

• Customized DP for Cisco’s processes using hook-ups Pre and Post requisite checks

Application start/stop scripts

• Merge & Apply many patches in a plan

• Rolling Patching

• Rules to notify failures/success


Dire

ct

Indire

ct

1000 DBs

Savings 3 DBAs.

Reduced

Incidents

Reduced

Human Error

Highly

Scalable

Disclaimer: Cisco specific numbers

Downtime (Hrs)

4000

Before OEM

After OEM

$65K

Before OEM

After OEM

Effort (Hrs)

660

4000 Before OEM

After OEM 2000

$385K

Cost


• Cause : Duplicate Cluster Target; unable to override named credential

• Disabled Auto-Discovery of Targets

Jobs Failed to Submit

• Cause : Opatch Bug 17417108

• STF : Group into multiple plans Multiple

Patch Plans

• Working with Oracle Performance Issue in UI

• Cause : Corrupted software library

• Fix : Rebuilt software library

Opatch download

process fails


• Run Patch Analysis before maintenance window

• Use MOS Online option for Patching, helps in direct patch download and conflict resolution

• Use Rolling Patch option to minimize downtime for RAC DBs

• Include multiple patches in a single plan to go in one downtime

• Manage Oracle resources using Central Inventory


Ease of patching for 12 Node Clusters

2 DBAs to patch 42 DBs in 6 Hrs.

* Increased Productivity

* Met Auditing timelines


Reduce Risk and Downtime further • Out of Place Patching of Standalone and RAC

Reduce Documentation:

• Better use of Patch Template to reduce

documentation and communication to DBAs


Q1FY14

OEM MAA (Level-4) using MetroCluster

Agent Restart

Monitor TimesTen

Auto Space Addition

Q2FY14

Cron Less Monitoring

Auto Agent Install

FS Clean Up

DB Upgrade

DG Build

Monitor Golden Gate

Q3FY14

EM Target Self Management

DB Mender

Monitor FMW

Upgrade OMS


Manager

• Ravi Ammamuthu

Architect

• Sarayu Damodaran

• Kunal Chakrabarty

Technical Team

• S Venky Sadagopan

• Pascal Laporte

• Manickaraja Kumarappan

• Surekha Jampala

• Venkat Kiran Gunturi

• Manikandan Ganesan

• Sairam Kantheti

PM

• Meena Chanda


Thank You

Ravi Ammamuthu Sarayu Damodaran Virendra Galotra Sep 25, 2013 · •Sarayu Damodaran •Virendra...

Documents

Transcript of Ravi Ammamuthu Sarayu Damodaran Virendra Galotra Sep 25, 2013 · •Sarayu Damodaran •Virendra...