Best Practices for Upgrading to Oracle Enterprise Manager ... › technetwork › oem ›...
Transcript of Best Practices for Upgrading to Oracle Enterprise Manager ... › technetwork › oem ›...
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Best Practices for Upgrading to Oracle Enterprise Manager 13c: Learn from CustomersCON6989
Sumesh BalakrishnanPrincipal Product Manager, Oracle
Yong Kwong LungSystem Integration Advisor, DELL
Presented withSeptember 22, 2016
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Agenda
Why Upgrade to EM 13.2 ?
New! Features of EM Lifecycle Management
Best Practices for a Smart Upgrade
Let’s hear it from -- DELL
1
2
3
4
Copyright © 2016, Oracle and/or its affiliates. All rights reserved.
Why Upgrade to EM 13.2Building On Success
Enterprise Manager 12c
Enterprise Manager 13.1
Enterprise Manager 13.2
• Enhanced Hybrid Cloud Management
• Fleet Maintenance
• TLS1.2 & IPV6 Support
• Stability & Performance Improvements
• Latest FMW Stack
• Enhanced Always on Monitoring
• Unified Hardware Management
• Compliance Management
• DB: Fine Grained Access Control
• Engineered Systems Management
• Database Consolidation Workbench
• Auto Discovery, Agent Push
• Patch Advisor
• Patch Plans
• Patch Automation
• Database management
• My Oracle Support Integration
• RAC provisioning
• Application Performance Management
• Always On Monitoring
• Unified Hardware Management
• Drift, Consistency and Compliance Management
• Database Management: Fine Grained Access Control
• Engineered Systems Management
• Database Consolidation Workbench
• Auto Discovery, Agent Push
• Patch Automation
• Database Management
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Upgrade Step by Step Planning
Pre-Flight– Plan your upgrade
– Upgrade paths
– Fallback strategy
– Talk to us (Oracle) to avoid pitfalls
Production– Stage the new Software
– All in one downtime window
– OMS Bundle Patches
Test System– Development system upgrade
– Test system upgrade
Post Upgrade– Performance Monitoring
– New features
1 3
24
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Upgrade Paths to EM 13.2
• Supported Direct Upgrade Path
12.1.0.4/12.1.0.5/13.1 13.2
• Pre-12.1.0.x Upgrade Path
11g /10.2.0.5 12.1.0.5 13.2
13.2
12.1.0.4 12.1.0.5 13.1
11.1/10.2.0.5
12.1.0.3
Copyright © 2016, Oracle and/or its affiliates. All rights reserved.
EM 13.2 Lifecycle Management
• Install / Upgrade
– Support on IPV6 platform
– Support for TLSv1.2 for DBs
• EM Prereq kit utility in GUI mode
• Reduced downtime: Smart-Upgrade
• Transition to DR Readiness – Storage Based Replication
• JDK 1.7U111 uptake with SHA2 certificate
• 12c Agent [version 12.1.0.5] deployment from 13.2 EM
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
EM Repository Health Check
• Enhanced EM Prereq Kit execution in OUI mode
– Option to installer em_linux64.bin EMPREREQ_KIT=true
Upgrade / Fresh Install
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
12.1.0.5 Agent Support on OEL 5.x
• PS4 Agent deployment on OEL 5.x platform from 13.2 OMS
• Deployment method supported
– Only emcli submit_add_host
• 12c Agent Life cycle– Agent Core Bundle Patch
emcli submit_add_host -
host_names=example.com -platform=226 -
installation_base_directory=/opt/agent -
credential_name=oracle -
privilege_delegation_setting=sudo -
agent_version="12.1.0.5.0"
EM 13.2
12c Agent on OEL/RHEL 5.x Hosts
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
EM 13.2 – Upgrade Recommendations
• Upgrade options supported : 1-System
• OMS and Repository Upgrade
– Upgrade your existing repository database to 12.1.0.2
– Apply latest Bundle Patch on top of 12.1.0.2 Database
– Perform Software only install with Plug-ins (Less Downtime - Recommended)
• Agent Upgrade
– Upgrade Target Agents using Gold Agent Image
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Upgrade — runInstaller (OUI)
I. Mandatory: Upgrade Repository DB to 12.1.0.2
II. OMS Downtime & execute em_linux64.bin (Use1-system option in Installer)
– Copying the bits and creating new homes– Start the configuration assistants
III. Use Gold Agent Image console to upgrade agents to 13.2
Reduce the downtime using – Smart Upgrade
Upgrade – Software + Plug-in only Install
I. Mandatory: Upgrade Repository DB to 12.1.0.2
II. Advanced Option - Copying the bits/plug-ins and creating new homes – OMS + Plug-ins
III. Apply 13.2 OMS and Plug-in bundle patch in ‘bit-only’ mode (optional )
IV. OMS Downtime Start - Configuration$<MW_HOME>/sysman/install/ConfigureGC.sh
V. Use Gold Agent Image console to upgrade agents to 13.2
DOWNTIME REDUCED
Copyright © 2016, Oracle and/or its affiliates. All rights reserved.
Upgrade Best Practices Repository DB should be 12.1.0.2 version with latest bundle patch
Run EM diag kit.
Before Upgrade
Remove obsolete plug-ins from OMS/Agent e.g. Exalogic Elastic Cloud, EMC Symmetrix …etc.
Download the latest version of All Oracle plug-ins from otn e.g. Ebiz, PSFT, GoldenGate …etc.
Upgrade on certified platforms. Refer to MOS certification matrix for details
Perform Software only Install with Plug-ins to reduce downtime
Weblogic customizations are not carried...
pre-upgrade remove them and post upgrade redo the customization
Upgrade ALL Agents to 12.1.0.4/5 before Upgrading the OMS
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
EM 13.2: Upgrade & Transition to DR ReadinessSingle Domain Setup with Storage Replication
Primary Site Standby Site
OMS 1 (Active)Physical: Host 1Alias: emoms1
OMS 2 (Active)Physical: Host 2
Alias: emoms2
OMS 1 (Passive)Physical: Host 3
Alias: emoms1
OMS 2 (Passive)
Alias: emoms2
Physical: Host 4
STORAGE REPLICATION
STORAGE REPLICATION
STORAGE REPLICATIONOMS1 SoftwareGCDomain
OMS2 SoftwareGCDomain
ReposRAC DB
ReposRAC DB
Shared or ReplicatedSWLIB and BIP Shared
Storage
OMS1 SoftwareGCDomain
OMS2 SoftwareGCDomain
Shared or ReplicatedSWLIB and BIP Shared
Storage
Alias Host Name configuration for OMS hosts
OMS and Central Agent binaries are on replicated storage
OMS and Central Agent inventory is on replicated storage
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
EM 13.2: Single Step Adoption of Level-4 HA
• EM 13.2 Installer has new mode Upgrade and Transition to DR Readiness
– Enabled by passing new parameter UPGRADE_TRANSITION=true
– Also requires additional new parameter OLD_INVENTORY_LOCATION
– Extends support for existing parameter ORACLE_HOSTNAME
– Uses existing parameter –invPtrLoc
– Only supported in OUI ( no support for silent or Software Only or ConfigureGC.sh )
• Installer performs the following in this mode– Relocation of OMS to replicated storage (software and inventory)
– Configure the OMS to use alias hostname
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Update Agents with Gold Image and propagate with ease
Create Gold Image and versions for each platform series.Subscribe agents to the images
Select source agent from the system and apply patches and test the agent
Push ImageAnd Update
Agents
Create Gold Image &
SubscribeAgents
Agent Upgrade Using Gold Agent ImageSimplified Agent Upgrade Operations at Scale
Patch SourceAgent
Ver. 3
Ver. 2
Ver. 1
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Agent Upgrade Using Gold Agent Image321
2. Create Gold Images and Subscribe
Agent Image – Versioning
Ver. 3*
AUG PSU + couple more patches*
AUG 2016
JULY PSU + Config Changes
JULY 2016
Ver. 2
Ver. 1
APR 2016
APR PSU + 2 patches
A. Define Agent software as Images and Versions.
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Subscribes to
Subscribes to
Agent Upgrade Using Gold Agent Image2. Create Images and SubscribeB. Subscribe Agents to the images
12.1.0.4/5/13.1 Agent(s)
321
Image ALinux x86_64_AGT_IMG
Ver. 3*
Ver. 2
Ver. 1
Subscribes to
Image BSolaris SPARC_AGT_IMG
Ver. 3*
Ver. 2
Ver. 1
Image CWindows x64_AGT_IMG
Ver. 3*
Ver. 2
Ver. 1
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Agent Upgrade Using Gold Agent Image3. Propagate Changes and Upgrade Agents with Ease
321
DBAs Application Owners
Ver. 2
Push Agent Image on Targets Ver. 3*
Ver. 1
Ver. 3*
Ver. 2
Push Agent Image on TargetsVer. 3*
Ver. 1
Ver. 3*
Ver. 2
Push Agent Image on TargetsVer. 3*
Ver. 1
Ver. 3*Ver. 3*
Ver. 3*Ver. 3* Ver. 3
Ver. 3
Ver. 3
Ver. 3
Ver. 3
Ver. 3
Ver. 3
Ver. 3
Ver. 3
Image ALinux x86_64_AGT_IMG
Image BSolaris SPARC_AGT_IMG
Image CWindows x64_AGT_IMG
Linux x86_64 BATCH 1 (1000 agents)
Solaris BATCH 2 (1000 agents)
Windows BATCH 3 (1000 agents)
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Quick Patching – 13.2
No Auto Rollback of Patches during apply
Minor changes while packaging bundle patches
Time Reduction while applying bundle patches
Pre-Requisite:
Update the new OMSPatcher in OracleHome
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Traditional vis-à-vis Quick patching
BP – Bundle Patch
M I
N U
T E
S
1.754.22
14.95
34.45
1.753.47
10.1
17.22
0
5
10
15
20
25
30
35
40
Feb BP1 Apr BP2 July BP3 Aug BP4
Regular Patching Smart Patching
50%Time Reduction
Global Marketing
Oracle Enterprise Manager Migration from 12c to 13.1
Yong Kwong LungSystems Integration Advisor
Dell | IT Engineering, Infrastructure & Operation
Global Marketing
Agenda Summary
• Overview Oracle Enterprise Manager at Dell
• Enterprise Manager integration in Dell (Upstream and Downstream Apps)
Oracle Enterprise Manager in Dell
• Overall planning of hardware, architecture, key focuses – at a glance
Architecture, Hardware Spec & key Focuses
• Best practices / Activities during OEM13c migration
• Resource allocation plan
• Mass agent deployment via gold image and application plugin such as E-Business Suite, SOA, and Siebel
• Gold Agent Image Result
• Targets Discovery via EMCLI
• Lesson Learned, Team Feedback
Dell Approach on OEM Migration
Global Marketing
Oracle Enterprise Manager in Dell
Global Marketing
Overview Oracle Enterprise Manager at Dell
7.4KURL Monitoring
29KDatabase Targets
57KApplication Targets
(Middleware/EBS/Siebel/PS)
App/DB Servers Monitored in Production Enterprise Manager
App/DB Servers Monitored in Non-Production Enterprise Manager
2.6KAgents
350Reports
5DownstreamApplication
1.4KUDM
DB/Host
15Jobs /
Automation
1.5KUser
3KGrouping
Global Marketing
Monitored Targets Dell In-house ApplicationsOracle Enterprise Manager
Baseline Metric
Admin Group
Incident Rule
User Access
Reports
UDM
Notify Method
Targets DiscoveryAgent Installation
Compliance Policies
Groups
Targets Properties
OMR
(Database)
NT Authentication
PAC Group
Email Notification
SMTP Server
Incident (Remedy)Ticketing
Incident (ServiceNow)
JsonIncident
Convert to ServiceNowDec/2016
RMAN BackupDeployed in 2007
DMMR
Database Metadata Repository
Deployed in 2010
DBCM
Database Compliance Mgmt
Deployed in 2010
KPI
Key Performance Indicator
Deployed in 2013
DMP
Database Maintenance Planner
Deployed in 2012
Deployed in 2007
Oracle DB Monitoring
10g, 11g, 12c
Pilot Test 2011
Deployed in 2012
Oracle SOA Monitoring
10g, 11g
Pilot Test 2013
Deployed in 2015
Pilot Test 2013
Deployed in 2016
Oracle EBS Monitoring
11g, 12c
Oracle SIEBEL Monitoring
10g, 11g, 12c, 13c
Pilot Test & Deployed in 2015
URL Monitoring
N/A
Enterprise Manager in action at Dell
Automation
EMCLI
Jobs
Automation Service Improvement
Since 2010-2016
Global Marketing
Architecture, Hardware Spec & Key Focuses
Global Marketing
Support Disaster Recovery
Zero downtime for OEM rolling patching
New Enterprise Manager ArchitectureFrom 12c Level 3…
DataGuard
Oracle
Management Service
(13c)
Oracle Management
Repository (13c)
Oracle Database 12c
Oracle
Management Agent
13c
Archive: 512GB, FRA: 512GB, Data: 3TB Archive: 512GB, FRA: 512GB, Data: 3TB
Production Application Server Cluster
(U.S., Austin, PC1) Dell 13g M630'sDR Application Server Cluster
(U.S., Austin, PS3) Dell 13g M630's
EM12P3.PrdRAM 128GB
CPU 36 Core/u01 100GB
EM12P1.PrdRAM 128GB
CPU 36 Core/u01 100GB
EM12P2.PrdRAM 128GB
CPU 36 Core/u01 100GB
EM12P4.PrdRAM 128GB
CPU 36 Core/u01 100GB
Primary Cluster DB
(U.S., Austin, PC1) Dell 13g M630's
DR Cluster DB
(U.S., Austin, PS3) Dell 13g M630's
EM12P1.DRRAM 128GB
CPU 36 Core/u01 100GB
EM12P2.DRRAM 128GB
CPU 36 Core/u01 100GB
EM12P3.DRRAM 128GB
CPU 36 Core/u01 100GB
EM12P4.DRRAM 128GB
CPU 36 Core/u01 100GB
400GB
Additional OMSRAM 128GB
CPU 36 Core/u01 250GB
Additional OMSRAM 128GB
CPU 36 Core/u01 250GB
Primary ServerRAM 128GB
CPU 36 Core/u01 250GB
Additional OMSRAM 128GB
CPU 36 Core/u01 250GB
Primary ServerRAM 128GB
CPU 36 Core/u01 250GB
Additional OMSRAM 128GB
CPU 36 Core/u01 250GB
Additional OMSRAM 128GB
CPU 36 Core/u01 250GB
Agent Agent Agent Agent Agent Agent
400GBRsync
Additional OMSRAM 128GB
CPU 36 Core/u01 250GB
Additional OMSRAM 128GB
CPU 36 Core/u01 250GB
Additional OMSRAM 128GB
CPU 36 Core/u01 250GB
…to 13c Level 4
Shared Storage
Production Application Server Cluster
(U.S., Austin, PC1)
EM12P3EM12P1 EM12P2 EM12P4
em12p.prd.amer.dell.com (U.S., Austin, PC1)
Additional OMSAdditional OMSPrimary Server Additional OMS
Agent Agent Agent
Global Marketing
Monitored Targets Dell In-house ApplicationsOracle Enterprise
Manager
Baseline Metric
Admin Group
Incident Rule
User Access
350 Reports
1.5K UDM
Notify Method
Targets Discovery2.6 Agent Installation
Compliance Policies
3K Groups
Target Properties In-house Custom App build on Oracle APEX
OMR
(Database)
Non-Standard DBSNMP Password
Oracle DB Monitoring
10g, 11g, 12c
Convert to ASMSNMP Account
Oracle SOA Monitoring
10g, 11g
Linux (V5, V6 & V7)
Windows Servers
Existing 12cR2 Agent
Oracle EBS Monitoring
11g, 12c
Oracle SIEBEL Monitoring
10g, 11g, 12c, 13c
Unhealthy / Pending Decommission Targets
URL Monitoring
N/A
Project Implementation Key Focuses
Automation
EMCLI
Jobs
15 Automation for Service Improvement
High – Resource & Time
Medium – Resource & Time
Low– Resource & Time
NT Authentication
PAC Group
Email Notification
SMTP Server
Incident (Remedy)Ticketing
Incident (ServiceNow)
JsonIncident
Convert to ServiceNowDec/2016
RMAN Backup
DMMR
Database Metadata Repository
DBCM
Database Compliance Mgmt
KPI
Key Performance Indicator
DMP
Database Maintenance Planner
Global Marketing
Dell Approach on OEM Migration
Resource Plan
Migration Plan
Agents Installation
Targets Discovery
Global Marketing
Group 3:
OEM Administrator
Gro
up
2:
Do
wn
stre
am
Ap
ps In
teg
ratio
n
Gro
up
1:
OE
M1
3c A
ge
nt M
igra
tio
nOracle Enterprise Manager
Baseline Metric
Admin Group
Incident Rule
User Access
350 Reports
1.5 UDM
Notify Method
Targets DiscoveryAgent Installation
Compliance Policies
3K Groups
Target Properties
OMR
(Database)
Oracle DB Monitoring
10g, 11g, 12c
Oracle SOA Monitoring
10g, 11g
Oracle EBS Monitoring
11g, 12c
Oracle SIEBEL Monitoring
10g, 11g, 12c, 13c
URL Monitoring
N/A
Resource Allocation Plan
350 Reports1.5k UDM
ComplinacePolicies
3K GroupsGroup 4:
Overall OEM Functional Validation, Monitoring health check and performance test
Project Manager
Automation
RMAN Backup
DMMR
Database Metadata Repository
DBCM
Database Compliance Mgmt
KPI
Key Performance Indicator
DMP
Database Maintenance Planner
Global Marketing
Migration Plan:Two System Approach
Business Decision
• Downtime > 2 hrs.
OMR Health Concern
• Potential upgrade failure
End to End Infra Changes
• SLB, OMS, OMR
End to End Integration Testing
• Upstream & Downstream App
UDM convert to Metric Ext
• Complexity of the conversion
Customer Sign-Off
• Go live sign-off
Enterprise Manager 13c
OMR
Current OMR Server New OMR Server
OMR
PrimaryServer
AdditionalOMS
Agent 12c
/u02
/u01/u01
Enterprise Manager 12c
PrimaryServer
AdditionalOMS/u02
/u01/u01
Agent 13c Agent 12c Agent 13c Agent 12c Agent 13c Agent 12c Agent 13c
Global Marketing
Create Gold Agent Images
{APP}_{OS}{PLATFORM}
E.g.: odb_linux64
Pre-requisite• On OMS Server
• 24308359-August Bundle Patch 13.1.0.0.160816
• 24364619-Plugins 13.1.1.0.160816
• 22741755-Support OEL5.x
• On Agent Target Server (one only)
• Deploy 13.1 Agent
• Plugin Deploy
• 24308442-August Bundle Patch 13.1.0.0.16081
{APP}_{OS}{PLATFORM}_{VER}
E.g.: odb_linux64_v1
{APP}_{OS}{PLATFORM}
E.g.: odb_linux64
Application OS Platform Gold Image Version
Oracle E-Business Suite Linux 64bit EBS_Linux64 ebs_linux64_v1
Oracle Database Linux 64bit ODB_Linux64 odb_linux64_v1
Oracle SOA Linux 64bit SOA_Linux64 soa_linux64_v1
Oracle SIEBEL Win 64bit SIEBEL_Win64 siebel_win64_v1
1
2
3
4
5
6
Global Marketing
Agent Installation using Gold Agent Images
1
2
3
Add New Server
Gold Agent Image
Installation Details
Start Agent Deployment
Fresh Agent Installation
Agent Upgrade 1
3
Subscribe Agent(s)
Start Agent Deployment
Select Agent(s)
2
Global Marketing
Gold Agent Images Result
Day 1
•333 DB hosts
Day 2
•4 SEIBEL hosts
Day 3
•26 EBS hosts
Day 4
•195 SOA hosts
Day5
•229 DB hosts
604 hosts success
Pilot Test Result183 of 787 host failed:
• SSH Daemon Error
• Agent File-system Full
• Lack of Account Permission
• Un-contactable Servers
•Agent Installation =
•Patches Apply =
•Plugin Deploy =
•Parameter Update =
With Gold Agent Images (5 Hosts)
30Minutes
•Agent Installation = 24 minutes
•Patches Apply = 20 Minutes
•Plugin Deploy = 20 Minutes
•Parameter Update = 10 Minutes
Without Gold Agent Images (5 Hosts)
74 Minutes
604 Agents
Agent yet to be updated
Agent without images subscribe
Image Name with Total Agents
Compliance Status
Global Marketing
• Retrieve Targets Configuration
• gc$target
• gc$target_properties
Preparation
• Generate EMCLI command
• emcli add_target
Converting• Register Targets to OEM13c via EMCLI
Execution
Enterprise
Manager 12c
Enterprise
Manager 13c
Targets Discovery
ClusterCluster DB
Weblogic Domain
Cluster ASMDB InstanceListener
User Registration
ASM Instance
GroupingTarget’s Properties UDM conversion
User Role Granting
Fusion Middleware
Global Marketing
select
t.target_name,
t.host_name,
'$ORACLE_HOME/bin/emcli add_target -name=""' ||t.target_name
|| '"" -type=""oracle_listener"" -host=""' || t.host_name
|| '"" -properties=""LsnrName:' || p.lsnrname
|| ';ListenerOraDir:' || p.listeneroradir
|| ';Port:' || p.port
|| ';OracleHome:' || p.oraclehome
|| ';Machine:' || p.machine || '""'
from gc$target t
inner join ( select
target_guid,
max(decode(property_name, 'LsnrName', property_value, null)) as lsnrname,
max(decode(property_name, 'ListenerOraDir', property_value, null)) as listeneroradir,
max(decode(property_name, 'Port', property_value, null)) as port,
max(decode(property_name, 'OracleHome', property_value, null)) as oraclehome,
max(decode(property_name, 'Machine', property_value, null)) as machine
from mgmt_target_properties
group by target_guid) p
on t.target_guid = p.target_guid
where t.target_type ='oracle_listener'
Listener Target Discovery Example:Preparing EMCIL command line
Target Created
Target’s status healty
Global Marketing
Feedback by Dell Project Team…
Gold Images were excellentGood so far
Less effort and time
Strategic guidance AND execution are clearOEM13c Better UI & Performance
“Improve my knowledge”
Save CostEasy to follow up
Customer Loves it!Never patch Agent againSpeed up targets migration via EMCLI
Less communication gap
Proud to be part of the team!!
Global MarketingDell - Restricted - Confidential
Thank You
Yong Kwong Lung
Systems Integration Advisor
Dell | IT Engineering, Infrastructure & Operation
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Zero Downtime UPGRADE
Zero Downtime PATCHING
Parallel OMSUPGRADE
What’s Next..
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Additional Resources
Join the Conversation
Twitter.com/oracle_em
Oracle.com/newsletters
Facebook.com/oracleenterprisemanager
Blogs.Oracle.com/OEM
Enterprise Manager Page on O.com
Install and Upgrade Page on OTN
Upgrade Guide (Documentation)