Aaron Rimel Team Lead

October 2015

Upgrading Oracle Enterprise Manager: Why and How

© 2011 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information which may not be reproduced or transmitted without the express written consent of Cerner. 2

Innovating at the intersection of Health Care and IT

World’s largest standalone health care IT company•$2.9 billion in revenues•21,000 associates+ worldwide•14,000 client facilities in 30+ countries•4,900 physician practices •590 ambulatory facilities •3,500 extended care facilities •150 employer sites •1,790 retail pharmacies

R&D commitment and clinical culture•$3.8 billion of cumulative R&D •3,000+ person software development org•1,800 clinicians•More than 140 health IT patents

#4Top 100 Healthiest Workplaces in America

#22Most innovativecompanies 2014

© 2011 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information which may not be reproduced or transmitted without the express written consent of Cerner. 3

CernerWorks52,403 Systems

1,034 Network Circuits90 PB Disk Storage

CernerWorks52,403 Systems

1,034 Network Circuits90 PB Disk Storage

Managed services

386 contracts248,576 Contracted Users / 247,856 Peak Users

245,851 Beds = 27 % of total beds in United States36,971 Facilities

Managed OMS

33 Clients

DBA4 Clients

DisasterRecovery

33 Clients

Remote Hosted356 Clients

ITWorks15 Clients

CloudCerner Wellness359 Databases

Patient Portal943 (290 Acute, 684 PWx)

Healthe Record RHIO / HIE13 Sites

Healthe Transaction/Exchange3 TPA Clients

Skybox180 Clients

SaaSPowerWorks2,718 Clients

20,215 Concurrent Users

CommunityWorks72 Clients

© Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information belonging to Cerner Corporation and/or its related affiliates which may not be reproduced or transmitted in any form or by any means without the express written consent of Cerner.

4

EM @ CERNER

• Oracle Management Server version 12.1.0.4.4

• Three EM installations, Production/Sandbox/Development

• Repository version:11.2.0.3 – 3.5tb in size

• EM Agents: 1500 Agents, 97% RHEL, 3% other

• Targets 25k

• Tableau, Vertica

• AWR Warehouse

© 2011 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information which may not be reproduced or transmitted without the express written consent of Cerner. 5

OEM Monitoring at Cerner

60M Raw Metrics Daily

60M Raw Metrics Daily

540 Tickets

33,000 Incidents33,000

Incidents

18,000 Incidents18,000 Incidents

2,300 Alarms2,300

Alarms

© 2011 Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information which may not be reproduced or transmitted without the express written consent of Cerner. 6

Corrective Action statistics

© Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information belonging to Cerner Corporation and/or its related affiliates which may not be reproduced or transmitted in any form or by any means without the express written consent of Cerner.

7

EM 13.1.0.0 – Way to go !

• We love the ease of installation. =)

• Gold agent image, ora check plug-in

• Single point solution for Agent Lifecycle

• Flash is gone… (almost)

© Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information belonging to Cerner Corporation and/or its related affiliates which may not be reproduced or transmitted in any form or by any means without the express written consent of Cerner.

8

EM 13.1.0.0 : Upgrade Vs Install ?

• 13.1 Upgrade or Fresh Install?• Large repository, extended upgrade time• Creating a new multi EM installation• Repository clean-up, upgraded from 10.1.x,10.2.x,11.1,12.1.x• Methodical upgrade, allows for a client focued upgrade, making

changes to client systems when it is right for them.

© Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information belonging to Cerner Corporation and/or its related affiliates which may not be reproduced or transmitted in any form or by any means without the express written consent of Cerner.

9

How we Manage Agents with Same Configuration

• We were way behind the Oracle Patch recommendations.• Prior to Golden Agent Images, upgrading/patching agents would

have taken a FTE. We almost never patched agents.• EM now has a patch/psu rotation. We can now plan and execute

our EM agent upgrades staying current with PSU’s or applying a one-off to fix an immediate problem.

• Upgrading agents prior to GAI, took nearly a year to accomplish due to the vast number of agents that exist.

The implementation of Gold Agent Images has altered our deployment schedule to be in-line with PSU.

© Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information belonging to Cerner Corporation and/or its related affiliates which may not be reproduced or transmitted in any form or by any means without the express written consent of Cerner.

10

Solution: Gold Agent Lifecycle Feature

Requirements (One time activity)• Software Library Configured • Source Agent to Create Gold Image – Fresh deploy or use existing agent

Step1: Apply patches, configuration changes to the Source AgentStep2:Test the Source Agent – Check if agent is up, uploadingStep3: Create Gold ImageStep4: Use Gold Image to deploy, update, upgrade Agents (Recurring)Step5: Apply any new patches, Config changes to Source Agent Step6: Create another version of Gold Image

© Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information belonging to Cerner Corporation and/or its related affiliates which may not be reproduced or transmitted in any form or by any means without the express written consent of Cerner.

11

Solution: Gold Agent Lifecycle Feature

• We have updated /etc/sudeors to allow for our OS user to run commands specific to the agent, ie root.sh, allroot.sh to make the upgrade even easier by removing the need for root access to the machines.

Activity with Agent Gold Image Password Required Root.sh execution

Fresh Deploy Password required and supports SSH keys, PBRUN, Sudo, all EM supported authentications

Yes

Update (Patching) agent with newer version of Gold Image

Password required Root.sh depends on what is the type of updatea)If the patch need sbin changes then it requires root.sh execution.b)For config property changes, root.sh is not required

Upgrade agent using Gold image No password required, uses existing OMS-Agent communication

© Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information belonging to Cerner Corporation and/or its related affiliates which may not be reproduced or transmitted in any form or by any means without the express written consent of Cerner.

12

• oem ALL=(root,oracle) NOPASSWD:/bin/su - oracle*, /usr/bin/id, /u01/oracle/product/oemagent/agent13c/*/agentdeployroot.sh, /u01/oracle/product/oemagent/agent13c/*/*/agentdeployroot.sh, /u01/oracle/product/oemagent/agent13c/*/sbin/nmosudo *, /u01/oracle/product/oemagent/agent13c/sbin/nmosudo *, /u01/oracle/product/oemagent/agent13c/*/root.sh

© Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information belonging to Cerner Corporation and/or its related affiliates which may not be reproduced or transmitted in any form or by any means without the express written consent of Cerner.

13

• Base Directory Structure of Fresh Agent Install Using a Gold Image

<agent_base_directory> |_____core |_____<agent_software_version> |_____plugins |_____agent_inst |_____sbin |_____plugins.txt |_____plugins.txt.status |_____agentimage.properties

• Base Directory Structure of Upgraded Agent Using a Gold Image

<agent_base_directory> |

_____GoldImage_<gold_image_name>

|_____core |

_____<agent_software_version> |_____sbin |_____plugins |_____agent_inst

• Base Directory Structure Agent is upgraded or updated using a gold image multiple times

<agent_base_directory> |_____GoldImage_<gold_image_name_1> |_____core |

_____<agent_software_version> |_____sbin |_____plugins |_____GoldImage_<gold_image_name_2> |_____GoldImage_<gold_image_name_3>.. |_____agent_inst

Changes to Agent Directory Structure

Tip: Sudoers file should have $oracle_home set and agent user profile with right oracle home. Don't hard core the agent base dir/sbin in the sudoers file.

© Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information belonging to Cerner Corporation and/or its related affiliates which may not be reproduced or transmitted in any form or by any means without the express written consent of Cerner.

14

Why Out of Place Agent Updates ?

• Simple rollback process - If Gold Agent Update fails then your old Agent will come up automatically

• Simplified old Agent home cleanup

• Reduce downtime • Multiple Agents with multiple patches in a single downtime• While updating old Agent is brought down later in the process

© Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information belonging to Cerner Corporation and/or its related affiliates which may not be reproduced or transmitted in any form or by any means without the express written consent of Cerner.

15

Solution: Gold Agent Lifecycle Feature

Dashboard, real time tracking, maintenance via clicks

Evaluated as compliance against the latest ‘image’

Agents yet to be updated with Gold Image

© Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information belonging to Cerner Corporation and/or its related affiliates which may not be reproduced or transmitted in any form or by any means without the express written consent of Cerner.

16

Solution: Gold Agent Lifecycle Feature

Drifters – Detects manual Patching and drifts from Current Gold Image

Chart showing subscriber’s versions

Displays all artifacts included in the version of Gold Image

© Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information belonging to Cerner Corporation and/or its related affiliates which may not be reproduced or transmitted in any form or by any means without the express written consent of Cerner.

17

Solution: Gold Agent Lifecycle Feature

1.Gold Image can be Imported/Exported from a Test to Prod EM

2.Agents can be updated with Gold Image using Push or Pull

3.Define Batch size, success criteria and frequency which will reduce workload on OMS and help you plan your updates

4.Updates can be planned during maintenance and comfortable time periods

5.Get email notification for Success/Failed/Status for each Agent update in a Batch

6.Ability reclaim space by cleaning out old Agent Oracle Homes

7.Run just Prerequisites before actual update operation

1

2

3

4

5

6

7

© Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information belonging to Cerner Corporation and/or its related affiliates which may not be reproduced or transmitted in any form or by any means without the express written consent of Cerner.

18

Solution: Gold Agent Lifecycle Feature

Export Gold Images across Test /Dev/Prod EM environments

1

2

3

© Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information belonging to Cerner Corporation and/or its related affiliates which may not be reproduced or transmitted in any form or by any means without the express written consent of Cerner.

19

Solution: Gold Agent Lifecycle Feature

Single view for all activities

© Cerner Corporation. All rights reserved. This document contains Cerner confidential and/or proprietary information belonging to Cerner Corporation and/or its related affiliates which may not be reproduced or transmitted in any form or by any means without the express written consent of Cerner.

20

Solution: Gold Agent “EMCLI” scripting option

• Used EMCLI Gold Agent commands for updating 2500 Agents!

• Worked like a charm

• Repeatable and reliable process

• Use cases executed: • Upgraded Agents 12.1.0.3 to 12.1.0.4 • Updated all Agents 12.1.0.4 to 12.1.0.4.1 • Again updated Agents from 12.1.0.4.1 to 12.1.0.4.2