PROJECT INFORMATION Enterprise Template Architecture ESB Regional... · format and sends these...

of 61

PROJECT INFORMATION Template

Enterprise Architecture

Title:

Enterprise Integration Platform Regional Design

Pre-Transfer

Unique Identifier: 206-1313

Document Type: ETE

Revision: 3.1

Total pages: 61

Revision date: September 2017

Classification:

PUBLIC/CONFIDENTIAL/ CONTROLLED DISCLOSURE

of 61

Table of Contents 1 Project Information ..................................................... Error! Bookmark not defined.

1.1 Relevant Business Units ...................................... Error! Bookmark not defined.1.2 List of Attached Supporting Documents ............... Error! Bookmark not defined.

2 Introduction ................................................................................................................ 33 Background ................................................................................................................ 34 Scope ......................................................................................................................... 55 Design Approach ........................................................................................................ 66 EIP Architecture Design ............................................................................................. 8

6.1 Business Architecture ........................................................................................... 86.1.1 Business Strategies and Plans ...................................................................... 86.1.2 Business Processes and Policies .................................................................. 96.1.3 Business Organisation Design ....................................................................... 9

6.2 Business Information Architecture ........................................................................ 96.2.1 Business Information Flow/Inventory ............................................................. 96.2.2 Information Integrity and Custodianship ........................................................ 96.2.3 Information Access and Confidentiality ........................................................ 106.2.4 Information Related Business Continuity ..................................................... 10

6.3 Data Architecture ................................................................................................ 116.3.1 Data Models ................................................................................................. 11

6.4 Application Architecture ...................................................................................... 116.4.1 EIP Regional Platform Application Architecture Deployment ...................... 116.4.2 EIP Application Functional Decomposition .................................................. 126.4.3 Application Strategy ..................................................................................... 226.4.4 Application Development ............................................................................. 22

6.5 Integration Architecture ...................................................................................... 226.5.1 Integration Interface Map ............................................................................. 226.5.2 Integration Services Directory ...................................................................... 236.5.3 Data Migration ............................................................................................. 23

6.6 Technical Architecture ........................................................................................ 246.6.1 Basic Infrastructure ...................................................................................... 24

6.7 Network Architecture .......................................................................................... 421.1.1 IT & DMZ Restrictions .................................................................................. 421.1.2 ODA Placement ........................................................................................... 441.1.8 Firewall Access ............................................................................................ 48

6.8 Security Architecture .......................................................................................... 517 High Availability and Disaster Recovery .................................................................. 608 Operational Support Structure ................................................................................. 60

of 61

1 Introduction

The Enterprise Integration Programme (EIP) was launched to replace the legacy integration systems also known as the Enterprise Service Bus (ESB). For more than 10 years, Eskom has utilised SUN Java CAPS and SeeBeyond 4.5.3 as the platform for intra-application communication, across multiple divisions and business areas. In 2010 Oracle acquired SUN and a directive was sent that stipulates that the SUN Java CAPS tooling would be discontinued. In November 2014 Eskom acquired Oracle’s Fusion Middleware (OFMW) suites as a replacement for its Seebeyond and JavaCaps platforms. The 1st phase of this program was to design and implement OFMW platform in central environment and this was completed in December 2015. The program is currently in the migrating the centrally hosted integration paths that are deployed on the legacy platform onto OFMW. In addition to the central environment some regional Operating Units (OU) also has legacy ESB implementations; these are extensions of the central legacy integration platforms. The primary objective of these implementations is to allow the central corporate environment to integrate with the regional operational technology environment (OT) environment. The purpose of this document is to describe the detailed EIP platform infrastructure architecture that is going to replace the above mentioned implementations with OFMW at the various regional OU sites. The document delves further into why there is a need for an integration platform at the regions followed by the scope of the design. Thereafter the EIP design pertaining to the various architecture domains including security will be detailed.

2 Background An Enterprise Service Bus (ESB) as shown in Figure 1: ESB Concept is an open standard, message-based, distributed integration infrastructure that provides routeing, invocation and mediation services that facilitate interactions of disparate distributed applications and services in a secure reliable manner. The ESB is usually implemented through service containers distributed across a networked environment. These containers host integration services like routers, transformers, application adapters and provide them with a range of communication facilities.

of 61

Figure 1: ESB Concept

The current ESB platform deployed at the regions allows for information integration between the applications deployed on the corporate networks, also known as the IT network, and applications deployed on regional OT networks at the various OUs. The following solutions leverage the existing regional ESB platform:

1. The SmallWorld system is responsible for managing the geographical data for network devices and substations. It assists in maintaining the data on SCADA by means of a Network Change Set message.

2. The Engineering Network Schematics (ENS) system resides between SmallWorld GDC and SENS and generates schematic representations of the Geographic Data within SmallWorld.

3. SENS (SCADA ENS) system adapts the ENS schematics into the SCADA

format and sends these diagrams to SCADA by means of Network Diagram Change-set message. SENS also requires Telemetry information from SCADA and this is sent in the form of SCADA Point Data Set, otherwise known as List of Values, messages from SCADA.

4. The Fault Management System (FMS) is responsible for fault management and

enables automated logging of network events. It performs network tracing and stores the trace data locally for future reference.

5. The Business Events Notifier (BEN) system is a web based real time notification system that creates alerts based on the Alarm Event messages received from SCADA. These alerts are sent out in the form of an email or SMS.

of 61

The approach the EIP project is taking with regards to the implementation is that the legacy platform will be maintained and operational until all the regional integration solutions are migrated over to the new EIP platform in a phased approach. Thus the interim architecture for a duration will have two platforms co-existing hosting various integration solutions. The architecture roadmap for this transition illustrates how the final integration architecture will manifest itself.

Figure 2 : Architecture Roadmap

3 Scope The scope of this design is the implementation of the regional ESB platforms which will be leveraged in the migration of existing or future integration solutions that require an ESB capability in the regions. It excludes actual migration of the solutions that are currently deployed in the regions. This will be initiated as a separate project whilst the integration platform implementation is being executed. Details of the current integration solutions are outlined in the Background . The design will encompass the following environments:

• Production for each region • Disaster Recovery (Swap out strategy) • Quality Assurance • Development

of 61

4 Constraints The following constrains were imposed on the EIP program

1) The design of the platform should not impact existing business processes. 2) The platform should perform in a functional like-for-like manner to the legacy

ESB as not to impact the implementation of end systems. 3) The legacy ESBs will only be decommissioned once all integration solutions have

been migrated to the OFMW.

5 Design Approach The preliminary phase of the design involved analyzing the existing deployment topology of the ESB platforms at the various regions. One fundamental finding was that the IT and OT environments were segregated using a demilitarized zone (DMZ) implementation. The purpose of this segregation is:

1. Decoupling – To shield either side from changes in implementation 2. Security – To prevent direct access to the OT network via the IT network

This segregation manifested itself via two fundamental network topology patterns depending on the site in question. For the rest of this document these patterns will be referred to as the Type A and Type B pattern. The Type A deployment pattern shown in Figure 3 : Type A Regional ESB Deploymentconsists of:

1. An ESB platform connected to the IT network deployed at an OU. The purpose of this platform is to extend the central integration IT footprint to the region.

2. Dual firewall DMZ, the purpose of this DMZ is to segregate the IT and OT network.

3. An ESB platform implementation deployed in the above mentioned DMZ , the purpose of this platform is extend the local IT integration platform footprint into the OT environment allowing for an indirect communication channel between the central and regional OT environment

of 61

Figure 3 : Type A Regional ESB Deployment The table below details the sites that conform to the Type A pattern Table 1: Type A Sites

SiteName TypeSimmerpan ABellville ABloemfontein A–ITOnlyMkondeni AEastLondon A The Type B pattern shown in Figure 4: consists of:

1. Two IT ESB platforms that are geographically isolated by being deployed at separate regional OUs. The purpose of these platforms is the same as the Type A IT deployment.

2. Dual firewall DMZ, the purpose of this DMZ is to segregate the IT and OT network. This DMZ is deployed at a 3rd regional OU separate that is separated to the above mentioned point 1 deployment.

3. An ESB platform deployed in the above mentioned DMZ. The purpose of this

platform is to allow for indirect connectivity between the single OT network and the two IT implementations mentioned in point 1

of 61

Figure 4: Type B Regional ESB Deployment

The table below details the sites that conform to the Type B design

Table 2: Type B Sites SiteName Type

Duvha B–DMZOnlyPolokwane B–ITonlyWitbank B–ITonly

Due to the constraints mentioned in section 4 we had to follow the current infrastructure deployment of Type A and Type B patterns when designing the new EIP ESB platform.

6 EIP Architecture Design The following sections detail the architecture design of the ESB platform that will be deployed per regional site and its functionality relative to the central ESB implementation. This is achieved by describing each of the TOGAF 9.1 standard architecture domains Business Information (Application) Data Technology (BIDAT) and the security architecture that applies to the EIP regional ESB platform.

6.1 Business Architecture

6.1.1 Business Strategies and Plans The Enterprise Integration Programme seeks to replace the legacy integration systems. The 2012 Group IT Business Plan - Enterprise Integration Programme initiative is unique and crucial to business success as it underpins the transport of key information in Eskom’s business, including the safety and revenue systems/application The current short –term objective of the EIP platform is to migrate all existing SeeBeyond and Java Caps integration interfaces to the new OFMW platform. The in

of 61

addition to this any new integration requirements will be developed on the EIP platform where applicable.

6.1.2 Business Processes and Policies The purpose of the platform as previously mentioned will impact the major EHPUM processes and policies due to the fact that it is the primary platform for transporting data between all major systems within the Enterprise.

6.1.3 Business Organisation Design The functional responsibility of the Integration Centre of Excellence (ICoE) is the deliver and maintain the IT systems integration within it the organisation . Hence the business owner of the EIP platform would be the ICoE. The figure below illustrates where the ICoE is placed relative to the rest of the enterprise. The support of the platform will be shared across multiple entities within Group IT in partnership with Oracle and MSA provider..

Figure 5 :ICoE Business Context

6.2 Business Information Architecture

6.2.1 Business Information Flow/Inventory TBC

6.2.2 Information Integrity and Custodianship The Table 3: Oracle Fusion Middleware Users, Groups and Roles define how user access can be restricted on the platform. Please note that these are not application level groups and roles but rather specifies how individuals can access platform from an operational perspective.

of 61

Table 3: Oracle Fusion Middleware Users, Groups and Roles Group Membership

Administrators By default, this group contains the user information entered as part of the installation process (that is, the Configuration Wizard), and the system user if the WebLogic Server instance is running Compatibility security. Any user assigned to the Administrators group is granted the Admin security role by default.

Deployers By default, this group is empty. Any user assigned to the Deployers group is granted the Deployer security role by default.

Operators By default, this group is empty. Any user assigned to the Operators group is granted the Operator security role by default.

Monitors By default, this group is empty. Any user assigned to the Monitors group is granted the Monitor security role by default.

AppTesters By default, this group is empty. Any user assigned to the AppTesters group is granted the AppTester security role by default.

CrossDomainConnectors By default, this group is empty. Any user assigned to the CrossDomainConnectors group is granted the CrossDomainConnector security role by default.

AdminChannelUsers By default, this group is empty. Any user assigned to the AdminChannelUsers group is granted the AdminChannelUser security role by default.

OracleSystemGroup By default, this group contains the user OracleSystemUser and is granted the OracleSystemRole role by default.

6.2.3 Information Access and Confidentiality The platform itself as previous mentioned is a means to transport data from one enterprise system to therefore minimal enterprise information that will be persisted on it. The only instances where information that would be persisted is,

1) To maintain state of an integration path 2) Error and audit logs

In both instances access to these data stores is restricted by the various out of the box OFM security measures. Data is either persisted in the file system or in data bases and both have restricted access by virtue of user name and password depending on the context of the data store. The section on the Error! Reference source not found. of the platform will detail this further

6.2.4 Information Related Business Continuity The EIP platform is classified as a safety and revenue critical system therefore,

1) It is a 24x7 systems 2) Recovery point objective of data being - TBC 3) Recovery time objective of – TBC

of 61

6.3 Data Architecture

6.3.1 Data Models Not applicable

6.4 Application Architecture

6.4.1 EIP Regional Platform Application Architecture Deployment All of Eskom’s IT integration requirements and development is managed by a single body, the ICoE (Integration Centre of Excellence), and as result of this that all integration requirements will follow the same governance and architecture standards. Taking this into account the EIP regional ESB deployment design follows a centralized federated integration topology as shown in Figure 6. Centralized Federation Topology is a specific setup in which the ESBs are deployed at the regions using a well-defined hierarchical relationship.

Figure 6: Centralized Federation Topology

The deployment consists of :

1) Centrally hosted ESB which acts as the master which has already been deployed

2) Two types of regional deployment patterns

of 61

a. Two Scaled down instance of the master bus region deployed at each of the regions IT and DMZ environments as slaves consistent with the current deployment pattern as in Figure 3 : Type A Regional ESB Deployment.

b. A scaled down instance of the master bus deployed in the DMZ and two scaled downs instances deployed in the IT environment consistent with the current Figure 4: Type B Regional ESB Deployment.

This topology requires all the regional or federated Service Buses to route the requests to one another via the central backbone Master Service Bus. One key assumption is that the regional integration instances currently do not have a need to communicate directly with each other. The rationale for this design is,

1) Central service bus acts as the integration mediator and orchestrator between the end systems deployed in central environment and the regional OT systems.

2) The regional services buses act as integration service intermediaries to the regional OT systems. This means that communication to the OT systems will be routed via these regional instances.

3) The regional service bus composition is further decomposed into two instance per region because

a. Direct access to the OT environment is not allowed from a security perspective

b. Integration buffering capability on the IT and OT environments. The capability will buffer traffic in the event that either the IT or OT network is disrupted any reason.

The master Service Bus has already been deployed as part of the 1st phase of the EIP project. This system is currently hosted in MWP on the Oracle Engineered Systems hardware. The 2nd phase of the project which is the focus of this design is the deployment architecture the regional slave service bus implementations. The following section will detail the fictional capability provided by each instance of the regional ESBs’

6.4.2 EIP Application Functional Decomposition Each instance of the regional service bus will be implemented using the Eskom standard OFMW .The OFMW suite is a purpose built ESB platform that provides industry standard ESB functionality to the enterprise. OFMW was chosen as the ESB standard in Eskom in March 2015 and was implemented in the central environment. In keeping with the central integration standard the regional integration platforms is also implemented using OFMW.

of 61

This suite gives the enterprise the capability to develop different types of middleware applications. This achieved by leveraging components such as,

• Business Process Management • BPEL process manager • Proxy • Technology and vendor adapters (connectors) • Enterprise Service Repository • Web Services Manager • Business Activity Monitoring • Could Management Console

The platform is built on top of the Oracle J2EE application server Weblogic and uses this as its runtime environment. In addition to this Weblogic provides the following functionality,

• Clustering • Performance Monitoring • Diagnostics • Java Messaging Services (asynchronous integration) • Diagnostics • Caching Frameworks

The Enterprise embarked on an unlimited user licences (ULA) agreement with Oracle which allows the organisation to have an unlimited number of instances of a particular application contained in the Oracle ULA. The Figure 7: OFM Suite Decomposition below defines the middleware applications that are covered by the ULA and are deployed by the EIP.

Figure 7: OFM Suite Decomposition

of 61

The key solution building blocks that define the define middleware application architecture for the regions is defined below,

1. Oracle SOA Suite

o ORACLE SERVICE BUS (OSB)

Oracle Service Bus is an enterprise service bus (ESB) that provides a virtualization layer required for any integration architecture. Using Service Bus, organizations can decouple service consumers from changes that might occur in the backend. They can also hide from developers the often intricate and complex details of underlying implementations of back-end applications, such as legacy protocols

o BUSINESS PROCESS EXECUTION LANGUAGE (BPEL) PROCESS

MANGER – (BPEL-PM)

Oracle BPEL provides a set of discrete services into an end-to-end process flow, reducing the cost and complexity of process integration. It executes standard BPEL processes and provides a “dehydration” capability so that the state of long-running flows is automatically maintained in a database, enabling clustering for both fail-over and scalability. The built-in human workflow capabilities of Oracle SOA Suite allow for people to be included in these processes for approvals and reviews

o CONNECTORS

Oracle SOA Suite provides a connectivity layer, enabling connectivity to a data source inside as well as outside the enterprise. Oracle Adapters (Database, JMS, file etc.) are available for on-premise applications and technology. In addition, B2B & Managed File Transfer capabilities are included to extend processes to external business partners.

2. Oracle Web Service Manager (OWSM) Oracle Web Service Manager provides the first line security for client agent and last line security via server agents. Whether services are accessed within the enterprise or externally they may require authentication and authorization in accordance with the organization’s security policy or regulatory compliance. OWSM is centralized, declarative, externalized and consistent.

3. Weblogic Server

Oracle WebLogic Server is a Java Platform, Enterprise Edition (Java EE) application server. The WebLogic Server infrastructure supports Oracle Fusion Middleware and other applications and is a foundation for building SOA based applications.

4. Oracle Enterprise Manager (OEM) Cloud Control

of 61

Oracle Enterprise Manager provides visibility into your application servers and their resident applications. Oracle Enterprise Manager and the associated SOA Management Pack plug-in provide these capabilities in an easy to use web console. Using Oracle Enterprise Manager, you can monitor your running servers, applications and service engines: to facilitate trouble-shooting at runtime within your enterprise SOA environment. The SOA Management Packs for Oracle Enterprise Manager 12c introduce the Java VM Diagnostics as a Service capability, which allows applications and middleware administrators to provide Java VM diagnostics capabilities directly to developers and QA engineers on an as needed basis. Users are provisioned automatically and receive their own self-service portal for accessing diagnostics capabilities. Oracle Enterprise Manager does more than provide visibility into your enterprise SOA environment: it also works with Oracle Web Services Manager to allow you to define security policies for your services and components and to apply those security policies as needed. This separates security management from application development, a best practice in the security world. This allows you to evolve and implement your security strategy outside of application development, providing you with greater agility and flexibility

5. SOA Management Pack

SOA Management Pack with Oracle Enterprise Manager Cloud Control facilitates monitoring of Oracle SOA Suite and OSB. It provides administrators of the SOA environment with a consolidated browser-based view of the entire enterprise. This enables administrators to monitor and manage all of their components from a central location. It stores the collected metric and configuration data in a central repository, thereby enabling administrators to analyse metrics through various historical views and facilitate strategic trend analysis and reporting.

6. WebLogic Management Pack

WebLogic Server Management Pack Enterprise Edition greatly improves server as well as application performance by providing unique functionality to automatically detect performance bottlenecks; quickly diagnose these performance problems, and identify their root cause.

WebLogic Server Management Pack Enterprise Edition provides common administration operations - traditionally available from the Oracle Enterprise Manager Fusion Middleware Control console or the WebLogic Server Administration Console – directly from the Cloud Control console. Consequently, a single console can be used to centrally administer multiple domains.

The for each regional service bus instance the recommended deployment model was to expose a single central Enterprise Service Bus (ESB) with multiple SOA Domains. These logical domains will be used to containerize integration components that require some form of state to be maintained, and it will be split on non-functional requirements such as volumes, payload sizes and transaction times, as shown in Figure 8: SOA Federation

of 61

Figure 8: SOA Federation The platform is architecturally comprised of 4 tiers,

1. Application tier – Systems that leverage that integration platform for intra-application communication

2. Middle tier – The integration components are deployed on this tier 3. Monitoring tier- Oracle Enterprise Manager will act as a central monitoring and

management tool for all the databases and middleware components deployed 4. Data tier - Oracle Fusion Middleware is reliant on the database as it uses it to

manage state The above mentioned is depicted in Figure 9 : EIP Platform Tiring

of 61

Figure 9 : EIP Platform Tiring

6.4.2.1 Regional Middleware Tier Domains and Clustering Structure

A domain is an interrelated set of WebLogic Server resources that are managed as a unit. A domain also contains the application components deployed in the domain, and the resources and services required by those application components and the server instances in the domain. In each domain, one WebLogic Server instance acts as the Administration Server - the server instance which configures, manages, and monitors all other server instances and resources in the domain. Each Administration Server manages one domain only

WebLogic Server clusters provide scalability and reliability by distributing the work load among multiple instances of WebLogic Server. Incoming requests can be routed to a WebLogic Server instance in the cluster based on the volume of work being processed. In case of hardware or other failures, session state is available to other cluster nodes that can resume the work of the failed node. Domains can contain both managed servers and clusters.

The server instances that constitute a cluster can run on the same machine, or be located on different machines. You can increase a cluster's capacity by adding additional server instances to the cluster on an existing machine, or you can add machines to the cluster to host the incremental server instances.

of 61

All server instances in a cluster must reside in the same domain. Clustered WebLogic Server instances behave similarly to non-clustered instances, except that they provide failover and load balancing. The process and tools used to configure clustered WebLogic Server instances are the same as those used to configure non-clustered instances.

Based on the Federated SOA Domain architecture, two Traffic Director and one Service Bus will be deployed with multiple SOA Domains, each optimized for specific non-functional requirements. These domains will be supplemented by a JMS domain to host all queues and topics. Each of the middleware tier components are segregated into Weblogic domains. Each of these domains are specialized for its intended purpose of the products deployed in it. The regional ESB domains shown in Figure 10 are:

• OTD – The OTD domain is specialized for the purposes of Oracle Traffic Director. This will be used for load balancing within the ODA system as well as fronting incoming communications.

• OSB – The OSB domain is specialized for Oracle Service Bus. Oracle Service Bus is used for simple mediation, routing and transformation that requires no orchestration.

• SOA – The SOA domains are specialized for SOA Suite. SOA Suite is BPEL-based and hence process-aware. The SOA Suite functionality has been subdivided into two domains optimized for different purposes:

o SOA 1 - Optimized for small payload, high volume and low latency interfaces.

o SOA 2 - Optimized for large payloads where streaming will enhance performance and increase stability.

• JMS – The JMS domain will be optimized for supporting JMS servers.

of 61

ODA

IT DMZ

OTD

OSB

SOA1 SOA2

JMS

SOADB JMSDB CSMDB

ODA

OTD

OSB

SOA1 SOA2

JMS

SOADB JMSDB

Figure 10: EIP Domain Structure for the IT and DMZ instance at each site

The Table 4 : Summary of EIP Domains summarizes the domains that will be deployed, and the intended use for each domain

Table 4 : Summary of EIP Domains

Domain Components Deployed Usage SOA Suite 1 • SOA Suite

• Web Services Manager • Vanilla WebLogic

Cluster for JEE

Domain will be optimised for small payload, high volume and low latency interfaces

SOA Suite 2 • SOA Suite • Web Services Manager • Vanilla WebLogic

Cluster for JEE

Domain will be optimised for large payloads where streaming will enhance performance and increase stability

OSB • OSB • Web Services Manager

Mediation, Routing and Transformation

JMS • JMS Asynchronous messaging OTD • OTD Used for load balancing

within the ODA system.

6.4.2.1.1 OTD Domain An instance of Oracle Traffic Director will be deployed to handle internal communication and inbound load balancing requirements. The instance will be clustered across two compute nodes for HA purposes

6.4.2.1.2 OSB Domain

of 61

OSB will be the entry point for all synchronous integrations within each environment, and as such it will be clustered across four vServers to start off with. Capacity can easily be increased by scaling the domain horizontally should the need arise. Oracle Web Services Manager (OWSM / WSM) will be deployed as part of the domain in order to enable the service bus to apply security policies to inbound and outbound services. OWSM will be deployed on separate managed servers within a separate cluster as per the recommended approach. This reduces the load on the OSB servers when security has to be applied. The admin server can also be failed over to each of the vServers due to the u01 shared storage that will be hosting the domain and admin server. The following sections will detail the clusters and managed servers will be configured for the OSB domain.

6.4.2.1.3 Oracle SOA Suite Domains As detailed in the overview of the Application Architecture, multiple SOA domains will be deployed split on non-functional requirements. This allows for specific SOA-INFRA and JVM optimizations to ensure solutions run optimally within the SOA domains.

The following table defines the decision making Table 5 : SOA Domain decision matrix should be used as input when deciding which domain will host a specific solution

Table 5 : SOA Domain decision matrix

Domain Payload Size > 5120 KB

Throughput > 10ms/sec

Default

prdsoad100 X X prdsoad200 X

SOA Domain 2 will be extended with Enterprise Scheduling Services and Managed File Transfer components in order to future proof the domain architecture. Apart from the above mentioned components, the domains will have the same managed server and component configurations. Ports will differ between the two domains to simplify the OTD load balancing algorithms. Again the admin server can be failed over between the vServers for a domain due to the domain and admin server running from /u01 shared storage. The following section will detail the domains, clusters and managed servers deployed for each SOA domains.

6.4.2.1.4 JMS Domain The JMS domain will be responsible for all asynchronous messaging within the integration platform. Two JMS clusters will be deployed within the domain, one dedicated to functional, business impacting messages and the other to non-functional, less critical messages like “error & audit” messages. From a JMS persistence perspective, data tier will be used.

of 61

Domain level JMS configuration is the highest level of configuration in terms of a JMS with regards to WebLogic. The domain level configuration are broader arching and are categorized as either environmental-related or application-related. Environmental configuration examples are definitions and identification of the JMS servers and data sources, persistent stores and network addresses, JMS Modules, Bridging and Store-And-Forward. These JMS configurations are stored as modules defined in XML similar to standard J2EE and can be deployed as J2EE managed modules or standalone modules within a domain. These specific configurations usually differ between domains.

JMS Servers are targeted at a specific managed server, and all managed servers within a cluster share availability of managed resources, in turn making them available to all deployments within that cluster.

Different JMS Servers are targeted at different Managed Servers. These are also managed from the WebLogic administrator console. JMS Servers are targeted at specific clusters, making them available to all deployments within that cluster.

JMS Modules are the configuration hub for lower level JMS items such as Connection Factories, Queues and Topics and their JNDI contexts. The JMS Modules are all responsible for their security at a module level, which means each module can have specific roles and security policies depending on requirements.

There are different types of Queues and Topics configurations available on WebLogic, such as standard Queues (specific to a module) or a Distributed Queue (which is uniform across a cluster)

Queues and Topics have the lowest level of configuration in the hierarchy, but the most specific. You can configure things such as thresholds and quotas, overrides in terms of delivery, security and control.

6.4.2.2 Data Tier Domain and Cluster Structure

Database tier currently has two functions

1. Maintain state for the OFM components 2. Non-functional integration components such as Error and Audit

The database version Oracle 12.1.0.2.0 will be installed for Eskom. The Table 6: EIP Databases below define the databases that will be deployed per environment. The following Fusion Middleware components interact with the database layer:

• OWSM connect to the database to access the OWSM Repository. The repository stores OWSM metadata, such as policies, policy sets, assertions templates, and policy usage data.

• Managed File Transfer (MFT) stores configuration data in an Oracle Metadata Repository. You can edit, back up, and restore this configuration data

• Oracle Business Process Management (BPM) stores process instance data in a database schema called "SOAINFRA"

• SOA Suite is using the database to store state information.

of 61

• For Oracle Service Bus (OSB) most of the internal data-structures that are required by OSB are stored in-memory. Only reporting functionality of OSB requires DB tables to be accessible.

• Weblogic Java Messaging Server (JMS) uses persistent storage (JDBC-accessible database) for storing persistent message data

Table 6: EIP Databases Database Name Database

Description Service Name State

SOA Database will host all the Fusion Middleware Schemas

OSB_Service Active SOA_Service

JMS Used as JMS persistence store.

JMS_Service Active

CSM Application database Change Set Management

CSM_Service Active

6.4.3 Application Strategy Please refer to Business Strategies and Plans for the migration strategy

6.4.4 Application Development Oracle J Developer will be used as the platform to build OFM components that are going to be deployed in the EIP environments.

6.5 Integration Architecture

6.5.1 Integration Interface Map

of 61

Figure 11 : System Integration Map The following Enterprise Systems have been integrated with EIP to deliver the required platform functionality

1) F5 – Load balancing traffic into the OFM environment 2) OEM – Monitoring of the fusion environment 3) netIQ LDAP – Users authentication 4) DNS – Hostname resolution 5) EIP – Is the master service bus already implemented in the central IT

environment 6) Regional EIP Instances – IT/OT ESB instances deployed at the various sites

6.5.2 Integration Services Directory The current 250 candidate interfaces of EIP will be migrated

6.5.3 Data Migration No data will be migrated from the any on the existing databases

of 61

6.6 Technical Architecture

6.6.1 Basic Infrastructure In following with the Eskom Hardware standard for EIP the infrastructure that was chosen to host OFMW was the Oracle Data Appliance (ODA). Essentially this appliance is a scaled down version of the engineered system deployed in the central environment. It is a self-contained appliance that encompasses both the application and storage tier. The environments that are in the scope of this implementation are:

1) Production 2) QA 3) Dev 4) Swap out DR

• Production

The following deployment pattern will be applied for the production environment in the regions:

• Type A region

1) One (1) ODA in the regional IT network 2) One (1) ODA in the DMZ

• Type B region

1) One (1) ODA in site 1 IT network 2) One (1) ODA in site 2 IT network 3) Two (2) ODAs in site 2 DMZ – the second ODA is to accommodate for the

eventual OT fragmentation of this site

• Non-Production

The following deployment pattern will be applied for the non-prod environment in the regions:

1) One (1) ODA in141 to simulate the IT regional environment 2) One (1) ODA in EAL to simulate the OT regional environment

• Disaster Recovery

The DR strategy is to have standby ODAs that will be used to swap any regional units that experiences a DR scenario. The rationale behind this is that currently the network work infrastructure between the sites and the central DR environment was deemed not sufficient to implement a centralised DR solution.

of 61

6.6.1.1 Hardware Virtualisation on EIP

• ODA Virtualization

Multiple layers will be deployed while provisioning Oracle Fusion Middleware on the Oracle engineered systems. From a deployment architecture perspective, Oracle Fusion Middleware will be deployed on the top layer (Virtual machine or vServer) of the hardware virtualization layer. The Figure 12 : EIP Hardware Virtualization Structure details the relationship between the different layers on the ODA machines

Figure 12 : EIP Hardware Virtualization Structure

Each ODA comprises multiple compute nodes, similar to blade servers, each with CPU’s, Memory and Network interfaces. OVM will be deployed on each of the compute nodes to act as the hypervisor layer where the vServers or Virtual Machines, running Oracle Enterprise Linux or OEL, will be hosted on. Hardware is the only limiting factor when determining how many vServers can be deployed, with the resources assigned to all the As previously mentioned both the application and database tier will be deployed on a single ODA instance per site. This is achieved by dividing the ODA platform into multiple virtualization domains (this is not the same as Weblogic domains) as shown in the Figure 13. Each of the 2 nodes consists of the following:

• DOM0 (Domain 0) • ODA-BASE (Domain 1) • Multiple DOMU’s (Domain U or User Domain)

DOM0 is the management domain or also known as the host domain. DOM0 is started by the hypervisor on boot. This is a privileged domain that starts and manages all other domains. DOM0 only has access to the local storage (350 GB free space) on each node.

of 61

The ODA_BASE is a privileged virtual machine domain, specifically for hosting the Oracle Grid Infrastructure and Oracle Databases. The ODA_BASE domain contains Oracle Appliance Manager, which includes commands and utilities to manage virtual machines, virtual disk, and templates. The ODA_BASE domain is a hardware virtualized, with par virtualized drivers (PVHVM) domain running an unmodified Unbreakable Enterprise Kernel (UEK). DOMU is for Virtual machines which are provisioned to host non-database workloads such as applications and middleware.

Figure 13: ODA Virtualization

6.6.1.2 EIP Central Hardware Specifications Table 7: ODA Hardware Specification and Utilization

Environment Production CPUs 64 cores, 89% used

Storage 64 TB (double-mirrored), 10% used MemoryOFM 208 GB, 41% used

Environment Disaster Recovery

CPUs 64 cores, 89% used Storage 64 TB (double-mirrored), 10% used

MemoryOFM 208 GB, 41% used

Environment Non-Production CPUs 48 cores, 67% used

Storage 64 TB (double-mirrored), 10% used MemoryOFM 224 GB, 44% used

of 61

6.6.1.3 Physical Deployment Architecture

6.6.1.3.1 Production The following section will detail the Oracle Fusion Middleware deployment architecture for Production. From a domain architecture perspective, the same Oracle Fusion Middleware deployment template will be applied to both the IT and DMZ environment.The

ODA

ComputeNode ComputeNode

DOM1 VM

VM VM

VM

VM

VM VM

DOM1VM

VM VM

VM

VM

VM VM

OracleTrafficDirector

OracleServiceBus

OracleSOASuite

OracleSOASuite

JMS

DOM0 DOM0DatabaseRAC

Figure 14: High-Level Production Middle Tier Domain Deployment illustrates how the domains are configured on the production regional ODAs

of 61

ODA


DOM1 VM

VM VM

VM

VM

VM VM

DOM1VM

VM VM

VM

VM

VM VM

OracleTrafficDirector

OracleServiceBus

OracleSOASuite

OracleSOASuite

JMS

DOM0 DOM0DatabaseRAC

Figure 14: High-Level Production Middle Tier Domain Deployment

The diagram below depicts the database architecture deployed for the Database 12c Production environment. The ODA in each region on the OT and IT network is 2 Nodes RAC.

of 61

Figure 15: EIP Data Tier Domain Deployment

From a deployment perspective, three different mount points will be created;

• /u01 – To host all installation binaries as well as the domain and admin server configuration files. This mount point will be shared across all the vServers for a specific domain.

• /u02 – This will host all the managed server structures. This is separated from the domain structure in /u01 to facilitate library isolation and to create a split between admin and managed servers.

• /u03 – Dedicated to logging. This enables a central location for all managed server log files on a vServer and reduces the operational overhead of having to navigate to each managed server to view log files.

The following table displays a sample file system structure for a SOA domain. Table 8: Middle tier mount points Mount Point Sample Path Comments /u01 /u01/oracle/products/….. Binary installation path /u01 /u01/oracle/config/domains/prdsoad100/… Domain and admin

server location /u02 /u02/oracle/config/domains/prdsoad100/… Domain location for

managed servers /u03 /u03/oracle/logs Central logging location

of 61

for all managed servers

• OTD Deployment

ODA

ComputeNode1 ComputeNode2

vServer vServer

OTD

OTD1

/u02 /u03

OTD2

/u02 /u03

/u01

AdminNode AdminNode

Figure 16 : OTD Production Deployment

Table 9 : OTD Specifications for managed servers and clusters Environment vServer #CPU’s Memory u01SharedStorage u02 u03Production <reg>odavm<num> 4 16GB

250GB50GB 100GB

<reg>odavm<num> 4 16GB 50GB 100GB

• OSB Deployment

of 61

ODA


vServer vServer vServer vServer

OSBDomain

AdminServer(7001)

WLS_OSB1

WLS_WSM1

/u02 /u03

AdminServer(7001)

WLS_OSB2

WLS_WSM2

/u02 /u03

/u01

AdminServer(7001)

WLS_OSB3

WLS_WSM3

/u02 /u03

AdminServer(7001)

WLS_OSB4

WLS_WSM4

/u02 /u03

Figure 17: OSB Production Deployment Table 10: OSB Specifications for managed servers and clusters Environment Domain Cluster Managed

ServersMemory

Production prdosbd100 prdosbcl100 prdosbms101 4GBprdosbms102 4GBprdosbms103 4GBprdosbms104 4GB

prdwsmcl100 prdwsmms101 4GBprdwsmms102 4GBprdwsmms103 4GBprdwsmms104 4GB

Environment vServer #CPU’s Memory u01SharedStorage u02 u03Production <reg>odavm<num> 4 12GB

50GB50GB 250GB

<reg>odavm<num> 4 12GB 50GB 250GB<reg>odavm<num> 4 12GB 50GB 250GB<reg>odavm<num> 4 12GB 50GB 250GB

of 61

• SOA Suite Deployment

ODA


vServervServer

SOADomain1

AdminServer(7001)

WLS_SOA1

WLS_WSM1

/u02 /u03

AdminServer(7001)

WLS_SOA2

WLS_WSM2

/u02 /u03

WLS_EE1 WLS_EE2

/u01

vServervServer

SOADomain2

AdminServer(7001)

WLS_SOA1

WLS_WSM1

/u02 /u03

AdminServer(7001)

WLS_SOA2

WLS_WSM2

/u02 /u03

WLS_EE1 WLS_EE2

/u01

WLS_ESS1

WLS_MFT1

WLS_ESS2

WLS_MFT2

Figure 18: SOA Suite Production Deployment

Table 11 :SOA Suite Specifications for managed servers and clusters Environment Domain Cluster

NameManagedServers

Memory

Production prdsoad100 prdsoacl100 prdsoams101 4GBprdsoams102 4GB

prdwsmcl100 prdwsmms101 4GBprdwsmms102 4GB

prdeecl100 prdeems101 4GBprdeems102 4GB

prdsoad200 prdsoacl200 prdsoams201 4GBprdsoams202 4GB

prdwsmcl200 prdwsmms201 4GB

of 61

prdwsmms202 4GBprdeecl200 prdeems201 4GB

prdeems202 4GB Table 12: Coherence Specifications for managed servers and clusters Environment Domain ClusterName Clustering

ModeTransport

prdsoad100 prdcoherencecl200 Unicast UDP

prdsoad200 prdcoherencecl200 Unicast UDP

Environment vServer #CPU’s Memory u01SharedStorage u02 u03 <reg>odavm<num> 6 16GB 50GB

50GB 250GB

<reg>odavm<num> 6 16GB 50GB 250GB<reg>odavm<num> 6 16GB 50GB

50GB 250GB


• JMS deployment


vServer vServer

AdminServer(7001)

WLS_JMS1

/u02 /u03

AdminServer(7001)

WLS_JMS2

/u02 /u03

ODA


vServer vServer vServer vServer

JMSDomain

AdminServer(7001)

WLS_JMS1

/u02 /u03

AdminServer(7001)

WLS_JMS2

/u02 /u03

/u01

AdminServer(7001)

WLS_JMS3

/u02 /u03

AdminServer(7001)

WLS_JMS4

/u02 /u03

Figure 19: JMS Production Deployment

of 61

Table 13: JMS Specifications for managed servers and clusters Environment Domain Cluster

NameManagedServers

Memory

Production prdjmsd100 prdjmscl100 prdjmsms101 4GBprdjmsms102 4GB

prdjmscl200 prdjmsms103 4GBprdjmsms104 4GB

Environment vServer #CPU’s Memory u01SharedStorage u02 u03Production <reg>odavm<num> 2 8GB

50GB

50GB 250GB<reg>odavm<num> 2 8GB 50GB 250GB<reg>odavm<num> 2 8GB 50GB 250GB<reg>odavm<num> 2 8GB 50GB 250GB

6.6.1.3.2 Non-Production The following section will detail the Oracle Fusion Middleware deployment architecture for non-production environments, where an ODA will be deployed for each of the IT & DMZ environments respectively. From a domain architecture perspective, Development and QA will be clustered and identical from a deployment perspective, with only the virtual machine resources differing for each of the environments as shown in Figure 20 .

ODA


ODA


DEV

QA

DEV

QA

Figure 20 :High-Level Non Production Middle Tier Domain Deployment

of 61

Figure 21 : Non-Production Database Architecture In Figure 21 : Non-Production Database Architecture the clients will connect to the databases using services, the services will be running on the Node with the green color and it will be standby on the node with the orange color; This approach to maintain high availability and load balancing among all the DB nodes in the Oracle RAC. In Non-Production Exadata we have 3 environments Preprod, Development and QA We will have a separate database for each environment for the database PIEP and PINT and one database for SOA, OSB, JMS for each environment Database PRESOA for SOA, OSB and JMS for preprod environment Database DEVSOA for SOA, OSB and JMS for development environment Database QASOA for SOA, OSB and JMS for QA environment The following sections detail the Non-Production physical deployment and specifications of the domains covered in Regional Middleware Tier Domains and Clustering Structure

• OTD Deployment

of 61

ODA


vServer vServer

OTD

OTD1

/u02 /u03

OTD2

/u02 /u03

/u01

Figure 22 :Oracle Traffic Director – Non-Production

An instance of Oracle Traffic Director will be deployed to handle internal communication and inbound load balancing requirements. The instance will be clustered across two compute nodes for HA purposes in the Development and QA environments. The following table details the resource allocation.

Table 14 : OTD Non-prod Environment Specifications Environment vServer #CPU’s Memory u01SharedStorage u02 u03QA <reg>odavm<num> 2 8GB

250GB50GB 100GB

<reg>odavm<num> 2 8GB 50GB 100GBDevelopment <reg>odavm<num> 2 8GB

250GB50GB 100GB


• OSB Deployment

of 61

ODA


vServer vServer

OSBDomain

AdminServer(7001)

WLS_OSB1

WLS_WSM1

/u02 /u03

AdminServer(7001)

WLS_OSB2

WLS_WSM2

/u02 /u03

/u01

Figure 23: Oracle Service Bus – Non-Production

Table 15 : OSB Managed Servers and Clusters Environment Domain Cluster Managed

ServersMemory

QA qaosbd100 qaosbcl100 qaosbms101 4GBqaosbms102 4GB

qawsmcl100 qawsmms101 4GBqawsmms102 4GB

Development devosbd100 devosbcl100 devosbms101 4GBdevosbms102 4GB

devwsmcl100 devwsmms101 4GBdevwsmms102 4GB

Environment vServer #CPU’s Memory u01SharedStorage u02 u03

of 61

QA <reg>odavm<num> 2 8GB 50GB

50GB 250GB<reg>odavm<num> 2 8GB 50GB 250GB

Development <reg>odavm<num> 2 8GB 50GB 50GB 250GB<reg>odavm<num> 2 8GB 50GB 250GB

• SOA Suite Deployment

ODA


vServervServer

SOADomain1

AdminServer(7001)

WLS_SOA1

WLS_WSM1

/u02 /u03

AdminServer(7001)

WLS_SOA2

WLS_WSM2

/u02 /u03

WLS_EE1 WLS_EE2

/u01

vServervServer

SOADomain2

AdminServer(7001)

WLS_SOA1

WLS_WSM1

/u02 /u03

AdminServer(7001)

WLS_SOA2

WLS_WSM2

/u02 /u03

WLS_EE1 WLS_EE2

/u01

WLS_ESS1

WLS_MFT1

WLS_ESS2

WLS_MFT2

Figure 24 : SOA Suite Deployment

Table 16: SOA Suite Managed Servers and Clusters Specification

of 61

Environment Domain ClusterName

ManagedServers

Memory

QA qasoad100 qasoacl100 qasoams101 4GBqasoams102 4GB


qaeecl100 qaeems101 4GBqaeems102 4GB

qasoad200 qasoacl200 qasoams201 4GBqasoams202 4GB


qaeecl200 qaeems201 4GBqaeems202 4GB

Development devsoad100 devsoacl100 devsoams101 4GBdevsoams102 4GB


deveecl100 deveems101 4GBdeveems102 4GB

devsoad200 devsoacl200 devsoams201 4GBdevsoams202 4GB


deveecl200 deveems201 4GBdeveems202 4GB

Environment Domain ClusterName Members Clustering

ModeTransport

QA qasoad100 qacoherencecl200 qasoacl201qawsmcl201qaeecl201

Unicast UDP

qasoad200 qacoherencecl200 qasoacl201qawsmcl201qaeecl201

Unicast UDP

Development devsoad100 devcoherencecl200 devsoacl201devwsmcl201deveecl201

Unicast UDP

devsoad200 devcoherencecl200 devsoacl201devwsmcl201deveecl201

Unicast UDP

Table 17 :Coherence Managed servers and Cluster Specification

of 61

Environment Domain ClusterName Members ClusteringMode

Transport

QA qasoad100 qacoherencecl200 qasoacl201qawsmcl201qaeecl201

Unicast UDP

qasoad200 qacoherencecl200 qasoacl201qawsmcl201qaeecl201

Unicast UDP

Development devsoad100 devcoherencecl200 devsoacl201devwsmcl201deveecl201

Unicast UDP

devsoad200 devcoherencecl200 devsoacl201devwsmcl201deveecl201

Unicast UDP

Environment vServer #CPU’s Memory u01SharedStorage u02 u03QA <reg>odavm<num> 4 16GB 50GB

50GB 250GB


50GB 250GB

<reg>odavm<num> 4 16GB 50GB 250GBDevelopment <reg>odavm<num> 2 16GB 50GB

50GB 250GB


50GB 250GB


• JMS Domain

of 61

ComputeNode1

vServer

AdminServer(7001)

WLS_JMS1

/u02 /u03

ODA


vServer vServer

JMSDomain

AdminServer(7001)

WLS_JMS1

/u02 /u03

AdminServer(7001)

WLS_JMS2

/u02 /u03

/u01

Figure 25: JMS Domain

Table 18: JMS managed servers and clusters specification Environment Domain Cluster

NameManagedServers

Memory

QA qajmsd100 qajmscl100 qajmsms101 4GBqajmscl200 qajmsms102 4GB

Development devjmsd100 devjmscl100 devjmsms101 4GBdevjmscl200 devjmsms102 4GBdevjmscl200

Environment Domain Cluster

NameManagedServers

Memory

QA qajmsd100 qajmscl100 qajmsms101 4GBqajmscl200 qajmsms102 4GB

of 61

Development devjmsd100 devjmscl100 devjmsms101 4GBdevjmscl200 devjmsms102 4GBdevjmscl200

6.7 Network Architecture This section provides an overview of the:

1) Firewall restrictions between the OT, DMZ and IT networks

2) Identifies Application network requirements

3) IP Addresses

4) Firewall Access

6.7.1 IT & DMZ Restrictions The diagram below depicts the firewall design which is the pattern for all regional DMS sites:

• ThereareanumberofVirtualLocalAreaNetworks(VLANs):o ESKOMBUSLAN(EskomIT/CorporateNetwork)o DMZ_MGMT_LANVLAN40(DMZManagementNetwork)o DMZ_SERVICESVLANXX(DMZNetwork)o DMZLAN(DMZNetwork)

• Thesenetworksareseparated/isolatedbyafirewall(CHECK_PRI)

Figure 26: Regional Network/Firewall Design (pattern for all DMS sites)

To access a system/platform/service in another network, a rule must be added on the firewall. This rule must clearly indicate the address of the source and target system as well as the specific ports and network protocol which will be used

of 61

As noted above, a core Security requirement is that a physical server cannot be connected to both IT and DMZ networks. For this reason, there will be two Oracle Database Appliances (ODAs) deployed to a region – one allocated to the Corporate/IT network and one which will reside in the DMZ network.

of 61

6.7.2 ODA Placement

IT Network Figure 27 below depicts the network connectivity for the Oracle Database Appliance deployed to the regional IT/Corporate network. Both the management interfaces

Figure 27: ODA Placement in IT Network

DMZ Network

Figure 28 below depicts the network connectivity for the Oracle Database Appliance deployed to the regional DMZ network. The dedicated management network ports on the ODA will be connected to the management network to allow remote administration and management of the machine.

Figure 28: ODA Placement in DMZ Network

of 61

Type A Connectivity The diagram below illustrates the network connectivity between the Central and Regional Integration Platforms for a “Type A” site. As the reader will note:

• Connectivity between central and the regional site is achieved via the Eskom Wide Area Network (WAN)

• Connectivity between IT, DMZ and OT platforms is routed via a firewall (please refer to Section 6.7.1)

Figure 29: Central - Regional Network Connectivity (Type A)

Type B Connectivity The diagram below illustrates the network connectivity between the Central and Regional Integration Platforms for a “Type A” site. As the reader will note:

• Connectivity between central and the regional sites is achieved via the Eskom Wide Area Network (WAN)

• Connectivity between regional IT and regional OT sites is also achieved via the Eskom WAN

• Connectivity between IT, DMZ and OT platforms is routed via a firewall (please refer to Section 6.7.1)

Figure 30: Central - Regional Network Connectivity (Type B)

of 61

6.7.3 Regional Network Capability The table below provides a summary of the regional IT and DMZ/OT network capability:

IT DMZ/OT

Fiber Copper Fiber Copper Site 1Gb 10Gb 1Gb 10Gb Simmerpan 8x 10Gb LC available Yes None No UTP in

place None

Bellvile 10g available Yes None No UTP in place

None

Bloemfontein 10g avaialble Yes None No UTP in place

None

Mkondeni 8x 10Gb LC available Yes None No UTP in place

None

Polokwane To be confirmed Yes None No UTP in place

None

Witbank 8x 10Gb LC available Yes None No UTP in place

None

East London 8x 10Gb LC available Yes None No UTP in place

None

EAL No 10 Gig available only 1 gig utp/fiber

Yes None No UTP in place

None

Duvha TBC TBC TBC TBC TBC TBC 141 TBC TBC TBC TBC TBC TBC

6.7.4 Load Balancing OTD acts as the load balancer between client systems and application instances on the ODA. Data traffic between application instances will be performed on the internal private application network only. The backend network will be used for administrative purposes (OS configurations, management of applications etc.) Advantages of this design:

• Inter application traffic will stay within the ODA system directly on the Infiniband network. This traffic will benefit from lower latencies and higher throughput.

• Reduced complexity thru reduced number of network interfaces in application VMs.

• Security • Segregation of network traffic

6.7.4.1 OTD Oracle Traffic Director will be used to load-balance all HTTP or HTTPS traffic internally between all the fusion middleware components and domains as well as to front incoming traffic. One OTD instance, configured for high availability with failover between instance

of 61

nodes, will be deployed to handle inbound communication and to perform SSL offloading. It will also be responsible for internal communication between the various domains and components as shown in Figure 31: OTD Deployment.

From a security architecture perspective, OTD will only expose routing to OSB on the external network. All other routing will be done by binding to internal InfiniBand network addresses.

ODA

ExternalNetwork InternalNetwork

OSBSOA JMS

OTD

ExternalClient

Figure 31: OTD Deployment

Load Balancer Configurations

Load balancer Target Host Protocol External Application OTD HTTPS OTD OSB HTTP OTD SOA HTTP OTD SOA HTTP OTD EE HTTP OTD EE HTTP

As described in the previous section, OTD is also capable of load balancing other protocols. These can be configured as and when the need arises. Virtual IP Configuration As part of the network architecture we deployed Virtual IP’s to enable Admin server failover for all domains. Weblogic managed servers bind and connect to the Admin server on a pre-configured address, so in order for failover to work, the Admin address cannot change when moving between different hosts. This is achieved by binding the host that is hosting the Admin server to the Virtual IP assigned to the Admin server. External Virtual IP Environment Domain Servers Production PRDOSBD100 Admin PRDSOAD100 Admin PRDSOAD200 Admin PRDJMSD100 Admin QA QAOSBD100 Admin

of 61

QASOAD100 Admin QASOAD200 Admin QAJMSD100 Admin Development DEVOSBD100 Admin DEVSOAD100 Admin DEVSOAD200 Admin DEVJMSD100 Admin

6.7.5 Firewall Access OT à DMZ

Source Destination Port Business Reason (per port) ABB STCMS server on

DMZ ODA 18017 Write messages to STCMS queue. As noted

above, the STCMS is a legacy SeeBeyond JMS which must be maintained to receive messages form systems/platforms in the OT network.

FMS STCMS server on DMZ ODA

18017 Write messages to STCMS queue. As noted above, the STCMS is a legacy SeeBeyond JMS which must be maintained to receive messages form systems/platforms in the OT network.

DMZ à OT Source Port Business Reason (per port)

All virtual machines on the DMZ ODA

123 Time synchronization with the network time server

Oracle Fusion Middleware servers (OSB, SOA, WL) on DMZ ODA

1521 Database access to SCADA/Engineering database


1521 Database access to Universal Data Warehouse database

DMZ Outbound Source Business Reason (per port)


Communication to the NetIQ IAM LDAP services to manage and authenticate user accounts for the OS and Weblogic


SSH, SFTP access to the jump server


SSH, SFTP access to the IT ODAs from the Oracle Fusion Middleware Environment of the DMZ ODA.


Allow email access


Send SNMP alerts to Eskom SNMP server – for notifications regarding component outage.


Send syslog events to central logging server

All OFM Administration servers

Authentication of Oracle Fusion Middleware service and named users

of 61

on the DMZ ODA against netIQ IAM. Oracle Fusion Middleware servers (OSB, SOA, WL) on DMZ ODA

Access to web services deployed in IT domain.


Access to JMS resources in IT domain.


Access to databases in IT domain.

All virtual machines on DMZ ODA

Allows the OEM agents deployed to send metrics to central OEM server.

Simmerpan Source Business Reason (per port)


Send logs to Splunk regional collector


Netbackup/Infrastructure requirements

Bellville Source Business Reason (per port)





Bloemfontein Source Business Reason (per port)





Mkondeni Source Business Reason (per port)





Polokwane Source Business Reason (per port)





Witbank Source Business Reason (per port)



of 61



East London Source Business Reason (per port)





of 61

Eskom Academy of Learning (Non-Production)

Source Business Reason (per port) All virtual machines on the DMZ ODA




DMZ Inbound

Source Business Reason (per port) Communication to the NetIQ IAM LDAP

services to manage and authenticate against user accounts for the OS and Weblogic

SSH, SFTP access to the DMZ ODAs from the Jump Server.

Oracle Fusion Middleware servers (OSB, SOA, WL) on IT ODA

SSH, SFTP access to the DMZ ODAs from the Oracle Fusion Middleware Environment of the IT ODA.


Access to web services deployed in DMZ domain.


Access to JMS resources in DMZ domain.


Access to databases in DMZ domain.

All virtual machines on DMZ ODA

Allows the OEM agents deployed to send metrics to central OEM server.

Eskom Management Network

Oracle Integrated Lights Out Management: CD, Mouse & Keyboard, Diskette, Encryption, Authentication, Servicetag Daemon, Video, Serial

Jump Server Access to e-manger in the DMZ to be able to create queue

9 SCADA application server zone1 Eccvssaz0203 on Corporate LAN accessing STCMS zone in DMZ

SCADA application server zone1 Eccvssaz0204 on Corporate LAN accessing STCMS zone in DMZ

SCADA application server zone2 on Corporate LAN pushing messages onto message broker on DMZ LAN

6.8 Security Architecture The following section will detail security elements around the deployment architecture. The following diagram depicts a high level security architecture and all the paths that will be secured.

of 61

ODA

vServer

OTD

EnterpriseApplications

(1)

vServer vServer vServer

FMW FMW FMW

(2) (2)

(3)OperationalPerson

(3a)

(3b)

DB DB DB

InfinibandNetwork

(4)

(5)

LDAP

(3b)

EnterpriseApplications

(6)

NetIQ

Figure 32: Security Architecture

• (1) – When applications outside of the integration platform needs to communicate

with the integration layer, communication will happen via OTD and all communication will be secured using SSL. The application will also provide HTTP basic credentials that will be passed to the Oracle Fusion Middleware layer, where the WebLogic embedded LDAP store will be used to store these credentials.

• (2) – OTD will load balance both inbound and internal component traffic. The internal traffic will not be secured using SSL as it happens over a secured internal connection and is local to the Oracle Database Appliance. The unsecured load balancing rules will not be accessible from outside the ODA as the listening address will be bound to the same internal secured connection.

• (3a) – Operational resources need to login to the virtual machines via SSH. They will have to authenticate using username and password. Please refer to Section 0 User Management Strategy.

of 61

• (3b) – When operational resources logs into the WebLogic console, they will provide their Active Directory credentials. WebLogic will be configured to authenticate against Active Directory for non-system accounts.

• (4) – Connections to the integration databases will happen via the internal and thus no additional security is required.

• (5) – Credentials to connect to the database will be stored within the Weblogic JDBC configuration files. All these credentials will be encrypted using a unique encryption key per domain.

• (6) – Outbound communication to backend systems, like Maximo, will be handled by OSB. If communication to the backend system needs to be secured, the certificates will be stored within the OSB Weblogic keystore. Because the OSB instances are listening for incoming communications on the local network, this approach does not pose a security thread as OSB will not be reachable from outside the ODA.

The following table can be used as a security matrix when trying to determine if security will be applied and how. Connection Method Secured Technologies Inbound Application connectivity

Y • OTD SSL Offloading • HTTP Basic Authentication • WebLogic embedded LDAP

SSH to vServers Y • Username & Password • Active Directory

WebLogic console login Y • Username & Password • WebLogic embedded LDAP • Active Directory

OTD Internal Routing Y • Infiniband Secured Network Database Connectivity Y • Encrypted Username &

Password • Infiniband Secured Network

OSB Outbound Connectivity

Y • SSL • Certificates imported to keystore

OTD

OTD guarantees high performance through SSL/TLS offloading, content caching and effective HTTP compression. Flexible configuration of routing and load control on back-end servers through a selection or combination of Request-based routing, Content-based routing, Request rate acceleration and Connection limiting. The request load and quality of service are also tunable through request rate limiting and quality of service tuning. OTD is also capable of TCP load balancing if required.

SSL As part of the network architecture, OTD will be deployed and utilized within the environment. Each of these components have the ability to do SSL offloading and onboarding depending on the requirements. From a security perspective all communication into the integration landscape needs to be secured using SSL. OTD will do SSL offloading and pass unencrypted communication to the OSB instances. All inter-

of 61

component communication within the Oracle Database Appliance does not need to be secured as the internal network is considered to be secure and no packets can be sniffed from outside the ODA.

Keystores As part of the deployment process, custom keystores will be deployed to store server and client certificates. Weblogic will need a trust store if outbound communication to a backend system needs to be secured with SSL.

WebLogic Groups & Roles The following table details the default Oracle Fusion Middleware Users, Groups and Roles that defines how user access can be restricted on a WebLogic side. Please note that these are not application level groups and roles but rather specifies how individuals can access WebLogic from an operational perspective. The following table details the default groups. Group Membership Administrators By default, this group contains the user information entered

as part of the installation process (that is, the Configuration Wizard), and the system user if the WebLogic Server instance is running Compatibility security. Any user assigned to the Administrators group is granted the Admin security role by default.

Deployers By default, this group is empty. Any user assigned to the Deployers group is granted the Deployer security role by default.

Operators By default, this group is empty. Any user assigned to the Operators group is granted the Operator security role by default.

Monitors By default, this group is empty. Any user assigned to the Monitors group is granted the Monitor security role by default.

AppTesters By default, this group is empty. Any user assigned to the AppTesters group is granted the AppTester security role by default.

CrossDomainConnectors By default, this group is empty. Any user assigned to the CrossDomainConnectors group is granted the CrossDomainConnector security role by default.

AdminChannelUsers By default, this group is empty. Any user assigned to the AdminChannelUsers group is granted the AdminChannelUser security role by default.

OracleSystemGroup By default, this group contains the user OracleSystemUser and is granted the OracleSystemRole role by default.

of 61

The following table details the global roles that WebLogic Server defines in the security realm that it installs. The table also summarizes the access that the default security policies grant to each role and indicates which groups are in each role by default. Global Role Default Policies Grant

Access To… Default Conditions Include This Group…

Admin • View the server configuration, including the encrypted value of some encrypted attributes.

• Modify the entire server configuration.

• Deploy Enterprise Applications and Web application, EJB, Java EE Connector, and Web Service modules.

• Start, resume, and stop servers.

Administrators

AdminChannelUser Access the administrative channel, AdminChannel

AdminChannelUsers, Administrators, Deployers, Operators, Monitors, and AppTesters

Deployer • View the server configuration, including some encrypted attributes related to deployment activities.

• Change startup and shutdown classes, Web applications, JDBC data pool connections, EJB, Java EE Connector, Web Service, and WebLogic Tuxedo Connector components. If applicable, edit deployment descriptors.

• Access deployment operations in the Java EE Deployment Implementation (JSR-88)

Deployers

Operator • View the server configuration, except for encrypted attributes.

Operators

of 61

• Start, resume, and stop servers.

Monitor View the server configuration, except for encrypted attributes. This security role effectively provides read-only access to the WebLogic Server Administration Console, WLST, and MBean APIs.

Monitors

AppTester Access applications for testing purposes that are running in Administration mode.

AppTesters

CrossDomainConnector Make inter-domain calls from foreign domains.

CrossDomainConnectors

OracleSystemRole Assert identity on behalf of users whose WS-Security tokens have been authenticated. Note: This global role is provided for use by Oracle Web Services Manager.

OracleSystemGroup

of 61

File System Security

The Oracles Enterprise deployment guides suggests to create 3 mount points. As a result the oracle user owns all three mount points. In order to manage this effectively without the need to use the oracle user the following groups need to be created and assigned to each mount point to be able to administrate the different mount points. A default user “oracle” belongs to default group “oinstall”. This user owns all installed technology and mount points /u01, /u02 and /u03. This user should not be used to login and SSH. This “oracle” user is a local user to each virtual machine. Note: A key requirement is the installation of the NetIQ fan-driver before any additional OS users are created. This will ensure that the UID and GID of users does not change. Groups Mount Comments Group 1 /u03 and /u02 Used to administrate the Managed Servers and logging. Group 2 /u03 Used to administrate logging. oinstall /u01,/u02 and /u03 Default created group. Must only be assigned to administrator with full

access.

Operating System Groups

The following groups are required for the Operating System: • root • DBA • logging • osadmin • oinstallstaff

This section considers security from the following perspectives:

• User Management Strategy • Application Access • Server Access • Firewall Access

of 61

User Management Strategy

The table below provides a summary of the user management strategy: System Account Service Account User Account Description Built-In Account Software

Installation Service Access

Named User

Location Local Eskom IAM Eskom IAM Usage These are the

accounts which are provided “out-of-the-box” with the hardware/software elements.

These are the accounts which are used to install and configure the application software, for example: Oracle database, Oracle Fusion Middleware. These are also the accounts which are used for “programmatic” access, for example: web service or API to solutions deployed on the application stack.

These are operational, human users which access the infrastructure/application/solution for day-to-day Operation and Administration.

Sample of users to commission

root Primary OS account

oraagent OEM account

lukhans Shaun Lukhan

weblogic Primary OFM account

orarom Platinum services account

bhoolar Rakesh Bhoola

Implementation Requirements

None The Novell® Identity Manager Fan-Out Driver (detailed in Section 0) LDAP Authentication

The Novell® Identity Manager Fan-Out Driver (detailed in Section 0) LDAP Authentication

of 61

NetIQ Identify & Access Manager

As NetIQ has been established as the Identity & Access Management (IAM) standard in the Eskom environment, this will be installed on all Virtual Machines immediately after OS installation and configuration. The Novell® Identity Manager 4.0.2 Fan-Out Driver is an identity provisioning solution, based on Novell eDirectory™, Identity Manager, and related technology. With Identity Manager, you can manage the full user life cycle, delivering first-day access to essential resources, providing single login, and modifying or revoking access rights. Identity Manager also provides self-service features that enable users to maintain their own passwords and profile information. By adding the Fan-Out Driver, you can use Identity Manager to fan out identity provisioning to hundreds of systems with minimal effort. You can centrally manage user accounts and have them automatically created, configured, maintained, and removed when appropriate. It uses extensible scripts to manage account rights, home directories, and other resources as well as the user definitions themselves. Meanwhile, system administrators for the individual platforms in your enterprise can retain control over their areas.

Installation Scope The NetIQ Fan-Out Driver can be installed on the ODA base domain and any VM. However, the agent cannot be installed on the DOM0 or DOMU domain. DOM0 is running Oracle VM Server for x86, which provides a minimal Oracle Linux with virtualization technology included.

Application Access The table below provides a summary of the anticipated OFM Application access requirements for the Regional EIP environment from IT to DMZ and vice versa: Source Target Requirement Security Mechanism IT DMZ Web Services HTTPS, Basic Auth,

SSL JMS T3 over IIP, Basic Auth JDBC Username/Password

SSH/SFTP SSL, Certs, Basic Auth DMZ IT Web Services HTTPS, Basic Auth,

SSL JMS T3 over IIP, Basic Auth JDBC Username/Password

SSH/SFTP SSL, Certs, Basic Auth OT DMZ STCMS Username/Password DMZ OT JDBC Username/Password

• Table 19: Application Access Summary

of 61

7 High Availability From a physical resource mapping perspective, all domains will be deployed across two different compute nodes. This approach ensures that we achieve a highly-available architecture in conjunction with WebLogic clustering. If we lose one virtual machine or even a complete compute node, part of the cluster will continue running on the other compute node. The appliance is also designed with mission-critical requirements in mind, with hot-swappable and redundant components.

2 ODA X5-2 machines will be allocated to each region, deployed as follows:

Machine #1, Virtualized & Clustered Platform Optimized for Database and Applications – IT-ODA;

Machine #2, Virtualized & Clustered Platform Optimized for Database and Applications – DMZ-ODA;

The environment will be configured in a 2 node RAC (Real Application Cluster) for the databases to promote high availability in the regions.

The identified Disaster Recovery solution is the replacement of the physical infrastructure element that has failed. In the event of a whole server (ODA) failure, a replacement server (ODA) will be shipped to the affected site and will be configured and restored as necessary. In the event of a component failure, the failed component will be replaced. This strategy was selected to minimize infrastructure costs as well as due to the network infrastructure between regional sites and the Central platform.

The servers allocated for Disaster Recovery will be pre-built with the necessary software elements – such as virtualization, database, Operating System and Oracle Fusion Middleware Platform. Upon commissioning at the affected site, additional configuration can be applied to restore the platform. Database will then be restored from RMAN backups.

8 Disaster Recovery

There is 3 major failure points at each site : 1) Catastrophic failure on the regional OT system environment

2) Catastrophic failure regional ODA DMZ environment

3) Catastrophic failure regional IT ODA environment

of 61

• The DR for the 3 major failure points at each site is :

1. Catastrophic failure on the regional OT system environment

• Do nothing – the OT does not currently have DR

2. Catastrophic failure regional ODA DMZ environment

• Assuming the DMZ network is still up

• Swap out the DMZ ODA with a templatzied ODA

3. Catastrophic failure regional IT ODA environment

• Assuming the regional IT network is still up

• Re-route the traffic to a racked and stacked ODA connected to the IT 141

PROJECT INFORMATION Enterprise Template Architecture ESB Regional... · format and sends these...

Documents

Transcript of PROJECT INFORMATION Enterprise Template Architecture ESB Regional... · format and sends these...