AEMO - Upgrade PI System to 2012 - etouches · • PI use at AEMO • Upgrade PI System to 2012 ......

Presented by

© Copyr igh t 2013 OSIso f t , LLC.

AEMO - Upgrade PI System to 2012

Bryan Connell


Agenda• AEMO Overview• PI use at AEMO• Upgrade PI System to 2012

– Business drivers– How it was done– Where we are at now

2


Who is AEMO• Australian Energy Market Operator• Formed in July 2009, amalgamation of 5 electricity and

gas companies • Operational, planning and emergency management

roles in both gas and electricity sectors across eastern and south-eastern Australia

• AEMO operates independently and ownership is shared between government and industry

– 60% States and Commonwealth– 40% Industry

• Funded by Market Participants and operates on a cost recovery model

3

GRMO


Who is AEMO• Operators of the electricity markets and power systems across

eastern and south-eastern Australia – the NEM• Delivers planning advice in eastern and south-eastern Australia to

help the energy industry make investment decisions• In Victoria, AEMO oversees the introduction of new generation

capacity or connections to the electricity transmission network• Operators of the retail and wholesale gas markets in south-east

Australia, and the Victorian gas DTS (Declared Transmission System)

• Operators of the gas Short Term Trading Market (STTM) in Sydney, Adelaide and Brisbane

4


The National Electricity Market• Began operating as a wholesale spot market of

electricity in December 1998• Supports 19 million residents, comprising five

interconnected regions – QLD, NSW (inc. ACT), SA, Vic and Tas

• Extends over 5,000 km from Port Douglas (QLD) to Tasmania to Port Lincoln (SA)

• AEMO has roles in the NEM as – Independent system operator– Wholesale market operator– Retail market facilitator– Transmission planner

• Around $10 billion of electricity is traded annually in the NEM spot market

5


NEM – The parties• 6 State jurisdictions• Commonwealth jurisdiction• 170 Registered Participants

– 5 main transmission companies– 50 wholesale customer/retail businesses– 30 scheduled generation businesses

• Generation capacity about 48,000 MW• Maximum demand 35,680 MW (summer 2008/2009)

6


Wholesale Gas Markets

• AEMO operates the Victorian gas transmission network

• This services about 1.8 million customers• More that 220 petajoules a year are supplied• AEMO operate the gas Short Term Trading

Markets (STTMs) in QLD, SA and NSW with around 144 petajoules supplied a year

7


Control Centres• Electricity Control Centres/Data Centres in Brisbane and Sydney

– Control Centres operate as co primary• Gas Control Centre/ Data Centre in Melbourne (DR data centre in Brisbane)

8


PI Systems and AEMO• A PI System was purchased in 2001 as a historian and trending tool

for the electricity SCADA • Originally purchased to assist electricity system operations

personnel by – Collecting and storing large volumes of real-time data at fast scan rates

whilst being able to respond to user requests in a timely manner– Presenting the data to users in a variety of means without requiring

additional developments or applications– Allowing end users to retrieve the data directly, rather than being dependent

on others for its provision– Providing efficient long term data storage and retrieval

• It was considered a low risk entry option with easy expansion and 10,000 tags were purchased

9


PI Systems and AEMO• In 2004 the PI system was upgraded to dual PI servers at Brisbane

and Sydney data centres• In 2008 the PI system upgraded the a 4 server collective, 2 servers

at each data centre.– One of the first customers using multi server collective– Had to implement our own high availability for interfaces – used PI ACE for

this purpose • Enlisted OSISoft services for the previous upgrades. This upgrade

was handled in house

10


PI Systems and AEMO• PI Products

– PI Processbook– PI Datalink– PI ACE (Advanced Computing Engine)– PI MCN Health Monitor– PI ODBC Client– PI Asset Framework (after 2012 upgrade)– PI Notifications (after 2012 upgrade – was using RTAlerts)– PI Batch (not actively using)– PI Webparts (not actively using)

11


PI Systems and AEMO• PI Interfaces

– PI interface for ESCA HABConnect (interface for electricity SCADA)

– PI Interface for Relational Database (RDBMS via ODBC) (interface for Gas SCADA)

– PI interface for Performance Monitor– PI interface for SNMP– PI interface for HTML– PI interface for Ping– PI to PI interface– PI Interface for Universal File and Stream Data (UFL)

12


PI Systems and AEMO• AEMO PI system now licensed for 90,000 tags collecting from the following

sources– Electricity SCADA data– Gas SCADA data– IT Server performance data– Critical Control Centre site infrastructure such as UPS equipment, air conditioning

plant and switchboard information– Bureau of meteorology and control centre environmental data

• Have empowered AEMO staff to build and develop their own suite of PI Processbook displays and PI Datalink spreadsheets

– Improved efficiency of staff throughout the organisation in performing their required tasks

– Using historical and real time data in determining metrics helping to make better business decisions

13


PI Systems and AEMO• The PI system has become integral to AEMO operation

– Real time operational tool in the electricity control room e.g. voltage control and event monitoring– Frequency and time deviation reporting– AGC performance– Constraint performance– Power system incident investigations– Weather monitoring– IT equipment monitoring– Control Centre environmental monitoring– Information base for PI interfaced applications

• Wind Forecasting System• Demand Forecasting System• Settlements estimation for next day• Load history published to AEMO website• Solar Forecasting System (soon)

17


PI System – Archiving Issue• Archiving would stop on primary PI server• Happened about 6 times over two year period• Unable to really determine the cause – message logged error accessing archive file ..

Archiving disabled• Maybe a heavy hitter with an archive query, but could never really establish that this was

the cause• Difficult issue to detect

– In some instances it was a few hours before we knew of the problem– Processbook trends continued to update on snapshot data

• Difficult to alert we had a problem– Configured the archiving status tag in perfmon, however RTAlerts does not trigger alerts from

snapshot data• Even though we always recovered all data as it was buffered, the applications such as

Wind Forecast and Demand Forecast were affected as they use the data in real time. The data they were using was stale.

18


PI System Upgrade – Business Drivers• The PI System hardware components were at end of life. Implementing a upgrade at this

point in time allowed the PI System platform to be moved to a virtual software platform in alignment with the company server consolidation strategy, reducing hardware costs and simplifying future upgrades

• Deploying to a new platform enabled the PI System servers to be positioned in multiple security zones, so the PI System was in alignment with the AEMO production isolation initiatives

• Upgrading the PI System software to the latest version had the following major benefits– It leveraged Windows Integration Security, authentication of users through Active Directory, single

sign-on– It is built to fully utilise 64-bit technology, multi-processor/multi-core hardware, and the latest

operating systems, increasing levels of performance– Implements latest product improvements, enhancements and bug fixes

19


PI System Upgrade – Business Drivers• PI Asset Framework would enable a consistent representation of the data

and provide structure for the information. Implementation of standards and improvements in data management, and integrating Gas SCADA requirements at an administration level were seen as additional benefits

• PI Notifications was seen as a considerable improvement on RTAlerts for notification and escalation of critical events

• Upgrading data collection interface software supported high availability failover and disconnected startup making improvements to the availability and robustness of data collection

20


PI System Prior to Upgrade

• 4 server collective across sites. Primary PI server in Brisbane

• Primary/standby interface nodes collecting local data. Failover handled by PI ACE

• PI ACE collective. Primary server biased to Sydney - handled by procedure

21


New PI System Architecture• 4 server collectives for PI

Servers and AF Servers across sites and security zones. Primaries in Brisbane

• Primary/Standby ACE server across sites. Primary biased to Sydney by procedure

• Primary/Standby Notification schedulers across sites, 2 available notifications acknowledgement servers

• Primary/Standby interface Nodes across sites

22


PI System Upgrade

• 2012 Regional seminar training exercise on planning upgrade to PI Server 2012

• Engaged [email protected]

23


Migrate to the new hardware• Suggested approach was to add this new target

machine to the existing collective and then promote to primary

• Concerns with this approach– This did involve downtime– Further data loss implications because we use

buffer server not the buffer subsystem on our interface nodes (we have interfaces that send data to individual PI servers and not the collective)

– Move to new system without a bedding in period –seen as a risk

– Other changes we wanted to implement like windows integrated security would have to be applied before testing

• Approach taken was to build and run a parallel system

– Built a new 4 server PI Collective (version 3.4.380.70 64-bit), and 2 new PI Interface nodes on the new hardware

– Configured the new interfaces, collecting data from the same data sources as existing system, configured and tested for HA failover

– Once this was up and ready we reinitialised the primary server in new target system from a backup of the old PI system

– Used PI to PI to backfill the data missing between backup time and when the new system was up and running after initialisation

– Copied and registered all the archives from the old to the new

– Now had two PI Systems running in parallel, with no data loss

24


PI System Upgrade• With two PI Systems running in parallel,

the new system being bedded in– Built and commissioned a 4 server AF

collective (AF Server x64 2012). No issues with the MDB to AF synchronisation

– Upgraded the new system to PI Server 2012 (was impressed that the collective was happy with PI servers members running different versions)

– Build and commissioned PI Notifications 2012 Servers. Migrated RTAlerts to PI Notifications

– Built and commissioned PI ACE 2010 R2 SP1. Migrated ACE functions from the old to the new

• With the new PI system we implemented Windows authentication of users, disabling explicit login

• Did encounter an issue, although the two PI servers to be accessed by corporate users were put in a the corporate vLan, they were incorrectly put in the Prod Domain. Corporate users could not access the PI system as the authentication request went to a Prod Domain controller

25


PI System Upgrade• The last steps were to migrate users and

the applications using PI.• For the general users

– Used script modify registry settings pushed out new PI server configurations (HKLM\Software\PISystem\PI-SDK\1.0\ServerHandles)

– Used a custom addin to Processbook to modify the referenced PI collective on display objects

• PI applications were cut over eventually, another ‘archiving stopped’ issue out of hours expedited the cutover of the last systems

• Did take a long time to actually complete the upgrade

– The two systems ran in parallel for around 4 months

– The task of tag maintenance was duplicated during this time as both systems had to be kept up to date

• This process minimised the risk taken at any one step as changes and cutovers of systems occurred piecemeal not a one big cutover

26


Now and the Future• No issues with ‘archiving stopped’ on the

new system• Are very happy with PI Notifications,

have already expanded notification list and are planning more work in this area

• Using notification history to develop metrics on SCADA availability, and application availability

• Have put structure around the Gas SCADA, building a geographic hierarchy in AF assisting the relatively new PI users in the Gas part of the organisation

• Things on the PI agenda– Putting some AF structure around the

Electricity SCADA– Thinking of auto point creation for

electricity SCADA– Using PI WebParts to publish information

to corporate SharePoint site –dashboards, systems and applications metrics

28


Bryan [email protected] EngineerAEMO

29

Brought to you by


AEMO - Upgrade PI System to 2012 - etouches · • PI use at AEMO • Upgrade PI System to 2012 ......

Documents

Transcript of AEMO - Upgrade PI System to 2012 - etouches · • PI use at AEMO • Upgrade PI System to 2012 ......