PACI RAS Gen Group

Misoperation on 5-15-2018

Pacific Gas & Electric

July 24th, 2018

PAGE 2 PACI RAS Internal

Pacific A/C Intertie (PACI) RAS

Background

▪ Monitors TLO, DLO, and SLO combination of the 500kV lines within

PG&E and, based on predetermined System conditions, takes

automated corrective actions, including but not limited to

load/generation dropping, bypassing of series compensation,

insertion/removal of shunt devices and Intertie Separation

▪ Helps maintain 500kV Electric Grid Stability and thermal overloads of

the underlying Transmission system

▪ 3rd most important system for PG&E to have working after a major

catastrophe – Telecom, EMS, RAS

▪ Must be available at least 99.995% of the time (all but 26 minutes of the

year).

PAGE 3 PACI RAS Internal

Tracy

Dixon

Round Mt.

Table Mt.

MalinCapt. Jack

Olinda

Vaca

Vincent

Midway

Gates

DiabloCanyon

Los Banos

Metcalf

Tesla

MossLanding

RAS monitors status of Circuit Breakers and

Undercurrent Relays needed to determine line

status.

Controllers are designed to recognize line

outages and issue remedial actions within 30

milliseconds.

Remedial Actions are predetermined by

information stored in RAS Tables.

500KV NETWORK THROUGH PG&E

PAGE 4 PACI RAS Internal

Caribou/ButtValley

Helms

Wheeler RidgeWind GapBuena Vista

Pearblossom

Olinda

Tracy

Round Mt.

Four Corners

Munro ControlCenter (MCC)

SierraPacific Power

Cascade

Dittmer

Midway

Table Mt.

Chief Joseph BrakeGrand CouleeMcNary PHJohn Day PHWanapum PHWells PHRocky Beach PHShrum, MicaRevelstokeAGC Off

Pacific NW Remedials

Kern OilKern PPTaftMidsun Sw StaMidway Sub

Elk Hills

La Paloma 1&2

Sunset Gen

La Paloma 3&4

IRAS Gen GroupsIn Midway Area

El CerritoVaca DixonNewarkMonta Vista

PittsburgLakewoodMartin

San MateoGold Hill

Ignacio Fulton

Moss Landing Salinas Green Valley Stagg

IRAS Load Groups

Hyatt

Kinport

Maxwell

PACI RAS OUTPUTS (Remedials)

PAGE 5 PACI RAS Internal

Block Diagram

Field to Control Center

IED

IED

System A/C

IED

IED

System B/D

Substation 1

IED

IED

IED

System A/C

IED

IED

IED

System B/D

Substation 2

Channel

Banks

A B

IEDIED

GW

IED IED

GW

IED IED

C D

Data GatewaysProtocol Converters provide Cross-Over of data from Systems AC BD

AC and BD SystemsSubstations have separate System AC and System DB devices.The top (ACST) or (BDST) Chassis talks point-to-point with it’s counter-part in the Control Centers.

Inputs are Hardwired and Cross-Messaged.The IED device local to the controller is hard-wired.

All Remedial are hard-wiredOutput RA’s are hard-wired to local IED’s and Cross-wired to the other Controller in the same control center

The location of a fault in a

route determines if the data

can either reach one or

neither control center.

In other words, it’s possible for

Controller A to receive data on

Route 1 (system AC) while

controller C looses

communication.

Route 1 Route 2 Route 1 Route 2

PAGE 6 PACI RAS Internal

Past Performance

2010 2011 2012

Date of Event

7/16/2010 3/27/2011 3/17/2012

Customer Impact

22,070 customers 169,314 customers 58,296 customers

Root Cause Design error WPE – Script Deviation System Isolation--Field

Duration Approx. 24 Min. 1 hour momentary

Action Items /

Mitigations

-QA development

system (completed in

2013)

-Procedure to govern

administration of any

future PACIRAS

Testing

-Seeking changes in

required testing.

WECC approved testing

changes in late 2013.

Awaiting FERC approval

– 2014. Note: Testing

annually is not required

as RAS platforms are

continuously monitored

for performance.

PAGE 7 PACI RAS Internal

Event Details

PAGE 8 PACI RAS Internal

Sequence of Events

▪ On May 14, 2018, PG&E installed a new battery bank with two new

rectifiers at Midway substation.

▪ On May 15, 2018 at approximately 08:30 PST, most of the RFL 9745’s at

Midway substation started sending trips to the remote IRAS Generation

Groups 2, 3, 4 and 5.

o IRAS Gen Group 2 consists of Elk Hills CTG 1, CTG 2 and STG (System B/D)

o IRAS Gen Group 3 consists of La Paloma Units 1 and 2 (System B/D)

o IRAS Gen Group 4 consists of Sunrise CTG 1, CTG 2 and STG 3 (System B/D)

o IRAS Gen Group 5 consists of La Paloma Units 3 and 4 (System A/C)

▪ At the time of the event, only Elk Hills was generating. La Paloma and

Sunrise were off-line. The event resulted in the tripping of 400 MW of

generation at Elk Hills.

▪ Even after the initial event, trip signals were still being received at each of

the remote IRAS Gen Group sites.

PAGE 9 PACI RAS Internal

Sequence of Events (continued…)

▪ It was confirmed by the RAS Operations personnel that the trip signals were

not initiated the centralized PACI RAS Controllers nor any of the scheme’s

N60 devices.

▪ During the troubleshooting process, System Protection recommended to

cut-out the trips to IRAS Gen Group 2, IRAS Gen Group 3, IRAS Gen

Group 4 and IRAS Gen Group 5

▪ The source of the unintended trips signals was identified as a failure of one

of the rectifiers that had been installed just the previous day. The nature of

the failed rectifier resulted in a 80V AC signal being present from each of

the DC legs to ground.

▪ The bad rectifier was taken off line and the unintended trips signals cleared.

PAGE 10 PACI RAS Internal

Next steps

▪ An internal CAP (Corrective Action Program) Item was created and the

need to perform an Apparent Cause Evaluation (ACE) was identified.

▪ The projected due date for this ACE and the associated report is 8/15/2018

▪ The failed rectifier was sent back to the manufacturer for repair and was

recently returned to service earlier this month.

PAGE 11 PACI RAS Internal

WECC EPAS Questions

PAGE 12 PACI RAS Internal

How did the rectifier affect the RAS

system?

▪ The nature of the failure introduced an AC signal on the

DC legs of the rectifier.

▪ When the AC signal was positive and above the trip

setting, the erroneous signals were sent. This was

consistent with the continuous trip signal being

sent/received via the RFL equipment until the failed

rectifier was taken offline.

PAGE 13 PACI RAS Internal

What was the trip signal that was lost

(DC, Rectifier failure, low ac output, etc)?

▪ A trip signal was not lost in this case.

▪ Each Midway Gen group segment trip command has

physical output(s) at Midway Substation and each output

is wired via copper to the respective RFL 9745 input

(DTT Send). The RFL 9745 DC inputs sensed a positive

assertion (trip command) while the Ripple AC was

present.

PAGE 14 PACI RAS Internal

Please provide a diagram of the relevant

portions of the report

▪ See following two slides

PAGE 17 PACI RAS Internal

Questions ?