NERC Lessons Learned Summary LLs Published in September 2015.
-
Upload
godwin-dickerson -
Category
Documents
-
view
217 -
download
2
Transcript of NERC Lessons Learned Summary LLs Published in September 2015.
![Page 1: NERC Lessons Learned Summary LLs Published in September 2015.](https://reader036.fdocuments.us/reader036/viewer/2022081813/5697c02e1a28abf838cda41e/html5/thumbnails/1.jpg)
NERC Lessons Learned Summary
LLs Published in September 2015
![Page 2: NERC Lessons Learned Summary LLs Published in September 2015.](https://reader036.fdocuments.us/reader036/viewer/2022081813/5697c02e1a28abf838cda41e/html5/thumbnails/2.jpg)
Two NERC lessons learned (LL) to be published in September and one is pending • LL20150901- Loss of EMS Communications Due
to Lack of Validation on EMS Database RTU Configuration Parameter
• LL20150902 – Relay Design and Testing Practices to Prevent Scheme Failure
• Pending• Loss of EMS due to RTU LAN UPS failure
NERC Lessons Learned Published in 2015
![Page 3: NERC Lessons Learned Summary LLs Published in September 2015.](https://reader036.fdocuments.us/reader036/viewer/2022081813/5697c02e1a28abf838cda41e/html5/thumbnails/3.jpg)
Commissioning new remote terminal unit (RTU) results in EMS outage
One point incorrectly configured results in termination of the remote communication server Process coordinates the polling and commands of RTUs and the
insertion of telemetry into the real time database
Automatic failover (from primary and back-up) did not correct issue since the error was in the database configuration
EMS lost communications with all RTUsState Estimator and Contingency Analysis had stale dataMost changes verified prior to execution but the database
configuration parameters were not checked
Loss of EMS Communications Due to Lack of Validation on EMS Database RTU
![Page 4: NERC Lessons Learned Summary LLs Published in September 2015.](https://reader036.fdocuments.us/reader036/viewer/2022081813/5697c02e1a28abf838cda41e/html5/thumbnails/4.jpg)
Corrective ActionsParameter configuration was correctedScript was run to confirm the configuration error doesn’t exist
elsewhere Revised procedure to confirm EMS database RTU
configuration is correct in quality assurance prior to uploading a database change to production
Created a case with the EMS vendor documenting the defect Vendor will provide a new release with the defect resolved
Loss of EMS Communications Due to Lack of Validation on EMS Database RTU
![Page 5: NERC Lessons Learned Summary LLs Published in September 2015.](https://reader036.fdocuments.us/reader036/viewer/2022081813/5697c02e1a28abf838cda41e/html5/thumbnails/5.jpg)
LessonsEvaluate parameters being checked by the
database editor to determine if there are any gaps in the parameter validation If so, establish procedures for manual validation
SCADA software should be designed to generate error messages and avoid termination upon an incorrect parameter
Recovery strategy and procedures should be developed for quick recovery from failed updates
Loss of EMS Communications Due to Lack of Validation on EMS Database RTU
![Page 6: NERC Lessons Learned Summary LLs Published in September 2015.](https://reader036.fdocuments.us/reader036/viewer/2022081813/5697c02e1a28abf838cda41e/html5/thumbnails/6.jpg)
• Single-phase to ground fault occurred on a 230 kV three-terminal feeder
• Associated breakers tripped then the line reclosed at one end (as designed)
• Two protection equipment failures on two separate relay systems prevented the proper clearing after the automatic reclose
• 1: Loose connection in the trip aux relay coil cutoff contact string• 2: Believed to be intermittent connection issue
• Fault evolved to multi phase and remained on the system for 58 seconds
• Breaker Failure didn’t initiate since it was tied to same aux trip relay
• Fault cleared by backup ground protection on two 500 kV lines
Relay Design and Testing Practices to Prevent Scheme Failure
![Page 7: NERC Lessons Learned Summary LLs Published in September 2015.](https://reader036.fdocuments.us/reader036/viewer/2022081813/5697c02e1a28abf838cda41e/html5/thumbnails/7.jpg)
Corrective ActionsTransmission Owner (TO) evaluated existing
procedures to align periodic relay testing with planned transmission outages and perform circuit breaker trip testing
52a contacts rewired in parallel rather than in series Replaced a relay as neededNew relay eliminates need for the 52a contacts in the
breaker trip circuit
Relay Design and Testing Practices to Prevent Scheme Failure
![Page 8: NERC Lessons Learned Summary LLs Published in September 2015.](https://reader036.fdocuments.us/reader036/viewer/2022081813/5697c02e1a28abf838cda41e/html5/thumbnails/8.jpg)
LessonsKey issue: Breaker Failure Initiate (BFI) signal
originated from auxiliary trip relayOptions:
Use separate contact to provide BFI Use dedicated aux relay for BFI Connect protective relay trip contact directly to breaker
failure relay input if relay will accommodate voltage input
Avoid use of 52a contacts in seriesConsider periodic functional tests
Relay Design and Testing Practices to Prevent Scheme Failure
![Page 9: NERC Lessons Learned Summary LLs Published in September 2015.](https://reader036.fdocuments.us/reader036/viewer/2022081813/5697c02e1a28abf838cda41e/html5/thumbnails/9.jpg)
• Rack mounted uninterruptible power supply (UPS) Failed resulting in loss of remote terminal unit (RTU) local area network (LAN) and then the loss of EMS and ICCP for 50 minutes
• UPS added to accommodate EMS/SCADA hardware upgrade• Remained in service as a dual layer UPS
• UPS battery pack failed and the UPS did not have an internal bypass to house power
• Failover was not available since the front end processor for the RTU LAN at the back-up was pointed at a replacement system’
• The entity was in transition to a new software for their EMS
Loss of EMS Due to RTU LAN UPS failure
![Page 10: NERC Lessons Learned Summary LLs Published in September 2015.](https://reader036.fdocuments.us/reader036/viewer/2022081813/5697c02e1a28abf838cda41e/html5/thumbnails/10.jpg)
Corrective ActionsRTU LAN routers were plugged into another UPS and
rebooted, restoring EMS visibility and ICCP link connectivityFEP at the remote site was reconfigured to allow sending data
to both the replacement SCADA/EMS and the existing SCADA/EMS systems simultaneously by using additional IP addresses
Primary and Backup Routers at the Primary Site were plugged into alternate UPSs
Additional (temporary) rack mounted UPSs will be removed upon cutover to the replacement SCADA/EMS system (Vendor 1 to Vendor 2)
Loss of EMS Due to RTU LAN UPS failure
![Page 11: NERC Lessons Learned Summary LLs Published in September 2015.](https://reader036.fdocuments.us/reader036/viewer/2022081813/5697c02e1a28abf838cda41e/html5/thumbnails/11.jpg)
LessonsUPS systems should be checked to verify they have
an internal bypass Periodic maintenance and monitoring on any UPS
system is beneficial Some UPS systems perform battery maintenance/cycling
internally but additional checks may be performed to verify functionality
Loss of EMS Due to RTU LAN UPS failure
![Page 12: NERC Lessons Learned Summary LLs Published in September 2015.](https://reader036.fdocuments.us/reader036/viewer/2022081813/5697c02e1a28abf838cda41e/html5/thumbnails/12.jpg)
Directions to Lessons Learned:Go to www.NERC.com > “Program Areas & Departments”
tab > “Reliability Risk Management” (left side menu) > “Event Analysis” (left side menu) > “Lessons Learned” (left side menu)
NERC’s goal with publishing lessons learned is to provide industry with technical and understandable information that assists them with maintaining the reliability of the bulk power system. NERC requests that industry provide input on lessons learned by taking the short survey. The survey link is provided on each Lesson Learned.
Link to Lessons Learned
![Page 13: NERC Lessons Learned Summary LLs Published in September 2015.](https://reader036.fdocuments.us/reader036/viewer/2022081813/5697c02e1a28abf838cda41e/html5/thumbnails/13.jpg)
Questions?