Slide 1

Compiled by Chuck Newton Newton-Evans Research Company September 2012 Slide 2 2012 Usage Patterns and Trends in Control Center SOP, Visualization and Cyber Security Welcome to this briefing session: Slide 3 A Review of Findings from Three Studies Conducted in 2012 by Newton-Evans Research (1) Standard Operating Procedures For Control Room Operations. Larger utilities and ISO/RTOs Small Sample (26 IOUs, RTO/ISOs and Large Public Utilities) Study commissioned by American Engineering University (2) NERC CIP Compliance Topical Study Small-Midsize utilities Up to 200,000 customers More than 100 U.S. and Canadian Utilities Participated Study commissioned by Cyber Security Specialist Firm (3) Newton-Evans Study of Cyber Security for Protection and Control Larger Utilities (North America and International) More than 60 utilities from 30+ countries Participating Commissioned by CIGRE JWG B5 D2.46 Slide 4 Control Systems Operations Managers and Senior Staffers Slide 5 % driven internally %Regulatory and reliability organizations Public Power44%56% Cooperative64%36% Investor-Owned51%49% Vendors48%53% ISO/RTO40%60% Summary48%52% 1. How much of the details of real-time operating procedures are driven internally from the organization, how much driven from reliability (Reliability Coordinator, Transmission Operator, Balancing Authority) or regulatory organizations (NERC, FERC, state)? Slide 6 2. Please rank the following types of information based on their importance in making decisions for various real-time procedures in each scenario. Use a scale of 1-5, with 1=most important and 5=least important (Using each number only once.) 1.12 3.08 3.52 2.64 4.64 1.40 3.00 3.56 2.60 4.44 1.48 3.60 3.76 1.92 4.24 1.12 2.92 3.44 2.80 4.72 0.001.002.003.004.005.00 SCADA data Contingency Analysis data State Estimation data Verbal communications Other information sourcesNormal operationEmergency operation Restorative operationPost contingency Slide 7 Normal operation SCADA data Contingency Analysis data State Estimation data Verbal communicationsOther information sources Public Power 1.113.334.222.004.33 Cooperative 1.252.753.752.255.00 Investor- Owned 1.142.863.432.864.71 Europe 1.003.002.004.005.00 Vendor 1.003.002.004.50 ISO/RTO 1.003.502.503.005.00 Summary 1.123.083.522.644.64 Emergency operation SCADA data Contingency Analysis data State Estimation data Verbal communicationsOther information sources Public Power 1.333.114.002.444.11 Cooperative 2.252.753.751.754.50 Investor- Owned 1.142.713.572.864.71 Europe 2.004.001.003.005.00 Vendor 1.002.503.004.504.00 ISO/RTO 1.004.003.002.005.00 Summary 1.403.003.562.604.44 Summary 1.122.923.442.804.72 Slide 8 Restorative operation SCADA data Contingency Analysis data State Estimation data Verbal communications Other information sources Public Power1.563.674.001.893.89 Cooperative2.003.004.001.005.00 Investor- Owned1.143.433.712.144.57 Europe1.004.003.002.005.00 Vendor1.504.003.003.503.00 ISO/RTO1.504.503.501.504.00 Summary1.483.603.761.924.24 Post contingency SCADA data Contingency Analysis data State Estimation data Verbal communications Other information sources Public Power1.113.003.892.564.44 Cooperative1.252.753.752.255.00 Investor- Owned1.142.713.712.714.71 Europe1.003.002.004.005.00 Vendor1.003.002.004.005.00 ISO/RTO1.003.502.003.505.00 Summary1.122.923.442.804.72 Slide 9 4. Who in the operational hierarchy executes the EMS applications, the higher or lower reliability authority? (Check all that apply) Slide 10 5. Do any of the entities checked above in question #4 run EMS applications and compare results? Yes, all involved 71% No 17% Other 12% Slide 11 6a. How are actions coordinated for events near the boundaries of Balancing Authority Areas or Reliability Coordinating Areas? (Check all that apply) For the survey group as a whole, verbal communications is the dominant method for coordinating events near the boundaries of BAAs or RCAs. ICCP, however, is also frequently used among all of the domestic utilities and RTOs. Slide 12 6b. How is corrective action decided and carried out? By a impressive margin (88%), corrective action is decided and carried out through cooperative decisions and actions by both the Balancing Authority and Reliability Coordinating Areas. 88% 8% 4% 0%20%40%60%80%100% Cooperative decisions and actions by both the Balancing Authority and Reliability Coordinating areas Only the Balancing Authority decides and takes action Only the Reliability Coordinating area decides and takes action Slide 13 6c. If just one entity decides and takes corrective action, what is the MAIN driver of this decision? (Pick one) Equipment responsibility or ownership (44%) is the main driver for the eighteen respondents to this question. However, this value increases to seventy-one percent (71%) if only the responses from the seven investor owned utilities are considered. 0% 44% 17% 22% 17% 0%20%40%60% Proximity of event to boundary (i.e. further away) Equipment responsibility or ownership Severity of event Time-criticalness of response Other Slide 14 8. What type of control center wall board do you use? tile/magnet ic2D video3D videootherTotal Public Power33028 Cooperative03014 Investor-Owned43017 Europe01001 Vendor22002 ISO/RTO02103 Summary9141425 Slide 15 9. What visualizations are most relevant during NORMAL SECURE (NORMAL) OPERATION of the grid? Ninety-two percent (24 out of 26) of the survey respondents rated Topological Visuals as Extremely Important during Normal Secure (Normal) Operation of the grid. Dynamically Colored Visuals were also viewed as Extremely Important to 58% of respondents. Slide 16 10. What visualizations are most relevant during NORMAL INSECURE (ALERT/CONTINGENCY) OPERATION of the grid? 27% 92% 15% 54% 31% 69% 8% 58% 8% 31% 35% 15% 12% 31% 15% 0% 54% 12% 54% 19% 62% 0%20%40%60%80%100% Geographical Visuals Topological Visuals Contour Visuals Tabular Visuals Animated Visuals Dynamically Colored Dynamically Sized Extremely Important Somewhat Important Not Important to Our Operations Slide 17 11. What visualizations are most relevant during EMERGENCY OPERATION of the grid? Slide 18 12. What visualizations are most relevant during RESTORATION OPERATION of the grid? Slide 19 13. Which type of display (large control center board or desk top screen) is an operator more likely to use in the following situations? Desk top computer screens are more likely to be used by a control room operator during all operation phases. However, during Normal and Post Contingency Operations the respondents tend to use them slightly more (69% and 73% respectively) than during Emergency and Restorative conditions where the use of a control center board significantly Increases (from 19% to 31%). Slide 20 Responses from the survey group find that visualizations on the operators desktop computer screen offer significantly more benefits than the control center board. The control center board is cited as being better for wide area viewing. 14. What is the difference between the visualizations on the control center board and the operators desk top computer screen? Slide 21 15. Are certain visualizations more appropriate or efficient on the control center board or operators desk top computer screen? Overall, responses to this question provided a little more balance when comparing visualizations on the control center board to operator desk computer screens. However, there are some significant differences in three areas: Tabular Visuals efficiency on operators desk computer screen received an overwhelming preference (84%) to those of control center boards, Geographical Visuals were found to be more appropriate for control center board use by a 3 to 1 margin, while Topological Visuals were better suited to operator desk top computers by a 2 to 1 margin. Slide 22 16. How are critical events visualized (e.g. operating limit violations, line tripping, generator tripping, etc)? (Check all that apply) The two visualization methods predominantly used by the twenty-six respondents for critical events are Blinking Values (81%) and Highlighted and Blinking Values (73%). IOUs indicated the highest use of Character Tag and Exception List. 81% 38% 73% 12% 42% 8% 23% 0%20%40%60%80%100% Blinking values Character Tag Highlighted and Blinking Values Zooming Exception List Panning Other Slide 23 20. Please rank the relevancy of the following data in the visualization of the grid from 1-6 where 1=most relevant and 6=least relevant (Using each number only once). Device states (2.21), Voltage (2.54) and Power (2.79) were ranked the most relevant data in the visualization of the grid by survey respondents. For the most part, this was pretty consistent among all of the survey groups with the exception of the one European utility. 2.54 4.00 2.79 3.58 2.21 5.88 0.002.004.006.008.00 Voltage Current Power Frequency Device states Other--> Slide 24 Study of Small-to-Mid-Size Utilities Regarding NERC CIP Topics Study undertaken Jan-Apr 2012 More than 100 Utilities Participated Having from 20,000 to 200,000 customers Slide 25 1. Does your utility have Critical Cyber Assets under NERC CIP? In spite of the seeming change in definitions of what is a critical cyber asset, two-thirds of the respondents indicated that they had NO cyber assets that are considered critical under current NERC CIP definitions. Slide 26 2. How much did your utility spend on cyber security Operations and Maintenance in 2011? Responses here were reported across all dollar ranges. More than one-third spent less than $25,000 per year on cyber security O&M in 2011. Just over one third spent from $25,000 to $200,000. Thirteen percent spent more than $200,000. Four respondents indicated that cyber security O&M was not a budgeted item. Slide 27 4. What were your utilitys capital expenditures for cyber security in 2011? One half of the respondents to this question reported spending less than $25,000 in capital expenditures for cyber security during 2011. Nearly one quarter stated that expenditures ranged from $25,000 to $200,000. Thirteen percent replied that they had invested more than $200,000 for cyber security items. Again four respondents replied that cyber security was not a separately budgeted CAPEX line item. Slide 28 6. Have utility work practices and procedures changed as a result of NERC CIP requirements? Seventy percent of all respondents indicated that utility work practices and procedures have changed as a result of NERC CIP requirements. Importantly, 40% of all respondents stated that work practices and procedures have changed significantly due to NER CIP requirements. Most of the 30% reporting no change in work practices and procedures hold the view that they do not have critical cyber assets as currently defined by NERC. Slide 29 If NERC CIP requirements have caused changes, please explain : Respondent #1 We have added workflows to the process to demonstrate/document compliance. Respondent #4 Station access procedures Respondent #5 Limiting and logging access to dispatch & other source areas; lots of documentation & audit preparation; lots of effort to ensure compliance but not necessarily improve security. Respondent #6 Installation of new SCADA system required improvements in physical access requirements Respondent #7 Maintenance of CIP rules is a massive and continuous undertaking. It took 14 FTE's to get through the most recent audit Respondent #8 Device installation, testing, access management, patch management Respondent #9 NERC CIP does not apply Respondent #10 Level of reporting and documentation requirements required have increased significantly. Respondent #12 All actions and occurrences have to be verified under CIP regulations Respondent #13 While our utility does not currently fall under version 4 of the CIP standards we are actively preparing for full compliance because we anticipate version 5 will affect us significantly. At the very least a proactive cyber security program is a good practice and enhances the reliability of both the BES and non-BES power systems. Slide 30 Respondent #19 Add a tremendous burden for security and logging of activities Respondent #20 Process to develop a security program has begin. We are taking small incremental steps. Respondent #24 Sign in sheets required into certain areas during certain time frames; escorts required into certain areas Respondent #25 Since NERC CIP requirements are not yet applicable, our efforts are in anticipation of future changes. However, such future changes shown in pending drafts of the NERC standards will have a VERY significant impact to work practices and procedures. Respondent #26 At this time, our utility will not consider substation LANs for IEDs & RTUs due to pending NERC requirements. Respondent #27 We are distribution but have made changes in anticipation of NERC/CIP Respondent #28 We evaluate each new technology initiative for its ability to put us into CIP requirements. Using communicating faulted circuit indicators as an example, we chose a hosted service rather than bringing the data in-house to avoid any potential CIP changes. Slide 31 7. Does your utility offer in-house training for cyber security? Nearly one half (47%) of the survey respondents reported that their utility offers some form of in-house training for cyber security. Another 17% plan to offer such in-house training by 2014. More than one third (37%) of the survey sample do not offer cyber security training on an in-house basis. Slide 32 9. Do you currently outsource any cyber security tasks to a third party? More than one half (53%) of the survey participants indicated that they DO outsource at least some cyber security tasks to outside services to third parties. Another 10% plan to do so by year-end 2014. Slide 33 12. What are the certification requirements that your employees must have to work with projects involving cyber security Perhaps surprisingly, more than three quarters of the responding utility officials reported that there are currently NO certification requirements for employees in order to work with projects involving cyber security topics. Of the handful of utilities that indicated one or more certifications as requirements, CISSP (15%), CISM (11%), CISA (7%) and Comp TIA (4%) were specifically cited. A few reported other requirements such as CISCO Systems in the listing below the chart. Slide 34 16. Have NERC CIP requirements caused your utility to increase the number of full time employees dedicated to cyber security activity? Yes, 23% No, 77% Have NERC CIP requirements caused your utility to increase the number of full time employees dedicated to cyber security activity? Slide 35 19. Which of the following cyber security technologies/methods do you currently use? Slide 36 Larger Utilities (North America and International) Participation from More than 60 utilities in 30+ countries Conducted with P&C Managers; Operations Managers; Some IT Management Involvement Slide 37 1a. Are you offering your P&C System personnel (engineers and field technicians) any cybersecurity training for their job? ?responsibilities? Slide 38 3. Have your P&C system personnel signed acceptable use policies? Slide 39 1a. Are you offering your P&C System personnel (engineers and field technicians) any cybersecurity training for their job? responsibilities? Slide 40 1b. Do you tailor cybersecurity training to address the issues related to job responsibility? Slide 41 2a. How would you consider the quality and completeness of cybersecurity training in your organization? Slide 42 2b. In your opinion, could your cybersecurity training be improved? Slide 43 3. Have your P&C system personnel signed acceptable use policies? Slide 44 4. Do you have a cybersecurity incident response plan for your P&C system? Slide 45 5. Do you monitor P&C system personnel access to and use of P&C system components? Slide 46 6. Do you test P&C system patches to correct cybersecurity defects prior to deployment? Slide 47 7a. Do you have adequate controls in place to monitor P&C system behavior in order to indicate that a security incident has taken place? Slide 48 7b. Do you benchmark or maintain a scorecard of P&C system cybersecurity incidents? Slide 49 8. What is your short list of cybersecurity solutions needed to protect your P&C systems? Need to understand vulnerabilitiesNeed to plan to fix them Segmentation of networksUpdated password maintenance Actually, we've implemented cybersecurity measures by following NERC-CIP guideline since 2007. However, only control system has been secured but not for protection system. Do not connect relays to the network. Currently as little as possible connection to the outside Web. All personnel have dedicated pc's to connect to P&C systems. Anti-virus systemPhysical separated networkRestricted firewall Password Policy - # of characters and 90 day expiration was implemented Access control (authorization, etc.), closed network configuration (net separation)Audit Log, Backup Antivirus & Firewall software etc. Security of the network access to P&C Security of the computer devices used as tools Security practices of the personnel Remote secure access (through SCADA or IP solution)Secure mobile local access to devices Restricted physical accessRestricted electronic accessTraining and awareness Enfording NERC-CIP standards usage Implementing firewalls in the substation Implementing anti-malware software in DCS P&C devices have no connectivity to any system. None at this time outside the substation. Remote accessProtocolIPS Security gatewaysFirewallsHardwired telephone switch Perimeter access control, both physical and electronic Intrusion detection and prevention software Centralized software patches and password management Security enforcement points Centralized configuration management system Extend our existing remote access system Slide 50 11a. Do you allow employees to use their personal devices (i.e. personal flash drive, smart phone, tablet, etc.) for P&C maintenance or configuring P&C components? Slide 51 11b. If NEITHER to the above, are you planning to allow employees to use their personal devices (i.e. personal flash drive, smart phone, tablet, etc.) for P&C maintenance or configuring P&C components? BYOD could affect views Slide 52 11c. Do you support programs loaded on employee personal devices? Slide 53 11d. What is your estimate of the percent of employees using personal devices for P&C maintenance? Slide 54 11e. What is your estimate of the percent of employees using personal devices for configuring P&C components? Slide 55 11f. Do you enforce security policies and encryption for employee personal devices? Slide 56 12a. Do you allow third party support technicians to use their personal devices (i.e. personal flash drive, smart phone, tablet, etc.) for P&C maintenance or configuring P&C components? Slide 57 12b. If NEITHER to the above, are you planning to allow third party support technicians to use their personal devices (i.e. personal flash drive, smart phone, tablet, etc.) for P&C maintenance or configuring P&C components? Slide 58 12c. Do you support programs loaded on third party support technicians' personal devices? Slide 59 12d. What is your estimate of the percent of third party support technicians using personal devices for P&C maintenance? Slide 60 12e. What is your estimate of the percent of third party support technicians using personal devices for configuring P&C components? Slide 61 12f. Do you enforce security policies and encryption for third party support technicians' personal device? Slide 62 13a. Are your P&C cybersecurity policies and procedures derived from regulatory requirements? Slide 63 13c. If no, from where are your cybersecurity policies derived? 49% 40% 52% 54% 60% 52% 22% 0% 29% 22% 40% 16% 76% 80% 74% 7% 0% 10% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% SummaryNorth AmInternational Utility guidelines Generally accepted industry guidelines Professional association recommendations P&C staff recommendations IT department guidelines Other Slide 64 14. From the list below, rank the inhibitors for implementing strong security policies for P&C system operations in order from 1-4, with 1=strongest inhibitor and 4=weakest inhibitor. SummaryNorth AmInternational Cost to maintain and operate a strong security system 2.482.042.79 Perimeter security provided by and supported by IT is adequate 2.952.833.03 Lack of interoperability between P&C system components 2.472.542.41 P&C system components do not incorporate strong security mechanisms 2.102.581.76 Slide 65 16a. Does your utility have the technology or business processes needed to manage role-based access control (RBAC) for P&C systems? Slide 66 Upcoming EMS/SCADA/DMS Study 4 th Quarter 2012 Study of Control Systems usage patterns and plans among the worlds electric power delivery utilities. We need your help for this study to serve as the bridge between what YOU need and want in control systems and what systems providers-vendors need to know in order to develop solutions to meet your needs. Slide 67 Other 2012 Research Topics U.S. Manufacturing Readiness for Smart Grid Cloud Computing Outlook for Small-Midsize Utilities and Usage of Specific IT/OT Applications Packages Substation Processing Platform Options Fault Current Limiting Devices U.S. Market for Bus Duct Assessment of American Manufacturing Industry Readiness for Smart Grid Roll Out Slide 68 Prepared by Chuck Newton Newton-Evans Research Company September 2012 Thanks for sitting in on this briefing! Slide 69 Appendix Slides 2012 Findings on Cyber Security and 61850 Usage and Plans Slide 70 Looking at Smart Grid Opportunities for Growth at Mid- Year 2012 .What stands in the Way? by Chuck on June 25, 2012 Why we believe the near-term investment priority for utilities of all types must be cyber security-related! (Security is not always considered part of smart grid spending)! This year, Newton-Evans Research has already undertaken a number of national and international studies of cybersecurity issues, and the findings lead us to believe that the single most critical issue facing utilities of all types is the near-term requirement to shore up cyber defenses, policies and procedures. Unfortunately, these cyber security investments will likely continue to usurp funding from other smart grid activities, but this investment must be a priority, in my opinion. Slide 71 Findings from Jan-Mar 2012 Survey of Protection Engineers Relay Protocol Use North AmericaInternational Slide 72 Extent of Use of IEC 61850 in Substations North AmericaInternational Slide 73 Findings from Jan-Mar 2012 Survey of Protection Engineers Features of IEC 61850 Being Used/Planned North AmericaInternational