Critical Infrastructure Security: The Emerging Smart Grid
In Class: Cpt S 580-03, Cpt S 483-01, EE 582-02, EE 483-01
Online or Tri-Cities: Cpt S 580-03, Cpt S 483-01, Cpt S 483-02, EE 582-01, EE 483-01
Dr. Anurag K. Srivastava, Dr. Carl Hauser, Dr. Dave Bakken
Putting it together: network security practices for the (smart) grid
April 24, 2014
1
Acknowledgement
• I am enormously indebted to Andrew Wright of n-dimension solutions (www.n-dimension.com) for the work he has put into educating the power community about security and for permission to use materials and figures drawn from his presentations, including his TCIPG 2011 Summer School Lecture.
2
Source: NIST Smart Grid Framework and Roadmap, Conceptual Reference Diagram, January 2010
SCADA
HAN
Control Systems in the Power Grid
4
Historical ICS
• Proprietary
• Complete vertical solutions
• Custom
• Specialized communications – Wired, fiber, microwave, dialup, serial, etc.
– 100s of different protocols
– Slow; e.g. 1200 baud
• Long service lifetimes: 15–20 years
• Not designed with security in mind
5
Third Party Controllers, Servers, etc.
Serial, OPC or Fieldbus
Engineering Workplace
Device Network
Firewall
Services
Network
Third Party Application Server
Application Server
Historian Server
Workplaces Enterprise Optimization Suite
Mobile Operator
Connectivity Server
Control
Network
Redundant
Enterprise Network
Serial RS485
Modern ICS
IP
Internet
Enterprise Network
6
Technology Trends in ICS
• COTS (Commercial-Off-The-Shelf) technologies – Operating systems—Windows, WinCE, embedded RTOSes
– Applications—Databases, web servers, web browsers, etc.
– IT protocols—HTTP, SMTP, FTP, DCOM, XML, SNMP, etc.
• Connectivity of ICS to enterprise LAN – Improved business visibility, business process efficiency, meter to cash
– Remote access to control center and field devices
• IP Networking – Common in higher level networks, gaining in lower levels
– Many legacy protocols wrapped in TCP or UDP
– Most new industrial devices have Ethernet ports
– Most new ICS architectures are IP-based
7
• Some vulnerabilities are known and increasing; others unknown
• Consequences range from nuisance to severe
• Threat is ever changing
• Interdependencies and interconnections are the risk multipliers
Threat
Consequence Vulnerability
The Risk Equation
8
Sources of vulnerability in modern ICS
• COTS + IP + connectivity = many security risks
• All of those of Enterprise networks and more: Poor separation from enterprise nwk Legacy OSes and applications
No security monitoring Inability to limit access
Poorly secured 3rd party access Inability to revoke access quickly
Dialup modems Unexamined system logs
Unpatched systems Accidental misconfiguration
Limited use of anti-virus Improperly secured devices
Limited use of host-based firewalls Lack of security features
Improper use of ICS workstations Improperly secured wireless
Unauthorized applications Unencrypted links to remote sites
Unnecessary applications Passwords sent in clear text
Open FTP, Telnet, SNMP, HTML ports Password management problems
Fragile control devices Default OS security configurations
Network scans by IT staff Unpatched routers / switches
9
Known Security Problems in the Power Grid
• From NISTIR 7628
• 7.2 EVIDENT AND SPECIFIC CYBER SECURITY PROBLEMS 7.2.1 Authenticating and Authorizing Users to Substation IEDs 7.2.2 Authenticating and Authorizing Users to Outdoor Field Equipment 7.2.3 Authenticating and Authorizing Maintenance Personnel to Meters 7.2.4 Authenticating and Authorizing Consumers to Meters 7.2.5 Authenticating Meters to/from AMI Head Ends 7.2.6 Authenticating HAN Devices to/from HAN Gateways 7.2.7 Authenticating Meters to/from AMI Networks 7.2.8 Securing Serial SCADA Communications 7.2.9 Securing Engineering Dial-up Access 7.2.10 Secure End-to-End Meter to Head End Communication 7.2.11 Access Logs for IEDs 7.2.12 Remote Attestation of Meters 7.2.13 Protection of Routing Protocols in AMI Layer 2/3 Networks 7.2.14 Protection of Dial-up Meters 7.2.15 Outsourced WAN Links 7.2.16 Insecure Firmware Updates 7.2.17 Side Channel Attacks on Smart Grid Field Equipment 7.2.18 Securing and Validating Field Device Settings 7.2.19 Absolute & Accurate Time Information 7.2.20 Personnel Issues in Field Service of Security Technology 7.2.21 Weak Authentication of Devices in Substations 7.2.22 Weak Security for Radio-Controlled Distribution Devices 7.2.23 Weak Protocol Stack Implementations
7.2.24 Insecure Protocols 7.2.25 License Enforcement Functions 7.2.26 Unmanaged Call Home Functions
10
Known Security Problems in the Power Grid
• From NISTIR 7628 • 7.3 NONSPECIFIC CYBER SECURITY ISSUES
7.3.1 IT vs. Smart Grid Security 7.3.2 Patch Management 7.3.3 Authentication 7.3.4 System Trust Model 7.3.5 User Trust Model 7.3.6 Security Levels ... 7.3.33 Cyber Security Governance
NISTIR 7628, Guidelines for, Smart Grid Cyber Security, Volume 3, Chapter 7 11
Consequences • Loss of production
• Penalties
• Market breakdown (e.g. Enron)
• Lawsuits
• Loss of public trust
• Loss of market value
• Physical damage
• Environmental damage
• Injury
• Loss of life
• USSR pipeline explosion, 1982
• Bellingham pipeline rupture, 1999
• Queensland sewage release, 2000
• Davis Besse nuclear plant infection, 2003
• Northeast USA blackout, 2003
• Browns Ferry nuclear plant scram, 2006
• Stuxnet, 2010 $$$.$$ 12
Adversaries
• Script kiddies • Hackers • Organized crime • Competitors • Terrorists • Hactivists • Eco-terrorists • Nation states • Disgruntled insiders
– (security failures can make any of the others into an insider!)
13
Threat Model
• Targeted and untargeted threats – Targeted: human, specifically crafted worm/virus,
botnet
– Untargeted: generic worm/virus, script kiddy
• Assume adversary has: – Complete knowledge of network
– Beachhead in enterprise network
– Limited access to control network
– Limited physical access to field equipment
14
Of recent concern: APTs
• APT: Advanced Persistent Threats • Highly motivated teams of attackers with strong skills and
tenacity – Targeted: specific economic or political aims – Nation state backing – Exploit vulnerabilities very effectively
• Social engineering – spear-phishing • OS and other software vulnerabilities, including 0-day • Supply chain
• Goal: a beach-head in the victim’s systems from which further attacks can be launched at will – E.g. to steal data on an ongoing basis – To disrupt operations when the time is ripe
15
Fragile ICS Devices
• Many IP stack implementations are fragile
– Some devices lockup on ping sweep or NMAP scan
– Numerous incidents of ICS shut down by uninformed IT staff running a well-intentioned vulnerability scan
• Modern ICS devices are much more complex
– Some PLCs include web server for configuration and status
– More lines of code leads to more bugs
– Modern PLCs require patching just like servers
19
Unpatched Systems
• Many ICS systems are not patched current – Particularly Windows servers – No patches available for older versions of windows
• OS and application patches can break ICS • Uncertified patches can invalidate warranty • Patching often requires server reboot • Before installation of a patch:
– Vendor certification—typically one week – Lab testing by operator – Staged deployment on less critical systems first – Avoid interrupting any critical process phases
20
No Anti-Virus
• AV operations can cause significant system disruption at inopportune times
– 3am is no better than any other time for a full disk scan on a system that operates 24x7x365
• ICS vendors only beginning to support anti-virus
– Anti-virus is only as good as the signature set
– Signatures may require testing just like patches
21
Poor Authentication and Authorization
• Machine-to-machine comms involve no “user”
• Many ICS have poor authentication mechanisms and very limited authorization mechanisms
• Many protocols use cleartext passwords
• Many ICS devices lack crypto support
• Device passwords are hard to manage appropriately – Often one password is shared amongst all devices
and all users and seldom if ever changed
22
Poor Audit and Logging
• Many ICS have poor or non-existent support for logging security-related actions – Attempted or successful intrusions may go unnoticed
• Where IDS logs are kept, they are often not reviewed
• Various regulatory requirements are driving some change in this area – NERC—North American Electric Reliability Corporation
– FERC—Federal Energy Regulatory Commission
– DOE ARRA Funding cyber security requirements
23
Unmanned Field Sites
• Many unmanned field sites
• Some with high-speed connectivity to control center
• Most with poor authentication and authorization
• Many with dialup or wireless access
• Can be easy backdoor to the control center
24
Legacy Equipment
• Much legacy equipment
• Usually impossible to update to add security features
• Difficult to protect legacy communications
– but see IEEE P1711 for serial encryption
25
Unauthorized Applications
• Unauthorized apps installed on ICS systems can interfere with ICS operation
• Many types of unauthorized apps have been found during security audits – Instant messaging
– P2P file sharing
– DVD and MPEG video players
– Games, including Internet-based
– Web browsers
26
Inappropriate Use of ICS Systems
• Web browsing from HMI can infect ICS – Browser vulnerabilities
– Downloads
– Cross-site scripting
– Spyware
• Email to/from control servers can infect ICS – Sendmail and outlook vulnerabilities
• Disk storage exhaustion can crash OS – Storage of music, videos
27
Requirement for 3rd Party Access
• Firmware updates and PLC programming are sometimes done by vendor
– Many ICS have open maintenance ports
– Infected vendor laptops can bring down ICS
28
People Issues
• ICS network often managed by “Operations Department”, distinct from “IT Department” running enterprise network – ICS personnel are not IT or networking experts
– IT personnel are not ICS experts
• Significant fraction of control systems workforce is older and nearing retirement – Few young people entering this field
– Few academic programs
29
Security Assessments on ICS
• Various groups perform security assessments and penetration tests on ICS (generally under NDA) – Idaho National Labs – Sandia National Labs – N-Dimension Solutions – Other private organizations
• They always get in • Not a question of “if”, but “how long”
• N-Dimension prefers white-box assessments over
black-box penetration tests
30
Limited Information About Incidents
• Little information sharing about actual attacks – BCIT incident database has about 30 incidents per year vs.
100s of thousands of incidents per year in CERT database – Few cyber attacks on ICS for which details are public – National Electric Sector Cyber Security Organization
(NESCO) funded by DOE is ramping up a controlled sharing portal
• Difficult to estimate risk – Difficult to demonstrate ROI for security spending
• But… lots of data about significant financial losses in enterprise and e-commerce – Why would control systems be immune?
31
Other Issues
• Extreme environments
• Unusual physical and geographical topologies
• Many special purpose, limited function devices
• Static network configurations
• Multicast
• Long service lifetimes
• Much legacy equipment
32
How an Attack Proceeds—Step #1
Internet
Modem Pool
Web Server
Email Server
Business Workstation
Data Historian
Engineering Workstation
SCADA
RTU Control System Network
Enterprise Network
Database Server
Domain Name Server (DNS)
Enterprise Firewall
ICS Firewall
Attacker
PLC
PLC
Web Server
Management Console HMI
Initiated by Phishing or Spearphishing attack
33
How an Attack Proceeds—Step #2
Internet
Modem Pool
Web Server
Business Workstation
Data Historian
Engineering Workstation
SCADA
RTU Control System Network
Enterprise Network
Domain Name Server (DNS)
Enterprise Firewall
ICS Firewall
Attacker
PLC
PLC
Web Server
Management Console HMI
Email Server
Database Server
34
How an Attack Proceeds—Step #2b
Internet
Modem Pool
Web Server
Business Workstation
Data Historian
Engineering Workstation
SCADA
RTU Control System Network
Enterprise Network
Domain Name Server (DNS)
Enterprise Firewall
ICS Firewall
Attacker
PLC
PLC
Web Server
Management Console HMI
Email Server
Database Server
Initiated by Flash Drive
35
How an Attack Proceeds—Step #3
Internet
Modem Pool
Web Server
Business Workstation
Data Historian
Engineering Workstation
SCADA
RTU Control System Network
Enterprise Network
Domain Name Server (DNS)
enterprise Firewall
ICS Firewall
Attacker
PLC
PLC
Web Server
Management Console HMI
Email Server
Database Server
36
How an Attack Proceeds—Step #4
Internet
Modem Pool
Web Server
Web Server
Business Workstation
Data Historian
Management Console HMI
Engineering Workstation
SCADA
RTU Control System Network
Enterprise Network
Domain Name Server (DNS)
enterprise Firewall
ICS Firewall
Attacker
PLC
PLC
Vendor Web Server Email Server
Database Server
37
How an Attack Proceeds—Step #5
Internet
Modem Pool
Web Server
Web Server
Business Workstation
Data Historian
Management Console HMI
Engineering Workstation
FEP
RTU Control System Network
Enterprise Network
Domain Name Server (DNS)
enterprise Firewall
ICS Firewall
Attacker
PLC
PLC
Vendor Web Server Email Server
Database Server
38
How an Attack Proceeds—Step #5b
Internet
Modem Pool
Web Server
Web Server
Business Workstation
Data Historian
Management Console HMI
Engineering Workstation
SCADA
RTU Control System Network
Enterprise Network
Domain Name Server (DNS)
enterprise Firewall
ICS Firewall
Attacker
PLC
PLC
Vendor Web Server Email Server
Database Server
Initiated by Flash Drive on Control Network
39
How an Attack Proceeds—Step #6
Internet
Modem Pool
Web Server
Web Server
Business Workstation
Data Historian
Management Console HMI
Engineering Workstation
SCADA
RTU Control System Network
Enterprise Network
Domain Name Server (DNS)
enterprise Firewall
ICS Firewall
Attacker
PLC
PLC
Email Server
Database Server
40
How an Attack Proceeds—Step #7
Internet
Modem Pool
Web Server
Web Server
Business Workstation
Data Historian
Management Console HMI
Engineering Workstation
SCADA
RTU Control System Network
Enterprise Network
Domain Name Server (DNS)
enterprise Firewall
ICS Firewall
Attacker
PLC
Email Server
PLC
Database Server
41
How an Attack Proceeds—Step #8
Internet
Modem Pool
Web Server
Web Server
Business Workstation
Data Historian
Management Console HMI
Engineering Workstation
SCADA
RTU Control System Network
Enterprise Network
Domain Name Server (DNS)
enterprise Firewall
ICS Firewall
Command and Control
PLC
Email Server
PLC
Database Server
42
How are Power Grid ICS being Defended?
• Regulatory requirements
• Network design practices
• (Defense technology) – Firewalls, IDS & IPS, automated recovery systems, etc. (no time to discuss today)
43
Regulations: NERC CIP
• NERC – North American Electric Reliability Corporation
• FERC – Federal Energy Regulatory Commission
• NERC CIP – Critical Infrastructure Protection Standards
– Just one of several NERC standards related to reliably operating complex, interconnected power grids
44
NERC CIP Standards
• Industry writes standards with coordination by NERC; FERC approves them – Utilities have input to the process
– But ultimately FERC gets to say what is acceptable, as does its Canadian counterpart
• Standards aimed at protecting the Bulk Power System (transmission system) – therefore evolving how they apply to distribution entities
• Nearly 200 required documentation, management, and security practices; all must be auditable
• Fines for violation up to $1M/day!
45
NERC CIP Standards: Contents
• CIP-001-2a: Required reporting of incidents
– To ISO, to DOE, to NERC
• CIP-002-4: Critical Cyber Asset Identification
– Recognizes that some assets require greater protection than others: critical vs not critical
– Critical cyber assets are those that relate to critical power assets which are the things whose failure could disrupt reliable operation of the BPS
46
NERC CIP Standards: Contents (cont’d)
• CIP-003-4: Security Management Controls – Company executives are responsible
– Requires change control and configuration management
• CIP-004-4: Personnel and Training – People need to be aware of cybersecurity
– Background checks? Ongoing assessment of personnel for security risks
– Access control (cyber and physical)
47
NERC CIP Standards: Contents (cont’d)
• CIP-005-4a: Electronic Security Perimeters
– Identification of perimeters and ports
– Access control, logging, and monitoring
– Annual vulnerability assessments
• CIP-006-4d: Physical security of critical cyber assets
– Access control, logging and monitoring
48
NERC CIP Standards: Contents (cont’d)
• CIP-007-4: Systems Security Management – Testing when making changes
– Hardening prior to production use
– Patch management
– Malicious software prevention
– Account management
– Continuous security status monitoring
– Secure device disposal or redeployment
– Annual vulnerability assessments
49
NERC CIP Standards: Contents (cont’d)
• CIP-008-4: Incident Reporting and Response Planning – Have a plan to deal with cyber security incidents
– Know what has to be reported
– Document incidents
• CIP-009-4: Recovery Plans for Critical Cyber Assets – Have plan for backup and recovery
– Perform backup and recovery testing
50
Problem
• Standard largely addresses documentation and practices, not achievement of security – Nobody knows how to measure security
– If you can’t measure it, you can’t regulate it
• Consequence: attacks may succeed even when you are completely compliant with NERC CIP standards – Still worthwhile – less vulnerable, smaller
consequences, faster recovery (at least in theory!)
51
Defending ICS
• Separate control network from enterprise network – Harden connection to enterprise network
– Protect all points of entry with strong authentication
– Make reconnaissance difficult from outside
• Harden interior of control network – Make reconnaissance difficult from inside
– Limit single points of vulnerability
– Frustrate opportunities to expand a compromise
• Harden field sites and partner connections
• Monitor both perimeter and inside security events
• Monitor server and network behavior
• Periodically scan for changes in security posture
52
Availability, Integrity and Confidentiality
• Enterprise networks require C-I-A – Confidentiality of intellectual property matters most
• ICS requires A-I-C – Availability and integrity of control matters most – control data has low entropy—little need for
confidentiality – Many ICS vendors provide six 9’s of availability
• Ensuring availability is hard – Cryptography does not help (directly) – DOS protection, rate limiting, resource management,
QoS, redundancy, robust hardware with high MTBF
53
Defense in Depth
• Perimeter Protection – Firewall, IPS, VPN, AV
– Host IDS, Host AV
– DMZ
• Interior Security – Firewall, VPN, AV
– Host IDS, Host AV
– Application Whitelisting
– IEEE P1711 (AGA 12)
– NAC
• Monitoring – IDS
– Scanning
• Management
IDS Intrusion Detection System IPS Intrusion Prevention System DMZ DeMilitarized Zone VPN Virtual Private Network (cryptographic) AV Anti-Virus (anti-malware) NAC Network Admission Control
54
50000 Foot View Internet
Control Network
Field Site Field Site Field Site
Partner Site
VPN
VPN
FW
FW
IPS
IDS
IT Stuff
Scan
AV
FW IPS
P1711
FW
AV Host IPS Host AV
Proxy
Host IDS Host AV
IDS Scan
Enterprise Network
55
Control DMZ Architecture
• Enterprise Network contains typical business systems – Email, web, office apps, etc.
• Control DMZ provides business connectivity – Contains only non-critical systems that provide
connectivity btween Control and Enterprise Networks – Enforces separation between Enterprise and Control
Networks – May consist of multiple functional zones
• Separated by Firewall, IPS, Anti-Virus, etc.
• Control Network demarcates critical control systems – May consist of multiple functional zones
• Internally protected by Firewall, IDS, Anti-Virus, etc.
56
Control DMZ Perimeter Protection
Firewall with NAT Remote Access VPN Network Anti-Malware Intrusion Prevention
57
Control DMZ Design Principles
• Multiple functional security zones
• Traffic between zones undergoes firewall & IPS
• Only path in/out of Control Network
• Default deny for all firewall interfaces
• No/Minimal direct traffic across DMZ
• No common ports between outside & inside
• No control traffic to outside
• Highly limited outbound traffic
• No connections initiated from DMZ into Control Network
• Emergency disconnect at inside or outside
• No network management from outside
58
Control Network Design Principles
• Minimal number of connections to DMZ
• Control Network independent of DMZ, Enterprise – Separate Networking Hardware from DMZ
– Separate Time Server
– Separate AAA
– Allows emergency disconnect from DMZ
• QoS where applicable
• Redundancy where appropriate
59
Management
• Out-of-band management in DMZ
– Network and security management
– AAA servers
• Out-of-band local management in Control Network
– For disconnected operation
• In-band management for field sites
– Where local management is impractical
60
Securing Converged Smart Grid Networks
• no separate control network from enterprise network
• will use IP-based protocols, thus need IP-based defenses
• must consider defenses at all layers: – physical layer
– link layer
– internet layer
– transport layer
– application layer
• security will be a serious challenge!
• see Wright, Kalv, Sibery, “Interoperability and Security for Converged Smart Grid Networks”, Grid Interop 2010
61
References and links
• Andrew Wright, “Cyber Security for Grid Control Systems”, EEI Transmission, Distribution and Metering Conference, April 2009. http://www.eei.org/meetings/Meeting%20Documents/2009-04-05-Tues-3-Wright.pdf
• Jim Brenton. “Advanced Persistent Threats to the Electric Sector”, Grid Security Conference 2011, New Orleans, LA. http://www.nerc.com/files/7_Brenton_APT_NERC_GridSecCon_2011-10-19_DRAFT_1.pdf
• Robert H. McLanahan, “CIP Standards & Grid Reliability”, Grid Security Conference 2011, New Orleans, LA. http://www.nerc.com/files/3_GridSecCon_2011_10_19-McClanahan.pdf
• Andrew Wright, Paul Kalv, and Rodrick Sibery, “Interoperability and Security for Converged Smart Grid Networks”, Gridwise Forum 2010, http://www.gridwiseac.org/pdfs/forum_papers10/wrightpre_gi10.pdf
63
Top Related