Cyber Security for the Power Grid:

Cyber Security SolutionsFor <Client Name>

Cyber Securityfor the Power Grid:

Cyber Security Issues & Securing Control Systems

Andrew Wright CTO, N-Dimension

[email protected]

ACM CCS Conference TutorialNov. 2009

Power Grid Communications & Control Systems

borr

owed

from

NIS

T S

mar

t Grid

Tw

iki

Internet Control Systems

Agenda

• High-Level– Industrial Control Systems and Cyber Security Issues– Securing Control Systems

• Detailed– Security Issues in Industrial Control Systems– Today’s Threats– Securing Control Systems

A Control System

Sensor(s) +Actuator(s) +Controller(s)

Types of Industrial Control Systems (ICS)

Supervisory Control And Data Acquisition (SCADA)

Automation

Process Control Systems (PCS)

Distributed Control Systems (DCS)

Historical ICS

• Proprietary• Complete vertical solutions• Customized• Specialized communications

– Wired, fiber, microwave, dialup, serial, etc.– 100s of different protocols– Slow; e.g. 1200 baud

• Long service lifetimes: 15–20 years• Not designed with security in mind

Third PartyControllers,Servers, etc.

Serial, OPCor Fieldbus

EngineeringWorkplace

Device Network

Firewall

Services

Network

Third Party Application Server

ApplicationServer

HistorianServer

WorkplacesEnterprise

OptimizationSuite

MobileOperator

ConnectivityServer

Control

Network

Redundant

Enterprise Network

Serial RS485

Modern ICS Trends

IP

Internet

Enterprise Network

Technology Trends in ICS

• COTS (Commercial-Off-The-Shelf) technologies– Operating systems—Windows, WinCE, embedded RTOSes

– Applications—Databases, web servers, web browsers, etc.

– IT protocols—HTTP, SMTP, FTP, DCOM, XML, SNMP, etc.

– Networking equipment—switches, routers, firewalls, etc.

• Connectivity of ICS to enterprise LAN– Improved business visibility, business process efficiency

– Remote access to control center and field devices

• IP Networking– Common in higher level networks, gaining in lower levels

– Many legacy protocols wrapped in TCP or UDP

– Most new industrial devices have Ethernet ports

– Most new ICS architectures are IP-based

New IP-Based Industrial Control Systems

• ODVA (Rockwell)• Profinet• Foundation Fieldbus HSE• Telvent• ABB 800xA

• Honeywell Experion• Emerson DeltaV• Yokogawa VNET/IP• Invensys Infusion• Survalent

• IP to the Control Network or even Device Network• Not all are fully compatible with “ordinary IP”

Security Risks to Modern ICS

• COTS + IP + connectivity = many security risks• All of those of Enterprise networks and more

Worms and Viruses Legacy OSes and applications

DOS and DDOS impairing availability Inability to limit access

Unauthorized access Inability to revoke access

Unknown access Unexamined system logs

Unpatched systems Accidental misconfiguration

Little or no use of anti-virus Improperly secured devices

Limited use of host-based firewalls Improperly secured wireless

Improper use of ICS workstations Unencrypted links to remote sites

Unauthorized applications Passwords sent in clear text

Unnecessary applications Default passwords

Open FTP, Telnet, SNMP, HTML ports Password management problems

Fragile control devices Default OS security configurations

Network scans by IT staff Unpatched routers / switches

When ICS Security Fails

• Loss of production

• Penalties

• Lawsuits

• Loss of public trust

• Loss of market value

• Physical damage

• Environmental damage

• Injury

• Loss of life• USSR pipeline explosion, 1982

• Bellingham pipeline rupture, 1999

• Queensland sewage release, 2000

• Davis Besse nuclear plant infection, 2003

• Northeast USA blackout, 2003

• Browns Ferry nuclear plant scram, 2006$$$.$$

ACM CCS Tutorial

Nov. 2009

So How Do We Secure

Industrial Control Systems?

There is No Silver Bullet!

No Silver Bullet!

Defense in Depth

• Perimeter Protection– Firewall, IPS, VPN, AV– Host IDS, Host AV– DMZ

• Interior Security– Firewall, IDS, VPN, AV– Host IDS, Host AV– IEEE P1711 (AGA 12)

– NAC– Scanning

• Monitoring• Management

IDS Intrusion Detection SystemIPS Intrusion Prevention SystemDMZ DeMilitarized ZoneVPN Virtual Private Network (cryptographic)AV Anti-Virus (anti-malware)NAC Network Admission Control

Internet

Enterprise Network

Control Network

Field Site Field Site Field Site

PartnerSite

VPN

VPNFW

FW

IPS

IDS

IT Stuff

Scan

AV

FWIPS

P1711

FWAV

Host IPS Host AVProxy

Host IDS Host AV

IDS

Scan NAC

NAC 62351

Log Mgmt

Event Mgmt

Reporting

50000 Foot View

IT Stuff

VPN

ACM CCS Tutorial

Nov. 2009

Security Issues in

Industrial Control Systems

Availability, Integrity and Confidentiality

• Enterprise networks require C-I-A– Confidentiality of intellectual property matters most

• ICS requires A-I-C– Availability and integrity of control matters most– control data has low entropy—little need for confidentiality– Many ICS vendors provide six 9’s of availability

• Ensuring availability is hard– Cryptography does not help (directly)– DOS protection, rate limiting, resource management, QoS,

redundancy, robust hardware with high MTBF

• Security must not reduce availability!

DoS and DDoS Attacks

• Denial of Service (DoS) attack overwhelms a system with too many packets/requests– Exhausts TCP stack or application resources– Defenses include connection limits in firewall

• Distributed Denial of Service (DDoS) attack coordinates a botnet to overwhelm a target system– No single point of attack– Requires sophisticated, coordinated defenses– Weapon of choice for hackers, hacktivists, cyber-extortionists

• DoS, DDoS particularly effective when Availability is critical, i.e. against ICS

Fragile ICS Devices

• Many IP stack implementations are fragile– Some devices lockup on ping sweep or NMAP scan– Numerous incidents of ICS shut down by uninformed IT staff

running a well-intentioned vulnerability scan

• Modern ICS devices are much more complex– Some IEDs include web server for configuration and status– More lines of code leads to more bugs– Modern IEDs require patching just like servers

Unpatched Systems

• Many ICS systems are not patched current– Particularly Windows servers– No patches available for older versions of windows

• OS and application patches can break ICS– OS patches are tested for enterprise apps

• Uncertified patches can invalidate warranty• Patching often requires system reboot• Before installation of a patch:

– Vendor certification—typically one week– Lab testing by operator– Staged deployment on less critical systems first– Avoid interrupting any critical process phases

Limited use of Host Anti-Virus

• AV operations can cause significant system disruption at inopportune times– 3am is no better than any other time for a full disk scan on a

system that operates 24x7x365

• ICS vendors only beginning to support anti-virus– Anti-virus is only as good as the signature set– Signatures may require testing just like patches

• AV may be losing ground in enterprise deployments– impact on hosts, endpoint security not getting better– virus writers have learned to test against dominant AV

• application whitelisting can be a good alternative– enumerate goodness rather than badness

Poor Authentication and Authorization

• Machine-to-machine comms involve no “user”• Many ICS have poor authentication mechanisms

and very limited authorization mechanisms• Many protocols use cleartext passwords• Many ICS devices lack crypto support• Sometimes passwords left at vendor default• Device passwords are hard to manage appropriately

– Often one password is shared amongst all devices and all users and seldom if ever changed

– This is happening AGAIN in Smart Meter deployments!

Poor Audit and Logging

• Many ICS have poor or non-existent support for logging security-related actions– Attempted or successful intrusions may go unnoticed

• Where IDS logs are kept, they are often not reviewed• Various regulatory requirements are driving some

change in this area– NERC—North American Electric Reliability Corporation– FERC—Federal Energy Regulatory Commission– Sarbanes Oxley and PCAOB (Public Company Accounting

Oversight Board)– FISMA—Federal Information Security Management Act

Unmanned Field Sites

• Many unmanned field sites• Many with dialup access• Some with high-speed connectivity to control center• Most with poor authentication and authorization

backdoor to the control center!

Legacy Equipment

• Much legacy equipment• Usually impossible to update to add security features• Difficult to protect legacy communications

– but see IEEE P1711 for serial encryption

• Password protection is weak• Little or no audit and logging

Unauthorized Applications

• Unauthorized apps installed on ICS systems can interfere with ICS operation

• Many types of unauthorized apps have been found during security audits– Instant messaging– P2P file sharing– DVD and MPEG video players– Games, including Internet-based– Web browsers

Inappropriate Use of ICS Desktops

• Web browsing from HMI can infect ICS – Browser vulnerabilities– Downloads– Cross-site scripting– Spyware

• Email to/from control servers can infect ICS– Sendmail and outlook vulnerabilities

• Disk storage exhaustion can crash OS– Storage of music, videos

Little or No Cyber Security Monitoring

• internal monitoring is essential to detect low profile compromises– IDS– port scanning– vulnerability scanning– system audit

• without internal monitoring don’t know whether systems have been compromised

Requirement for 3rd Party Access

• Firmware updates and PLC, IED programming are sometimes done by vendor– Many ICS have open maintenance ports– Infected vendor laptops can bring down ICS

• Partners may require continuous status information– Partner access is often poorly secured– Partner channels can serve as backdoors

• 3rd parties may include:– ISO, transmission provider or grid neighbor,

equipment vendor, emissions monitoring service or agency, water level monitoring agency, vibration monitoring service, etc.

People Issues

• ICS network often managed by “Control Systems Department”, distinct from “IT Department” running enterprise network– ICS personnel are not IT or networking experts– IT personnel are not ICS experts

• Majority of control systems workforce is older and nearing retirement– Few young people entering this field– Few academic programs

Harsh Environments

• Temperature

• Vibration

• Dust

• Humidity

• Electrical Transients

Attack Vectors into Control Systems

Includes Infected Laptops and Is Growing

Source: 2003–2006 data from Eric Byres, BCIT

Security Assessments on ICS

• Various groups perform security assessments and penetration tests on ICS (generally under NDA)– Idaho National Labs– Sandia National Labs– N-Dimension Solutions– Other private organizations

• Vulnerability assessments always uncover problems• For penetration tests, we always get in

– Not a question of “if”, but “how long”

Other Issues

• Unusual physical topologies• Many special purpose, limited function devices• Static network configurations• Multicast• Long service lifetimes

For More Information ...

• See Smart Grid Cyber Security Strategy and Requirements, NISTIR 7628, www.nist.gov/smartgrid– particularly Appendices C and D

ACM CCS Tutorial

Nov. 2009

Today’s Threats

Hiroshima, 2.0 – Cyberspying of theUS Electric Grid (April 09)

Cyberspies penetrate electrical grid (April 09)

'Smart Grid' vulnerable to hackers (March 09)

CIA: Hackers Have Attacked Foreign Utilities (Jan 2008)

President Obama: securing the electric infrastructure is a national security priority (June 09)

Smart Grid Security Frenzy: Cyber War Games, Worms and Spies in Smart Grid (June 09)

earth2tech.com

Intense Media Visibility on the Cyber Security Issue

Limited Information About Incidents

• Little information sharing about actual attacks– BCIT incident database has about 30 incidents per year vs.

100s of thousands of incidents per year in CERT database– Few cyber attacks on ICS for which details are public

• Little information sharing about actual vulnerabilities– some are not easily or rapidly fixed– assessments are done under NDA

• Difficult to estimate risk– Difficult to demonstrate ROI for security spending

• But… lots of data about significant financial losses in enterprise and e-commerce– Why would control systems be immune?

Accidents Happen ...

Attacks Can Cause Similar Results

INL National Lab Aurora Demonstration, March 2007

Regulators provide Smart Grid Stimulus Funding criteria - cyber security is mandatory (June 09)

FERC releases Smart Grid Policy - cyber security mandatory for Utility rate recovery (July 09)

Strengthened Cyber Security Standards Approved for North American Utilities (May 09)

AMI-SEC working group developed security requirements for AMI

AMI-SEC Task Force

NIST developing interoperability and security standards for Smart Grid

Ontario Green Energy Act Drives Smart Grid With Security (May 09)

Cyber Security Regulatory Requirements

ACM CCS Tutorial

Nov. 2009

SecuringControl Systems

Adversaries

• Script kiddies• Hackers• Organized crime• Disgruntled insiders• Competitors• Terrorists• Hactivists• Eco-terrorists• Nation states

How an Attack Proceeds—Step #1

Internet

Modem Pool

Web Server

Email Server

BusinessWorkstation

DataHistorian

EngineeringWorkstation

FEPRTU Control

SystemNetwork

EnterpriseNetwork

Database Server

Domain NameServer (DNS)

enterpriseFirewall

ICSFirewall

Attacker

IED

IED

Web Server

Management Console HMI


Internet

Modem Pool

Web Server

BusinessWorkstation

DataHistorian


FEPRTU Control

SystemNetwork

EnterpriseNetwork


enterpriseFirewall

ICSFirewall

Attacker

IED

IED

Web Server


Email Server

Database Server


Internet

Modem Pool

Web Server

Web Server

BusinessWorkstation

DataHistorian



FEPRTU Control

SystemNetwork

EnterpriseNetwork


enterpriseFirewall

ICSFirewall

Attacker

IED

IED

Vendor Web ServerEmail

Server

Database Server


Internet

Modem Pool

Web Server

Web Server

BusinessWorkstation

DataHistorian



FEPRTU Control

SystemNetwork

EnterpriseNetwork


enterpriseFirewall

ICSFirewall

Attacker

IED

IED

Email Server

Database Server


Internet

Modem Pool

Web Server

Web Server

BusinessWorkstation

DataHistorian



FEPRTU Control

SystemNetwork

EnterpriseNetwork


enterpriseFirewall

ICSFirewall

Attacker

IED

Email Server

IED

Database Server

Defending ICS

• Separate control network from enterprise network– Harden connection to enterprise network– Protect all points of entry with strong authentication– Make reconnaissance difficult from outside

• Harden interior of control network– Make reconnaissance difficult from inside– Avoid single points of vulnerability– Frustrate opportunities to expand a compromise

• Harden field sites and partner connections– mutual distrust

• Monitor both perimeter and inside events• Periodically scan for changes in security posture

Internet

Enterprise Network

Control Network

Field Site Field Site Field Site

PartnerSite

VPN

VPNFW

FW

IPS

IDS

IT Stuff

Scan

AV

FWIPS

P1711

FWAV

Host IPS Host AVProxy

Host IDS Host AV

IDS

Scan NAC

NAC 62351

Log Mgmt

Event Mgmt

Reporting

50000 Foot View

IT Stuff

VPN

Logical Overlay on SP99 / Purdue Model of Control

Site Business Planning and Logistics Network

BatchControl

DiscreteControl

SupervisoryControl

HybridControl

SupervisoryControl

Enterprise Network

Patch Mgmt

Web Services Operations

AV Server

Application Server

Email, Intranet, etc.

ProductionControl

HistorianOptimizing

ControlEngineering

Station

ContinuousControl

Terminal Services

Historian (Mirror)

Site Operations and Control

Area Supervisory

Control

Basic Control

Process

ControlZone

EnterpriseZone

DMZ

Level 5

Level 3

Level 1

Level 0

Level 2

Level 4

HMI HMI

Logical Architecture

• Enterprise Zone contains typical business systems– Email, web, office apps, etc.

• DMZ provides business connectivity– Contains only non-critical systems that need access to both

Control and Enterprise Zones– Enforces separation between Enterprise and Control Zones– Consists of multiple functional sub-zones

• Separated by Firewall, IPS, Anti-Virus, etc.

• Control Zone demarcates critical control systems– Consists of multiple functional sub-zones

• Internally protected by Firewall, IDS, Anti-Virus, etc.

How NOT to connect Control / Enterprise

• Dual-homed server• Dual-homed server with Host IPS / AV• Router with packet filter ACLs• Two-port Firewall• Router + Firewall combination

• See NISCC Good Practice Guide on Firewall Deployment for SCADA and Process Control Networks, NISCC and BCIT, Feb 2005

Web Services Operations

ApplicationServer

HistorianMirror

DMZ

DMZ—Logical View

PatchMgmt

AVProxy

Terminal Services

No Direct Traffic

Emergency Disconnect

Emergency Disconnect

Multiple Functional Sub-Zones

VPN

IPS

Scan

FWAV

Host AV

Proxy

Host IPS

IDS

IDS

DMZ Design Principles

• DMZ contains non-critical systems• Multiple functional security sub-zones• Traffic between sub-zones undergoes firewall (& IPS or IDS)• DMZ is only path in/out of Control Zone• Default deny for all firewall interfaces• No direct traffic across DMZ• No control traffic to outside• Limited outbound traffic from Control Zone• Very limited inbound traffic to Control Zone• No common ports between outside & inside• Emergency disconnect at inside or outside• No network management from outside• Cryptographic VPN and Firewall to all 3rd party connections

DMZ Implementation (1)

DMZ LAN 3

DMZ LAN 4

DMZ LAN 2

NAT

RoutingFWIPS

SecurityApplianceWithMultiplePorts

DMZ/ControlInterconnect WAN/LAN

EnterpriseLAN

Anti-VirusProxy

Host IPS / Anti-virus

DMZ Implementation (2)

dot1qtrunkDMZ VLAN 3

DMZ VLAN 4

DMZ VLAN 2

NAT

RoutingFWIPSVLAN

SecurityAppliance

VLAN-capable L2 switch


EnterpriseLAN

Anti-VirusProxy

Host IPS / Anti-virus

NOT L3!

DMZ Implementation

• Sub-zones implemented by physical LANs or VLANs– Physical LANs require multi-port Security Appliance– VLANs require:

• VLAN-capable Security Appliance and Switch• anti-VLAN hopping protections on switch and FW• NO L3 (routing) on switch

• FW implements policy between– DMZ LANs, Enterprise Zone, Control Zone

• Anti-virus proxy controls outbound HTTP and/or FTP access to enterprise or Internet resources

• Host IPS and/or Host Anti-virus protects DMZ servers

Remote Access

DMZ

AAAServer

CertificateAuthority

Terminal Services


EnterpriseLAN

Remote Access Pool

Remote Access

VPN

Remote Access

• Security Appliance terminates Host-to-site VPN into remote access pool– IPSEC VPN, SSL VPN, PPTP VPN

• Authenticates user via:– AAA server, LDAP, Active Directory, etc.– Can enforce use of multi-factor hardware token

• Time-varying password tokens for vendor access

• Clients use VNC, Citrix, or Remote Desktop (RDP) to connect to Terminal Server

• Then VNC, Citrix, RDP, or Control System Apps to Control System Servers

Control Zone—Logical View

BatchControl

DiscreteControl

SupervisoryControl

HybridControl

SupervisoryControl

ProductionControl

HistorianOptimizing

ControlEngineering

Station

ContinuousControl

Site Operations and Control

Area Supervisory

Control

Basic Control

Process

ControlZone

Level 3

Level 1

Level 0

Level 2HMI

HMI

DMZ

Control Zone Design Principles

• Multiple functional security sub-zones• Firewall and IDS between sub-zones• Minimal number of connections to DMZ• Control Zone independent of DMZ, Enterprise

– Separate Security Appliance from DMZ– Separate Time Server– Separate AAA– Allows emergency disconnect from DMZ

• Cryptographic VPN and Firewall to all offsite IP connections (Field Site or Partner)

• IEEE P1711 for all offsite serial ICS connections• Host IDS, Host AV, or app whitelisting where feasible• Management only from management zone

Control Zone Implementation—Hierarchical

• Fast routing between VLANs via L3 switch

• ACLs between VLANs but no Stateful Firewall

Level 1

Level 2

Level 3

ControlZone

dot1q Trunks

L3L3

L2 L2

QoS, Shaping, PolicingPort Security

Gigabit

10/100

DMZ/Control Interconnect WAN/LAN

SPANIDS

Scan

FWFW

Host IDS Host AV

Control Zone Implementation—Ring

• Ring reduces wiring for linear sites like power dams

• but spanning tree can have problems with large rings

Level 1

Level 2

Level 3

ControlZonedot1q Trunks

L3L3

L2 L2

QoS, Shaping, PolicingPort Security

Gigabit

10/100

DMZ/Control Interconnect WAN/LAN

SPANIDS

Scan

FWFW

Host IDS Host AV

FirewallIDS/IPS

Client VPN

ProxyNetwork AVHost IDS/IPS

NAC

Site-to-site VPN

DMZ

Perimeter Protection in Utilities

IDSPort ScanVuln Scan

FirewallNAC

SCADA VPN

FirewallSCADA VPN

Port ScanIDS

Interior Protection in Utilities

LogAnalyzeReport

Compliance

ManagedSecurity

Monitor, Log, Analyze, Report

• Planning, processes, procedures, physical security, etc. are also important

• NERC CIP Regulatory Requirements provide reasonably good guidance in this area:

• CIP-001: Sabotage Reporting• CIP-002: Critical Cyber Asset Identification• CIP-003: Security Management Controls• CIP-004: Personnel & Training• CIP-005: Electronic Security Perimeters• CIP-006: Physical Security• CIP-007: Systems Security Management• CIP-008: Incident Reporting & Response Planning• CIP-009: Recovery Plans for Critical Cyber Assets

See www.nerc.com -> Standards -> Reliability Standards -> CIP

Beyond Network Security

Summary

• Today’s ICS are mix of modern and legacy– vulnerabilities due to both

lack of security design in legacy and security issues in newer equipment

• Defense in depth is essential– both perimeter (DMZ) and

interior security are crucial

• Regulation and government action is driving change

• Smart Grid must be designed with strong security

ACM CCS Tutorial

Nov. 2009

Thanks!

[email protected]

Standards Efforts

• NERC CIPs• NIST Smart Grid Interoperability Standards Project• NIST SP800-82• NIST SP800-53• NIST PCSRF Protection Profiles• AMI-SEC• ISA SP99• ODVA

• IEEE P1711 (AGA 12) -- serial SCADA encryption

A Few References

• www.nist.gov/smartgrid• Securing Your SCADA and Industrial Control

Systems, Version 1.0, DHS, ISBN 0-16-075115-8• Guide to SCADA and Industrial Control System

Security, NIST SP800-82• ISA99 Industrial Automation and Control Systems

Security, www.isa.org/MSTemplate.cfm?MicrositeID=988&CommitteeID=6821

• AGA 12/IEEE P1689 SCADA Encryption Standard, scadasafe.sf.net

Cyber Security for the Power Grid:

Documents

Transcript of Cyber Security for the Power Grid: