Protecting Inherently Vulnerable Devices

OTCSA White Paper October 2019

2

Table of content Cyber security risks and challenges in the Industrial IoT era ......................................................... 3

Un-patchable Applications and Systems ...................................................................................................... 3

Insecure Industrial and Networking Protocols ............................................................................................ 4

“Air-gap” Weaknesses and IIoT Connectivity Risks ..................................................................................... 4

Inherently vulnerable OT devices: use cases ................................................................................ 4

Baseline protection recommendations ........................................................................................ 5

Visibility......................................................................................................................................................... 5

Trust Boundaries and Enforcement Points .................................................................................................. 5

Policy and Response ..................................................................................................................................... 5

Device Trust .................................................................................................................................................. 6

Implementation guidelines .......................................................................................................... 6

Asset Identification, Visibility, and Management ........................................................................................ 6

Telemetry ..................................................................................................................................................... 7

Internal Segmentation, Trust Boundaries, and Zones ................................................................................. 7

Detection and Incident Response ................................................................................................................ 7

Virtual Patching ............................................................................................................................................ 8

Device Hardening ......................................................................................................................................... 8

Key and Certificate Management ................................................................................................................ 8

Endpoint Protection Platform ...................................................................................................................... 8

Summary ..................................................................................................................................... 9

About the Operational Technology Cyber Security Alliance (OTCSA) ............................................ 9

Acknowledgements ................................................................................................................... 10

Use of information ..................................................................................................................... 10

3

Cyber security risks and challenges in the Industrial IoT era The lifetime of deployed technology and devices in an Industrial Control System (ICS) is often in the order of 10-15 years or longer1 depending on the type of technology, device, and industry. Many such legacy devices that have been operating for years or decades, like Programmable Logic Controllers (PLC), process sensors, gateways, and workstations, are no longer patchable and cannot be upgraded due to technical or operational constraints. They present a significant cyber security challenge and pose serious risks.

The United States Council of Economic Advisors estimates that malicious cyber activity costs the U.S. economy between $57 and $109B per year2. The NotPetya attack alone cost more than $10B, with A.P. Moller-Maersk and FedEx experiencing an estimated $300M and $400M respectively in damages3. Recent vulnerabilities such as Wind River VxWorks’ Urgent11 and Schneider Electric TRITON/TRISIS revealed weaknesses that compromised the safety and reliability of ICS.

Complicating matters, the move toward Industrie 4.0 and Industrial Internet of Things (IIoT) is increasing the level of connectivity of control systems to the Internet. Industrie 4.0 refers to the concept of factories in which machines are augmented with network connectivity and sensors, connected to a system that can virtualize the entire production line. The preponderance of legacy devices and the increase in industrial system connectivity has propelled an urgent need to protect Inherently Vulnerable Devices (IVD). This paper provides recommendations and guidelines for addressing security and safety concerns for common IVD use cases.

Common challenges of protecting IVDs include:

• Un-patchable applications and systems.

• Insecure industrial and networking protocols.

• “Air-gap” weaknesses.

• Increased IIoT connectivity.

UN-PATCHABLE APPLICATIONS AND SYSTEMS Workstations and devices using unsupported versions of the Microsoft Windows OS are especially vulnerable because they are no longer patchable. For example, mainstream and extended support have ended for Windows 3.11, 95, 98, XP, and Vista, which means Microsoft no longer provides bug fixes and security patches. As an OS reaches its end of life, application developers begin reducing support of their applications running on these older operating systems. It’s not uncommon for applications running on ICS devices and workstations to reach their end-of-life within the lifespan of the device on which it runs. A system may be un-patchable for a number of other reasons, including, limited maintenance windows and incompatibility with other system components.

1 Guide to Industrial Control Systems (ICS) Security, Special Publication 800-82 Revision 2, NIST, May 2015 2 The Cost of Malicious Cyber Activity to the U.S. Economy, White House Council of Economic Advisers, February 2018 3 The Untold Story of NotPetya, the Most Devastating Cyberattack in History, WIRED, August 22, 2018

4

Average lifespan of a device:

• Distribution transformer: 25 years.

• Substation switchgear: 9 - 36 years.

• Wind turbine: 20 - 25 years.

• Manufacturing device: 10 - 20 years.

INSECURE INDUSTRIAL AND NETWORKING PROTOCOLS Legacy devices may be using insecure industrial protocols. Protocols such as Modbus, CANBUS, BacNet, and DNP3 have very little inherent security for authentication or encryption. Industrial device data may be transported in clear text—unencrypted—with very weak authentication. Communicating securely with such devices is challenging and remotely managing keys or digital certificates can be difficult.

“AIR-GAP” WEAKNESSES AND IIOT CONNECTIVITY RISKS The protection provided by “air gaps”, or physically segmented networks, has proven to be ineffective. For example, the Stuxnet attack demonstrated that USB sticks and phishing emails could be used to transfer malware through the gap. Building Management Systems (BMS), ICS, and automated equipment typically have their own closed, proprietary networks that are physically separated from IT networks. These and other OT environments are still operating under air-gapped design criteria, while today’s IIoT initiatives are introducing additional risks.

To respond to these challenges, organizations must embrace multiple layers of defense, including: network segmentation, traffic content inspection, secure protocols, authentication, encryption, and endpoint protection. Close collaboration between IT and OT teams is crucial. Enterprises must bridge the gap between IT and OT teams that have different priorities and approaches to ensure that data privacy, safety, and reliability concerns are addressed.

Inherently vulnerable OT devices: use cases An Inherently Vulnerable Device (IVD) is any component with a known Common Vulnerabilities and Exposures (CVE) entry or insecurity that can be exploited due its inability to be updated or upgraded by design or due to external operational circumstances. A component is a host device, network device, software application, or embedded device. IVDs can be grouped into five use-case categories:

1. Devices or systems for which the original vendor does not provide maintenance or support (e.g., end of support or end of life).

2. Devices or systems that an original vendor supports but cannot be protected by available patches or upgrades due to operational or integration challenges.

3. Devices or systems that lack security features by design (e.g., single user sign-on, lack of encryption).

4. Devices or systems that have been compromised due to unauthorized access, user error, or zero-day vulnerabilities.

5

5. Devices or systems where the attack surface has expanded due to changes to the operating environment.

Organizations should develop cyber security strategies that address these common use cases.

Baseline protection recommendations There is no silver bullet for cyber defense. OT security problems cannot be solved simply by buying a “solution-in-a-box.” Protecting control systems requires technology, processes, and the active participation of people from both IT and OT that are involved with securing OT operations. It is also important that enterprises implement the following baseline protection strategies.

VISIBILITY Keeping an accurate and up-to-date asset inventory is a vital first step in any comprehensive cyber security program. However, the lack of visibility into ICS assets is common across industries worldwide. A 2017 SANS Institute survey4 found that 40 percent of ICS security practitioners “lack visibility or sufficient supporting intelligence into their ICS network.”

To make informed decisions on prioritizing spending and creating security plans to safeguard its employees, reputation, and bottom line, an organization needs to have complete knowledge of its assets and their vulnerabilities. Automated asset discovery, monitoring, anomaly detection, and intelligence can help mitigate cyber risks.

TRUST BOUNDARIES AND ENFORCEMENT POINTS When a device cannot protect itself, a practical measure is to implement trusted boundaries, trusted zones, or micro-segmentation to isolate it from other components in the environment—both to prevent infection of the IVD and to prevent any compromised device from spreading malware through the network.

Due to the increasing complexity of the cyber security landscape, network mitigations should go beyond isolating OT from IT. Micro-segmentation should be implemented whereby a “flat network” is replaced with separate Virtual Local Area Networks (VLAN) or network subnets that group components that communicate with each other into separate segments or zones. Here, only those components that need to communicate with resources outside the segment are able to do so.

POLICY AND RESPONSE Without a comprehensive cyber security policy, enterprises are at higher risk of attack. Defining a policy is critical to helping an organization outline which assets to protect from what threats, who should protect them, and what rules and controls should be implemented to protect assets and the enterprise. This empowers an organization to prevent or at least proactively address cyber security incidents and risks.

4 Securing Industrial Control Systems—2017, SANS Institute, July 2017

6

The following steps can be taken to build a policy definition and response framework:

1. Define a security zones strategy based on the security functionality segmentation.

2. Mitigate zero-day attack vectors using threat prevention between security zones with the ability to scan files downloaded or copied to the endpoints while integrating asset management processes with threat detection and prevention technologies for accurate traffic inspection. Such technical controls are typically deployed between Levels 3 and 4 of the Purdue Model (see the OTCSA’s Introducing the Operational Technology Cyber Security Alliance white paper—OTCSA, October 2019—for information about the Purdue Model).

3. Align security segmentation with data classification (e.g., highly restricted/reserved, restricted/classified, internal/general, public, untrusted) and business priorities (e.g., critical, significant impact/high risk, minor impact/medium risk, informational/low risk).

4. Define an identity governance strategy to provide centralized visibility and control to achieve sustainable compliance over who has access to what.

5. Group all users according to the roles of their functions.

6. Define a threat defense architecture to protect portable devices (e.g., barcode scanners) from advanced attacks and provide secure access to information and applications with secure containers.

DEVICE TRUST In addition to having robust IT and OT security strategies, it is essential to have one for devices. This includes implementing security controls on the device or at the gateway to protect IVDs and ensure device and data integrity, device and user authentication, secure communications, and confidentiality. Some IVDs are upgradeable, while others are not. Non-upgradeable devices should be placed behind a hardened gateway. Operators and manufacturers should harden IVDs by upgrading their firmware with mutual authentication, encryption, and stronger key and certificate capabilities.

Because private keys on devices can be compromised, it’s critical that organizations implement secure processes and systems to protect keys and digital certificates. Where possible, Public Key Infrastructure (PKI) and stronger authentication methods should be used in place of Pre-Shared Keys (PSK) and self-signed certificates.

Because the cyber security landscape is ever changing, enterprise policies and procedures should be developed, reviewed, and maintained on a regular basis. For more information, refer to the OTCSA’s Vulnerability Management for Operational Technology white paper (OTCSA, October 2019)

Implementation guidelines Protecting IVDs is possible. There are practical approaches that organizations can adopt that will reduce operational risks. The following implementation guidelines can assist asset owners, operators, systems integrators (SI), and vendors to protect IVDs in any OT environment.

ASSET IDENTIFICATION, VISIBILITY, AND MANAGEMENT There are a number of approaches to asset identification and visibility. Note, however, that some can be time consuming, costly, error prone, and risky. To improve visibility and detect unauthorized traffic and malware, organizations should implement passive and active scanning. Using passive network monitoring solutions can identify assets accurately, safely, and cost effectively—but they come with

7

technical limitations. With passive scanning, silent devices and software vulnerabilities may not be detected with a reasonable level of accuracy. Where devices and protocols are known, selective scanning can improve visibility and compensate for the limitations of passive scanning. Asset identification and visibility should be used to define the architecture and network segmentation and to secure processes and procedures for managing and updating inventories and potential device vulnerabilities.

TELEMETRY Telemetry involves the collection of process sensor, application, and system-level data and their automatic transmission to monitoring systems. Combined with other asset inventory and network-monitoring data, telemetry provides a broader view to improve situational awareness and correlate events to detect anomalies faster.

Telemetry plays an important role in automating response to improve device protection. For example, using telemetry, firewalls and other network devices can segment anomalous traffic or alert administrators to potential risks. For more information on telemetry, see the OTCSA’s Vulnerability Management for Operational Technology white paper (OTCSA, October 2019).

INTERNAL SEGMENTATION, TRUST BOUNDARIES, AND ZONES Internal network segmentation, or micro-segmentation, is the act of dividing a network into sub-networks, each of which becomes a network segment. Traffic between IVDs on each network segment can be limited by network characteristics such as a port, IP address, protocol, etc. Segmentation is a foundational element of layered network defense. While segmenting environments in enterprise network areas is common, the use of real-time and industrial protocols that may not run over Ethernet or IP-based networks, makes segmenting much more challenging in OT/shop floor environments. Segmentation can be implemented using OSI layers 2 and 3 technologies such as firewalls, security gateways, data diodes, VLANs, Virtual Routing and Forwarding (VRF), and Software-Defined Networks (SDN).

The Purdue Model defines zones and conduits that can be used to establish network segmentation policies. Once defined, trust boundaries can be established using micro-segmentation. Additional network access controls paired with network segmentation are a more effective approach to protection. While essential, segmentation can be difficult to fully implement in an OT environment without compromising reliability.

DETECTION AND INCIDENT RESPONSE Every asset owner and OT operator should have a security incident response team. Detection and incident response in an industrial OT environment are more complex than in IT environments. While blocking the source IP address of an attacking host rarely causes issues in IT networks, it is often not feasible in OT environments that rely on real-time network performance with minimal delay. However, strong monitoring functions in OT allow tapping into a safety culture that can be used to respond quickly to operational incidents.

An organization should develop its own playbook depending on its needs and relevant risk ratings. Stakeholders should establish clear incident response policies and procedures. They should also perform periodic drills to test the effectiveness of their processes and review them regularly to ensure they are updated to account for new threats and their corresponding responses.

8

VIRTUAL PATCHING In many cases, patching systems with known vulnerabilities may not be possible due to the lack of a patch or lack of a maintenance window for patching a device. In such cases, virtual patching can effectively protect an IVD. Where the vulnerability’s signature is known, a virtual patch can be applied using Intrusion Detection and Protection Services (IDS/IPS). An inline IDS/IPS system can match signatures against vulnerabilities and modify or discard packets based on the implementation approach. Using this approach, the network protects the device by emulating a patch.

DEVICE HARDENING IVDs and especially legacy devices may not support strong authentication, encryption, and confidentiality. Components such as PLCs, gateways, and embedded devices often lack mutual authentication to verify the identities of both the client and server. Without mutual authentication, data coming from an unauthenticated device cannot be trusted. Such devices are vulnerable to replay attacks or cloning. Where possible, organizations should harden the device's firmware to support mutual authentication and 2-way certificate pinning. Use of other authentication methods such as Extended Authentication (EAP) or RADIUS is also recommended to provide an additional factor of authentication for passwords and tokens.

When IVD endpoint devices cannot be upgraded, organizations should consider upgrading gateways and network devices to isolate the I/O and protect devices in a zone. Oftentimes, gateways lack device integrity and security controls to make them tamper resistant. Industrial gateways, in particular, should include features that can make them tamper resistant by protecting the device, data, applications, and communications. Hardening an industrial gateway helps to ensure that devices can communicate securely outside the zone.

KEY AND CERTIFICATE MANAGEMENT When applications, operating systems or firmware cannot be patched, devices become vulnerable. One approach to protecting IVDs is to increase the frequency of updating device credentials, such as keys and digital certificates. Organizations should consider automating remote management of keys and device credentials in a way that does not alter the application. Special care should be taken to implement an update process that isn’t prone to human error or compromise. Authentication is based on the trustworthiness of a device’s credentials. When those credentials have been compromised or stolen, a device is vulnerable to Man-in-the-Middle (MITM) attacks in which one device impersonates the identity of another.

ENDPOINT PROTECTION PLATFORM An Endpoint Protection Platform (EPP) is deployed on server-based endpoint devices to protect against known vulnerabilities as well as zero-day exploits. It complements network-based attack mitigation techniques and adds a layer of security for a “defense-in-depth” approach for ICS assets. Typically, an EPP relies on a mix of tactics like antivirus, memory protection, behavioral techniques, and system hardening. These solutions are particularly effective in protecting devices running unsupported operating systems such as Windows XP and Server 2000 and 2003. Care should be taken to upgrade EPP software to ensure that antivirus and malware detection signatures are always updated.

9

Summary IVDs pose a substantial risk factor in industrial environments. Protecting them is critical to managing cyber security in OT systems. The growth of IIoT and Industrie 4.0 is increasing the attack surface by connecting previously closed networks to the Internet. To protect systems, organizations need to contend with a variety of unique and frequently changing conditions, including un-patchable software, unsupported operating systems, insecure protocols, and cloud connectivity.

To respond to these challenges, organizations should adopt the baseline recommendations and implementation guidelines presented in this OTCSA white paper. These practical guidelines will help protect IVDs against cyber attacks that compromise the safety and reliability of OT systems.

About the Operational Technology Cyber Security Alliance (OTCSA) The Operational Technology Cyber Security Alliance (OTCSA) is a group of global industry-leading organizations focused on providing operational technology (OT) operators with resources and guidance to mitigate their cyber risk in an evolving world. Founded in 2019, OTCSA is the first group of its kind to architect a technical and organizational framework–the who, what, and how–for safe and secure OT. Membership is open to all OT operators and IT/OT solution providers. Current members include ABB, BlackBerry Cylance, Check Point Software Technologies, Forescout Technologies, Fortinet, Microsoft, Mocana, NCC Group, Qualys, SCADAfence, Splunk Technology, and Wärtsilä.

To learn more about the OTCSA or becoming a member, visit https://otcsalliance.org.

10

Acknowledgements The following people served as contributors in the preparation of this document:

Name Affiliation

Peter Corrao Wärtsilä

Aleksander Cwik ABB

Bart de Wijs ABB

Shlomi Feldman Check Point Software Technologies

Dharmesh Ghelani Qualys

Massimiliano Mandolini Forescout Technologies

Ofer Shaked SCADAfence

Damon Small NCC Group

Dean Weber Mocana

Davide Zanetti ABB

Use of information Copyright 2019 Operational Technology Cyber Security Alliance (OTCSA)

Redistribution and use of this document AS IS, without modification, is permitted provided that the following conditions are met:

1. Redistributions of this work of authorship must retain the above copyright notice, this license and conditions, including the disclaimer listed below.

2. The name(s) of the copyright holder, the Operational Technology Cyber Security Alliance (OTCSA), or any of its members or contributors may not be used to endorse or promote any products or other offerings, without specific prior written permission.

THIS DOCUMENT IS PROVIDED BY THE OTCSA, COPYRIGHT HOLDER(S) AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE OTCSA, COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS DOCUMENTATION, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.