Building world-class security response and secure development processes

David Jorm, Senior Manager of Product Security, IIX

#ODSummit

Outline

• Introduction

• SDN attack surface

• Recent OpenDaylight vulnerabilities

• Defensive technologies

• Security response best practices

• Secure engineering best practices

• OpenDaylight security: current status

• OpenDaylight security: vision

#ODSummit

Introduction

• Software engineer for 15 years, climatology domain

• Last 5 years focusing on security, mainly Java

• Led Red Hat's Java middleware security team

• Currently manager of product security for IIX, and a founding member of the ODL security response team

• Based in Brisbane, Australia (beautiful place, shame about the timezone)

#ODSummit

SDN Attack Surface

#ODSummit

SDN Attack Surface

• Traditional networks conflate the control and data planes on a physical device

• Software-defined networks factor the control plane out to a SDN controller

• The controller uses a protocol such as OpenFlow to control switches, which are now only responsible for handling the data plane

• Security advantage: easy segregation of the control plane network from the production data plane network

• Security disadvantage: the SDN controller's ability to control an entire network makes it a very high value target

#ODSummit

SDN Attack Surface

#ODSummit

SDN Attack Surface

• SDN controllers are also exposed via the data plane

• When an OpenFlow switch encounters a packet that does not match any forwarding rules, it passes this packet to the controller for advice

• As a result, it is possible for an attacker who is only able to send data through a switch to exploit a vulnerability on the controller

• We will see a real-life example later in the presentation

#ODSummit

SDN Attack Surface

#ODSummit

Recent OpenDaylight Vulnerabilities

#ODSummit

Netconf XXE (CVE-2014-5035)

• Netconf (and restconf) API processes user-supplied XML

• By default, Java XML parsers do not disable external entity processing

• This led to a textbook XXE vulnerability

• Example of vulnerable code, with the patch applied: controller/opendaylight/netconf/netconf-util/src/main/java/org/opendaylight/controller/netconf/util/xml/XmlUtil.java

#ODSummit

Netconf XXE (CVE-2014-5035)

#ODSummit

Topology spoofing via host tracking (CVE-2015-1610)

• Most SDN controllers include host tracking, allowing hosts to migrate between different physical locations in the network

• Host tracking is based on monitoring of Packet-In messages, and does not require any validation, authentication, or authorization to identify the host

• An attacker can impersonate a host and make the SDN controller believe it has migrated to a physical network location controlled by the attacker

• Data plane access is sufficient for exploitation, so long as the attacker knows the MAC address of the target host

• Not patched in ODL l2switch

• Paper: http://www.internetsociety.org/sites/default/files/10_4_2.pdf

#ODSummit

DoS in ONOS packet deserializer (CVE-2015-1166)

• When an OpenFlow switch encounters a packet that does not match any forwarding rules, it passes this packet to the controller for advice

• It was found that the packet deserializers in ONOS would throw exceptions when handling malformed, truncated, or maliciously-crafted packets

• The exceptions were not caught and handled properly

• The top-level I/O thread exception handler would then disconnect the relevant switch

• Proves that attacks from the data plane are possible!

#ODSummit

Defensive Technologies

#ODSummit

Topoguard

• The same research team that reported the topology spoofing flaw developed topoguard to mitigate it

• Doesn't add authn/authz, but instead verifies the conditions of host migrations

• A legitimate host migration would involve a Port Down signal before the host migration finishes. The host would also be unreachable at its old physical network location after the migration is complete.

• Currently tightly coupled to the Floodlight controller

#ODSummit

Security-mode ONOS

• A new feature in the ONOS Cardinal release

• Effectively a mandatory access control (MAC) implementation for ONOS applications

• Applications can be constrained by a policy dictating which actions they are permitted to perform

• A vulnerability in an ONOS application could not be exploited to perform actions that are not permitted by security-mode ONOS. This is similar to the protection SELinux provides for applications running on Linux systems.

• Could this approach be a good model for OpenDaylight?

#ODSummit

Proposed Controller Shield Project

#ODSummit

Security Response Best Practices

#ODSummit

Open Source Security Response

• All information public

• Not just source code: bug trackers, mailing lists, etc.

• Security requires the opposite approach – information must be kept private until patches are available

• How do you handle this in the context of an open source project?

• Good models: ASF, major OSS vendors like Red Hat and SuSE

#ODSummit

Open Source Security Response

• Dedicated mechanism for reporting security issues, separate to normal bugs

• Dedicated team with a documented process for responding to these reports

• Ability to quick build a patch asynchronous to normal release schedules

• Clear documentation of the issue in an advisory, including references to patch commits (advantage of open source)

#ODSummit

Open Source Security Response

#ODSummit

Proprietary Security Response

#ODSummit

Secure Engineering Best Practices

#ODSummit

Open Source Secure Engineering

• No well established best practices

• Few good examples in the open source world. Proprietary software currently does a much better job, for example Microsoft's SDLC.

• OpenStack is one good example

• Separate VMT and OSSG organizations

#ODSummit

Open Source Secure Engineering

#ODSummit

Open Source Secure Engineering

• Secure development guidelines (relies on developers to implement)

• Developer training (expensive and difficult to roll out in a virtual environment with many contributors)

• Automated QE/CI jobs to catch known-vulnerable dependencies

• Automated QE/Ci jobs to catch security issues and enforce standards, e.g. via static analysis

#ODSummit

OpenDaylight security: current status

#ODSummit

OpenDaylight Security Response

• Security reporting mechanism

• Dedicated team with a private mailing list and documented process for handling issues

• Security advisories page: https://wiki.opendaylight.org/view/Security_Advisories

• Advisories sent to mailing lists

#ODSummit

OpenDaylight Security Response

#ODSummit

OpenDaylight Security Response

• Scope currently limited to OpenDaylight code, not dependencies

• Handling dependencies would involve capturing a manifest, and tracking all relevant upstreams

• Based on my experience, this would require one full time resource to be feasible

• Vulnerabilities in dependencies are sometimes handled when they are reported to the security response team

#ODSummit

OpenDaylight Secure Engineering

• Great analysis performed in May 2014: https://wiki.opendaylight.org/view/CrossProject:OpenDaylight_Security_Analysis

• Unfortunately quite dated now, and not much progress has been made implementing the recommendations

• It's a lot of work to implement things, and who has time?

• Enter the OpenDaylight summer internship program!

#ODSummit

OpenDaylight Secure Engineering – Intern Project

#ODSummit

OpenDaylight Secure Engineering – Intern Project

Establish automated QE/CI jobs to catch security issues and regressions. This will involve integrating the findsecbugs tool into Gerrit/Jenkins.

Establish automated QE/CI jobs to catch known-vulnerable dependencies. This will involve integrating tools such as dependency-check and victims into Gerrit/Jenkins.

Document a threat model for OpenDaylight

Improve documentation to capture security best practices at installation and configuration time

#ODSummit

✔

✔

✔

✔

✔

OpenDaylight security: vision

#ODSummit

OpenDaylight Security Vision - Reactive

High performing security response team

Equipped to handle vulnerabilities in dependencies

Able to co-ordinate disclosure and patches for issues across the community development team and affected vendors of OpenDaylight distributions or products

Geographically distributed and able to quickly respond in all timezones

#ODSummit

✔

X

✔

✔

OpenDaylight Security Vision - Proactive

Documentation of best practices, threat model, etc.

Remove default credentials

Security hardening features applying a sandbox or MAC to the environment

Automated checks for known-vulnerable dependencies

Automated static analysis checks

Security training for developers: considering donating javapentesting.com course content to the community

#ODSummit

X

✔

X

X

✔✔

Questions?

#ODSummit