Time Traveling: Adapting Techniques from the Future to Improve ReliabilityJacob KitchelJanuary 14, 2014

Presentation Title2

Bio

Present:

Security Architect at Exelon

Past:

Security & Compliance at Industrial Defender

ICS Risk Assessment (PT, VA, etc.)

Application Security research (Project Basecamp)

Enterprise Security Operations & Monitoring

Speaker (S4, EnergySec, ISA, API IT Security)

Hilarious LinkedIn Endorsements

Presentation Title3

Abstract

Technology in ICS environments lags the Enterprise by 10-15yr. This often leads to ICS companies having to stand by while other more nimble institutions are able to take advantage of new technology. What few people realize, is that our industry gets to watch the future happen out on the Internet and then pick and choose the best techniques to adapt and bring back in time.

Presentation Title4

How far have we come?

We have:• Compliance

• Incidents?

• Specialization

• Conferences

• Big Headlines?

• A LOT of vulnerabilities

Presentation Title5

Where has it gotten us?

Here we are:

• Multiple revisions of compliance requirements

• Basic improvements in security monitoring

• SOME patching happens

Presentation Title6

What is working against us?

Mountains or mole hills?• Refresh cycles

• “If it isn’t broken, don’t fix it”

• Skill set(s)

• Unknown unknowns

• Security v. Operations

• Budgets & time

Presentation Title7

Progress is sloooooowwwww….

Presentation Title8

What to do?

• Where do operations goals and security goals intersect?

• What is the lowest common denominator?

• What can have an impact?

It’s all about the customer…If you aren’t solving customer pain, then you aren’t doing anything

Presentation Title9

It’s about the customer

Operations

• Safety

• Reliability

• Uptime

Security

• Security

• Compliance

• Vulnerabilities

Where do these two areas intersect?

Customer

Presentation Title10

Where do Security and Operations Intersect?

• Patching

• Change Management

• Configuration Management

In other words…• Time-intensive

• Error-prone

• High-risk activities

Presentation Title11

Solving “Customer” problems lets you solve security

How can we do that?

Presentation Title12

Take a step back…to the future!

Presentation Title13

How?

Is there anyone that “looks” like us?Has anyone solved this problem before?

How can we:

• Reduce time commitments required

• Reduce errors

• Reduce risk

Presentation Title14

Know any of these names?

Presentation Title15

Internet-scale companies

• Millions of customers, world-wide

• High-availability, (near) zero downtime

• Complacency is death

• Some of the brightest minds >40

• Solving scale and complexity problems that we can barely imagine

• Leveraging software and hardware to dynamically define environments

• Have to be reliable and fast

Presentation Title16

How are they doing this?

They are doing it CONTINUALLY.

Continuous Delivery:

Changes to your environment are

proven to be deployable with predictable results

Presentation Title17

But you say, “There’s a catch!”

Continuous Delivery was popularized by Internet companies!

Internet companies deliver software and/or services as their products!

They’re not like us! We have a physical process!

Etc, etc, etc…

Guess What?Continuous Delivery is a collection of tools and processes – tools and processes that you use to focus your ability to deliver your physical process

Hint: You’re not getting off that easy! ;)

Presentation Title18

What does this mean to us?

• Major reduction in time and effort to push changes

What would a major time/effort reduction mean to your operations?• 500hr task takes 5 hours or 5 minutes?

• 40hr task takes 4hr or 4 minutes?

• How many times do all of your tasks gets repeated annually?

• What if you could save half of that time and effort?

Presentation Title19

How do we get there?

Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation Automation

Presentation Title20

Continuous Delivery in Practice

How do you move a mountain?

Presentation Title21

First steps first

• Follow your build/development process & write it all down• What takes the most time?

• What tasks are the most error-prone?

• What tasks require the most human intervention?– Automate these tasks FIRST!

• What tasks cause headaches or are time sinks?– Automate these next!

Presentation Title22

Facilitate Adoption

• Put everything into version control

• Add tests to verify that changes work

• Manage servers with configuration management tools

• Monitor EVERYTHING

Presentation Title23

Tools

• Software-defined infrastructure

• Monitoring

• Continuous Integration

• Version Control

• Code Review

• Configuration Management

• Orchestration

• Dashboards

End Goal• Quality

• Reliability

• Speed

Presentation Title24

Tool Specific Information

Presentation Title25

Software-defined Infrastructure

Tool example:

• Quali Systems TestShell

How to apply:

• Define common network architecture and system objects

• Create test topology

• Run tests and see what breaks, verify what works

Presentation Title26

Version Control

Tool examples:

• Git

• SVN

• CVS

How to apply:

• Track versions of clear-text configuration files

• Firewall, switch, router configuration files

• Application configuration files

Presentation Title27

Configuration Management

Tool examples:

• Puppet

• Chef

• Ansible

• Salt

• Microsoft SCCM

How to apply:

• Store all configurations in management tool

• As machines run, configuration management tool ensures declared configuration

Presentation Title28

Orchestration

Tool examples:

• Puppet

• Chef

• Mcollective

• Ansible

• Capistrano

• WinRM

How to apply:

• Determine order of components

• Leverage tools to operate, deploy, and automatically configure systems in proper order

Presentation Title29

Virtualization

Tool examples:

• Most common tool here is VMWare and is likely your vendor’s approved virtualization provider

How to apply:

• Mirror Dev, Test, and Production environments

• Bonus: backup/redundant assets

• Can begin to act as a “do over” button

Presentation Title30

Metrics & Dashboards

Tool examples:

• Logstash

• Graphite

• Nagios

• Cactii

How to apply:

MONITOR EVERYTHING

Presentation Title31

Continuous Delivery tool

Tool example:

• Thoughtworks Go

How to apply:

• Automate and streamline the build-test-release cycle

Presentation Title32

Automated Testing

Tool examples:

• Thoughtworks Twist

• BDD/TDD tools

How to apply:

• Write tests to verify functionality

• Run tests automatically every time new code, features, or configuration changes are made