Cybersecurity T&E and the National Cyber Range › images › pdf › conferences ›...

Cleared for Public Release 23 March 2017 Case # 17-S-1267


Cybersecurity T&E and

the National Cyber Range

Prepared for

2nd ITEA Cyber Security Workshop

"Challenges Facing Test and Evaluation “

24 March 2017

Prepared by

National Cyber Range Team

Peter H. Christensen

Range Director, NCR

Use or disclosure of data contained on this sheet is subject to the restrictions on the title page of this document.

What, Why, How?

• What do we want to accomplish?• Look back to where we were

• Provide insight into TRMC Cybersecurity Test and Evaluation

Infrastructure and the National Cyber Range

• Highlight Lessons learned and NCR successes

• Make some predictions about the future!

• Why is this important?• Cyberspace T&E is extremely challenging

• TRMC has “Operationalized” the NCR over six years of intense

operations

• NCR Team has delivered significant value added to DOD customers

• The future looks promising!

• How will we do it?• Look back to 2009 IA Policy Crosswalk and 2010 IT Acquisition

Reform

• Highlight TRMC/NCR progress and new T&E capabilities

• Look ahead to the future

2


Looking Back

2009: Information Assurance (IA) Policy

Cross Walk

• Proposed by DISA T&E Exec

• Chartered by DOT&E and DASD DT&E

• Examine current IA related T&E policies, directives, instructions and guidance

• Provide findings and recommendations

3


Key Findings from 2009

Information Assurance Policy Cross Walk

• Key Findings

• Focus on coordinating IA Test and Evaluation Activities

• Form Integrated Test Teams Early and enable “test by one, use by many”

• Promote Collaboration among Acquisition, Engineering and Test Teams

• Operationally Realistic IA Test Environment is Crucial to Successful Testing

• Threat Portrayal During Testing Must Reflect Current Threat Information

• Provide Adequate Resources & Expertise Essential for Testing

• Promote Acquisition-Related IA and Computer Network Defense (CND) T&E

as Critical to Ensure Secure DOD Systems

IA and CND testing in acquisition is a critical activity and can only be accomplished with

adequate resources and threat characterization!

4


2010 NDAA Section 804

IT Acquisition Reform

• Congress directed DoD to develop a new “IT Acquisition Process

• DCMO “Section 804” Task Force established

• T&E and Certification Government Lead

• Principal Deputy DASD DT&E

• Test Evaluation and Certification (TE&C) Process collaboratively

developed

• Supported an 18-month release goal, embraced both SE discipline

and Agile Methods

• Delivered to DCMO 01 Oct 2010

• Industry and DOD have embraced Agile

• DOD 5000 incorporates Incrementally Fielded Software Intensive

Programs

• Agile methods influenced “proposed” Cybersecurity T&E process

2010: “Strong Leadership is required to progress forward!”5


Key Elements of 2010 Draft TE&C

Policy/Process

• Agile TE&C execution

• Tailored to the IT acquisition

• Responsive to evolving requirements

• Risk based and mission focused

• TE&C infrastructure and tools

• Provides “testing as a service”

• Development, deployment, and sustainment

• Verified, Validated, and Accredited

(VV&A) Infrastructure

• Replicates an operational/production

environment

• Provides repeatable and defensible test results6


March 2017 TRMC Organization

Chief Financial Officer

Mr. Derrick Hinton (Acting)

Dir, TRMC

DD, T&E Range Oversight *

Agency RO **

Army RO **

AF RO **

Navy RO **

PM, CTEIP DD, Major Initiatives and Technical Analyses *

Principal Deputy, TRMC

Mr. Paul Mann (Acting)

Under Secretary of Defense for Acquisition, Technology & Logistics

Mr. “Jimmy” MacStravic (Acting)

Deputy EA for Cyber Test Ranges

PM, REP

Deputy PM, CTEIP

Range Director, NCR

Dr. Robert N. Tamburello

Deputy Range Director, NCR

Chief Operating Officer **

Sr. Security Advisor Admin. Officer

PM, T&E/S&T

Deputy PM, T&E/S&T

PM, JMETC

Director, TENA SDA* Supervisor

** Team lead

Updated 12 Dec 2016

7


TRMC Team Includes Government,

FFRDC, SETA And Contractor

• Infrastructure and Services

Include:

– Test Bed Design Support

– Integration of Custom Assets

− Software

− Hardware

− Wired and Wireless

− Remote Red/Blue Team Support

– Cyber and Testing Expertise

– Threat Vector Development

– Custom Traffic Generation

– Custom Sensor and Visualization

Support

– Custom Data Analysis

– End-to-End Test Support

Customers Identify Cyber T&E Requirements

TRMC Provides Infrastructure And Services To Satisfy Them!8


TRMC Cybersecurity T&E Infrastructure Components:

JMETC MILS and Regional Service Delivery Points (RSDPs)

• JMETC MILS provides secure connectivity for Cybersecurity T&E

• MILS VPN hosted on Defense Research and Engineering Network

• Peers with Joint Information Operations range

• Regional Service Delivery Points (RSDP)

• Computing and Storage Assets that host virtual Cyberspace Environments

• TRMC provides tools and services and instrumentation for traffic generation,

visualization, integrated event management, collaboration

• Geographically dispersed to minimize latency

and maximize usability

• Modular/Flexible architecture evolves with requirements

• TRMC provides Support Staff

• Available to help plan, design and execute events

TRMC will provision the right Cyber T&E Infrastructure based upon your needs!

9


What Is The National Cyber Range?

10

NCR Provides Cybersecurity Testing And Training As A Service!


NCR Executed 140 Events in Six Years

NCR 2016 Aviation Week Program Excellence Winner!

Helping developing programs understand, mitigate cyber-attack and prepare people to

defend U.S. systems

Note: FY-16: NCR unavailable for 9 Weeks during A&A

11


Cybersecurity Testing Lessons

Learned

1. Start Small and grow

2. Testing is an important Engineering and Design Tool that can be

used to refine requirements

3. Cyber Table Top is an effective tool to understand Mission Risks and

prioritize testing

4. Focus Cybersecurity Testing on the Mission!

5. Cybersecurity Testing must be executed with key IT Staff, Incident

Responders and Protection Teams

6. Customers need Cybersecurity T&E “As a Service”

7. Collaborative Approach to Coordinate Test Planning Critical

8. Effective Test Teams understand Cyber Offense and Defense

9. Automated Tool Suite creates efficiencies in event design,

development and deployment

10. Connectivity makes range location irrelevant

Lessons Learned from 140 + NCR Events

12


Cyber Testing

Engineering and Design Tool

• Cybersecurity Testing is Systems Engineering Tool

• Reduces technical debt, Identifies exposed vulnerabilities and provides engineering

alternatives

• Identifies “New” Cyber Requirements, exposes “Residual” Vulnerabilities!

13

No System will ever be 100% Secure! What is the Mission Risk?


• Phases are iterative and incremental! • Initial Phases reduce Type 1 Debt!

• Complements System Security Engineering and Risk Management

Framework SE and RMF Activities

• Later Phases reduce Type 2 Debt!

• Promotes Understanding of Mission Risk!

Cybersecurity T&E Helps Manage “Mission”

Risk

14

RMF Manages Acquisition Program Risk….Cybersecurity T&E Manages Mission Risk


Cyber Table Top (CTT) Effective Tool To

Prioritize Cybersecurity T&E

• What is a Cyber Table Top?

• Low technology, low cost, intellectually intensive wargame

• Introduces and explores Mission Effects of Offensive Cyber Ops

• Helps estimate “Mission Risk” to System, SoS or FoS

• Why is it used?

• Identifies Threat Vectors, Potential Vulnerabilities and Mission Risk

• Identifies new Functional and Non Functional Requirements

• Scopes the size and scale of Cybersecurity Testing

• Provides actionable recommendations

• What does it produce?

• Prioritization of vulnerabilities based upon likelihood of exploitation

and consequences to the mission

• High Risk Vulnerabilities must be evaluated first

• Medium Risk Vulnerabilities can be evaluated subsequently

• Low Risk Vulnerabilities may be deferred for later

• Cybersecurity Risk Matrices and Recommendations for

Cybersecurity Testing and Vulnerability Mitigation

15

CTTs Informed Several Events At The NCR!


Cyber Testing and Training Demand is for

Customer Centric Services (predominantly)

Customers come to the TRMC/NCR for Higher Tier “Live Cyber Environments,” Cyber

T&E expertise, and event support

Customers Need Cybersecurity T&E

“As A Service”

16


Automated Tools Simplify Event Design,

Development, Deployment/Redeployment!

• Reusable Content Includes:

• ADNS Emulation

• Round-robin NTP

• Full DNS infrastructure

• Whois

• Various Exchange Server

versions and architectures

• DNS registrar

• Webmail

• eCommerce sites

• Content Management

Systems (CMS)

17

Model Reuse Is Creating Efficiencies!


Connectivity Makes Range Location

Irrelevant!

• NCR demonstrated ability to

support Major Training Exercises!

• Remotely supported 1000’s of Users

• Connected numerous Logical Ranges

• 100’s of Enclaves & Subnets

• Thousands of Nodes

• NCR demonstrated ability to

support remote Testing

• NCR has both JMN and JIOR

Connectivity!

• Used remotely for multiple customers

Realistic Mission

Environments

JMNRSDPs

PSDPs

18

Connectivity Enables Customers To Create A “Virtual Enterprise”

For Testing And Training!


Pathfinder Event:

Avionics Bus Architectures

• NCR Objective

• Emulate Non IP Based Networks, Bus

Architectures: MIL Std 1553, ARINC 429 for

Cyberspace T&E

• Relevance

• Cyber Threats demonstrated ability to exploit

vulnerabilities in systems, subsystems and

components

• Some testing cannot/should not be conducted w/

actual aircraft, particularly in-flight

• Outcomes

• TRMC/NCR collaborated with NAVAIR to conduct a

Cyber Table Top to understand Cyber Risks

• NCR built-out Avionics Management System to

execute “Risk Reduction Event”

• Collaborated with Aircraft Cyber Threat Working

Group

19


Pathfinder Event:

Control Systems Cyber Security (CS2) Challenge

• NCR Objective

• Demonstrate ability to emulate representative

Industrial Control Systems Architecture

• Relevance

• Internet of Things makes Control Systems

connected and exploitable

• Developers fail to consider Cybersecurity

• Control Systems need to be tested in real-world

environments

• Outcome

• OSD Energy, Installations and Environment (E&EI)

requested NCR implement a representative DOD

Building Environmental Control System

• NCR successfully developed environment

• Follow on demonstrations will evaluate commercial

products

20

The “Internet of Everything” Exposes Control Systems to Attacks Not Previously Considered!

Image Source: Mario Morales (IDC)


What Does the Future Look Like

for Infrastructure?• Demand for Cybersecurity T&E Infrastructure will continue to increase

• NCR/RSDP/JMN Complex being provisioned to satisfy increasing demand

• Way Ahead: Leverage TRMC Investment

• Ensure future investments in automation tools are

Interoperable/Compatible/Un-encumbered

• Move Legacy SILs HW&SW, ICS/SCADA Labs, etc. to JMN

21



for Workforce?

• Demand for Cybersecurity T&E Workforce will continue to increase

• People are the most valuable and limited resource

• Workforce must have Acquisition and Cyberspace Skills

• Capabilities Development, Program Management, Contracting,

Systems Security Engineering, Risk Management Framework and

Cybersecurity T&E, Cyber Defense and Offense

• Way Ahead: Invest in “Wet Ware”

• Enhance Government Cybersecurity T&E Workforce

• Enable them with robust FFRDC, SETA and Industry Personnel

22

Procuring Cybersecurity T&E Infrastructure Is Easy

Greater Challenge: Developing The Cybersecurity T&E Workforce!



In Practice?

• Investments in T&E Infrastructure reduce

“Technical Debt” : PMs realizing

schedule/performance improvements

• Way Ahead:

• Provision infrastructure to support full

spectrum testing

• Not just one Program of Record

• Use incremental/iterative T&E to evaluate

System Functional and Non Functional

Requirements

• Autonomous Systems could adopt a

similar approach

• Adversarial Assessments should be Mission

Based Events

• Fully exercise Systems of Systems

• Include Cybersecurity Defense Providers

23

TRMC Is making Essential Investments!



For The Adversary?• Adversaries are sophisticated, persistent and getting better

• Conduct extended and sophisticated Cyber Campaigns

• Speed and agility enables them to get inside the

Acquisition Lifecycle

• DOD Acquisition systems and approaches lack agility

to keep up with the threat

• Industry has adopted Agile Methods/DEVOPS!

• Robust DEV/Test Infrastructure essential to support

continuous innovation

• Increases efficiency and “Delivery Velocity” enhances

Cybersecurity Posture and reduces ownership costs

• Enables Industry to evolve at the same pace as the

adversary

24

Infrastructure Investments Enable Agile Methods/DEVOPS!



For The Adversary? (cont.)• Way Ahead:

• Investment in Cybersecurity T&E

Infrastructure is having positive impact

• TRMC S&T and CTEIP Programs enablers

to help disrupt the Adversary Lifecycle

• Enable Agile Methods and DEVOPS with

“High Fidelity” reusable emulations/models of

Systems and Enterprise

• Exploit virtualization, Software Containers

• Consider provisioning “Digital Twins”

• Evolve with system as it matures

• Promote closer community engagement

• Development Community, Cybersecurity

Defense Providers and Cyber Mission

Forces

25

Effort Needed To Bring Together Development, Testing And Training Communities!


Summary

• Since 2009: DOD T&E Community has significantly advanced the practice of Cybersecurity T&E

• Past approaches to address Cybersecurity have created Technical Debt!

• Shift Left is Helping Programs improve Cybersecurity Posture

• Other Federal Agencies are adopting similar approaches!

• TRMC and the NCR are helping Testing and Training Customers!

• Deliver unique cybersecurity test, evaluation, and training capabilities

• Enable DOD to conduct focused cybersecurity test and evaluation

• Events are tailored to meet program requirements throughout the systems acquisition lifecycle

• TRMC is investing in the future!

• Workforce and Infrastructure investments are key enablers

• Without them advances in practice process cannot be achieved!

26

TRMC/NCR Team Are Making It Harder For The Adversary!



Superb effort organizing this workshop!

PROGRAM CHAIR - Ms. Chris Susman

SURVICE Engineering Company

PROGRAM TECHNICAL CHAIRS

Mr. Robert Laughman, US Army Evaluation Center

Duane Wilson, Ph.D., Wilson Innovative Solutions LLC

EXHIBITS & SPONSORSHIPS

Ms. Cathy Pritts and Mr. Jim Myers

Special Thanks!

27


The 2017 CG Classic

In Honor of Seaman Aaron N. Redd,

USCG• Please join the Chief Petty

Officers Association

Alexandria Chapter and the

family & friends of

• Aaron N. Redd, on Friday,

June 16th, as we host a

fun filled day of golf at the

Potomac Shores Golf

Club.

• All proceeds will be

donated to the Coast

Guard Enlisted Memorial

Foundation.

• http://www.cgclassic.com

28



Peter H. Christensen

Range Director, National Cyber Range

TRMC Office Phone: 571-372-2699

TRMC Email: [email protected]

Dr. Robert N. Tamburello

Deputy Range Director, National Cyber Range

TRMC Email: [email protected]

Address:

4800 Mark Center Drive

Suite 07J22

Alexandria, Va. 22350

Questions?

mailto:[email protected]

mailto:[email protected]

Cybersecurity T&E and the National Cyber Range › images › pdf › conferences ›...

Documents

Transcript of Cybersecurity T&E and the National Cyber Range › images › pdf › conferences ›...