Dr. Steven J. Hutchison Principal Deputy Developmental Test and...
Transcript of Dr. Steven J. Hutchison Principal Deputy Developmental Test and...
![Page 1: Dr. Steven J. Hutchison Principal Deputy Developmental Test and …itea.org/.../hutchison_firstpresentation_shiftleft.pdf · 2012-12-01 · Shift Left Nov 2012 Page-1 DISTRIBUTION](https://reader034.fdocuments.us/reader034/viewer/2022043005/5f8a3f5632aaf37ff50d5ba3/html5/thumbnails/1.jpg)
Shift Left Nov 2012 Page-1
DISTRIBUTION STATEMENT A – Cleared for Open Publication by OSR on November 21, 2012 – SR case number 13-S-0395
Dr. Steven J. Hutchison Principal Deputy
Developmental Test and Evaluation November 2012
![Page 2: Dr. Steven J. Hutchison Principal Deputy Developmental Test and …itea.org/.../hutchison_firstpresentation_shiftleft.pdf · 2012-12-01 · Shift Left Nov 2012 Page-1 DISTRIBUTION](https://reader034.fdocuments.us/reader034/viewer/2022043005/5f8a3f5632aaf37ff50d5ba3/html5/thumbnails/2.jpg)
Shift Left Nov 2012 Page-2
DISTRIBUTION STATEMENT A – Cleared for Open Publication by OSR on November 21, 2012 – SR case number 13-S-0395
Persistent, rapidly composable, secure representation of the Joint Information Environment
Test & Evaluation
Operations
Performance Reliability
DT&E for Complex Systems
System Integration Labs
Training
Experimentation
Modeling & Simulation
Cyber Range
JIOR
JMETC
Interoperability Information Security
![Page 3: Dr. Steven J. Hutchison Principal Deputy Developmental Test and …itea.org/.../hutchison_firstpresentation_shiftleft.pdf · 2012-12-01 · Shift Left Nov 2012 Page-1 DISTRIBUTION](https://reader034.fdocuments.us/reader034/viewer/2022043005/5f8a3f5632aaf37ff50d5ba3/html5/thumbnails/3.jpg)
Shift Left Nov 2012 Page-3
DISTRIBUTION STATEMENT A – Cleared for Open Publication by OSR on November 21, 2012 – SR case number 13-S-0395
The DoD Acquisition Model
![Page 4: Dr. Steven J. Hutchison Principal Deputy Developmental Test and …itea.org/.../hutchison_firstpresentation_shiftleft.pdf · 2012-12-01 · Shift Left Nov 2012 Page-1 DISTRIBUTION](https://reader034.fdocuments.us/reader034/viewer/2022043005/5f8a3f5632aaf37ff50d5ba3/html5/thumbnails/4.jpg)
Shift Left Nov 2012 Page-4
DISTRIBUTION STATEMENT A – Cleared for Open Publication by OSR on November 21, 2012 – SR case number 13-S-0395
Test, Evaluation, Certification
Late to Need!
DIACAP Security T&E
![Page 5: Dr. Steven J. Hutchison Principal Deputy Developmental Test and …itea.org/.../hutchison_firstpresentation_shiftleft.pdf · 2012-12-01 · Shift Left Nov 2012 Page-1 DISTRIBUTION](https://reader034.fdocuments.us/reader034/viewer/2022043005/5f8a3f5632aaf37ff50d5ba3/html5/thumbnails/5.jpg)
Shift Left Nov 2012 Page-5
DISTRIBUTION STATEMENT A – Cleared for Open Publication by OSR on November 21, 2012 – SR case number 13-S-0395
Hindsight is 20-20
What did we know?
What did we test?
To reduce discovery late in the acquisition lifecycle, • test in mission context, • against realistic threat,
and….. Shift Left!
DOT&E COCOM/Service
Interop & IA Assessments
Fielded systems: • Interoperability issues • IA Vulnerabilities
Compliance with IA Controls and
Interoperability Standards and Profiles
are necessary but not sufficient
![Page 6: Dr. Steven J. Hutchison Principal Deputy Developmental Test and …itea.org/.../hutchison_firstpresentation_shiftleft.pdf · 2012-12-01 · Shift Left Nov 2012 Page-1 DISTRIBUTION](https://reader034.fdocuments.us/reader034/viewer/2022043005/5f8a3f5632aaf37ff50d5ba3/html5/thumbnails/6.jpg)
Shift Left Nov 2012 Page-6
DISTRIBUTION STATEMENT A – Cleared for Open Publication by OSR on November 21, 2012 – SR case number 13-S-0395
Net Ready KPP New Role for DASD(DT&E)
New Language • “DISA will ensure JITC leverages
previous, planned and executed DT&E and OT&E tests and results to support joint interoperability test certification and eliminate test duplication.”
• “DASD(DT&E) shall approve Developmental Test and Evaluation plans in support of Joint Interoperability Test Certification as documented in the TEMP. JITC shall advise DASD (DT&E) regarding the adequacy of test planning in support of Joint Interoperability Test Certification.”
DASD(DT&E) approves adequacy of Interoperability test planning
CJCSI 6212
![Page 7: Dr. Steven J. Hutchison Principal Deputy Developmental Test and …itea.org/.../hutchison_firstpresentation_shiftleft.pdf · 2012-12-01 · Shift Left Nov 2012 Page-1 DISTRIBUTION](https://reader034.fdocuments.us/reader034/viewer/2022043005/5f8a3f5632aaf37ff50d5ba3/html5/thumbnails/7.jpg)
Shift Left Nov 2012 Page-7
DISTRIBUTION STATEMENT A – Cleared for Open Publication by OSR on November 21, 2012 – SR case number 13-S-0395
Information Assurance Policy
Information Assurance compliance activities need to be integrated into DT&E and included in the TEMP
![Page 8: Dr. Steven J. Hutchison Principal Deputy Developmental Test and …itea.org/.../hutchison_firstpresentation_shiftleft.pdf · 2012-12-01 · Shift Left Nov 2012 Page-1 DISTRIBUTION](https://reader034.fdocuments.us/reader034/viewer/2022043005/5f8a3f5632aaf37ff50d5ba3/html5/thumbnails/8.jpg)
Shift Left Nov 2012 Page-8
DISTRIBUTION STATEMENT A – Cleared for Open Publication by OSR on November 21, 2012 – SR case number 13-S-0395
Information Assurance What’s Changing?
• Implements Risk Management Framework (RMF) instead of Mission Assurance Category/Confidentiality Level (MAC/CL)
• Adopts new guidance from the National Institute of Standards and Technology (NIST) and Committee on National Security Systems Instruction (CNSSI) documents on Cybersecurity
• Goes beyond IA and adopts the term: “Cybersecurity”
• Lexicon Changes – “Certification and Accreditation” becomes “Assessment and Authorization” – “Designated Approving Authority (DAA)” becomes “Authorizing Official (AO)” – “Certifying Authority” becomes “Security Control Assessor”
Threat = Any event with potential to cause harm to the network Vulnerability = Absence/weakness of safeguards to protect the network
Risk = Likelihood that a threat will realize or exploit a vulnerability
![Page 9: Dr. Steven J. Hutchison Principal Deputy Developmental Test and …itea.org/.../hutchison_firstpresentation_shiftleft.pdf · 2012-12-01 · Shift Left Nov 2012 Page-1 DISTRIBUTION](https://reader034.fdocuments.us/reader034/viewer/2022043005/5f8a3f5632aaf37ff50d5ba3/html5/thumbnails/9.jpg)
Shift Left Nov 2012 Page-9
DISTRIBUTION STATEMENT A – Cleared for Open Publication by OSR on November 21, 2012 – SR case number 13-S-0395
Implementing Cybersecurity What’s Being Proposed?
DASD(DT&E):
• Oversight of test planning in support of Cybersecurity C&A(A&A)
• Establish procedures to ensure that DT&E authorities for acquisition programs verify that adequate DT&E is planned and resourced to address Cybersecurity
• Confirm DT&E can be executed in a timely manner prior to approval of program Test and Evaluation Master Plans (TEMPs)
DASD(DT&E) will ensure adequate Cybersecurity test planning
![Page 10: Dr. Steven J. Hutchison Principal Deputy Developmental Test and …itea.org/.../hutchison_firstpresentation_shiftleft.pdf · 2012-12-01 · Shift Left Nov 2012 Page-1 DISTRIBUTION](https://reader034.fdocuments.us/reader034/viewer/2022043005/5f8a3f5632aaf37ff50d5ba3/html5/thumbnails/10.jpg)
Shift Left Nov 2012 Page-10
DISTRIBUTION STATEMENT A – Cleared for Open Publication by OSR on November 21, 2012 – SR case number 13-S-0395
DT&E in the Cyberspace Domain
An Integrated T&E Enterprise Capable of Creating a Realistic Cyberspace Test Environment at All Required
Security Levels
Cyberspace Threat Representations
Systems Under Test
Test Tools
Instrumentation BAF
JPRIMES
ACETEF
CDS
IO Range
SDREN
TSMO
Desired Federated Cyberspace T&E Capability
Process
Methodology Infrastructure
Workforce
Persistent, rapidly composable, secure representation of the Joint Information Environment
![Page 11: Dr. Steven J. Hutchison Principal Deputy Developmental Test and …itea.org/.../hutchison_firstpresentation_shiftleft.pdf · 2012-12-01 · Shift Left Nov 2012 Page-1 DISTRIBUTION](https://reader034.fdocuments.us/reader034/viewer/2022043005/5f8a3f5632aaf37ff50d5ba3/html5/thumbnails/11.jpg)
Shift Left Nov 2012 Page-11
DISTRIBUTION STATEMENT A – Cleared for Open Publication by OSR on November 21, 2012 – SR case number 13-S-0395
DT&E Cybersecurity Process Summary
Step 1 Cybersecurity Test
Requirements Evaluation
Focus on initiating an approach to Cybersecurity DT&E at Milestone A or B, with update at Milestone C.
Step 4 Cybersecurity Test in
Realistic Cyber Environment
Focus is on Cybersecurity readiness in an operational mission environment to understand capabilities and limitations of the SUT and interconnections against a cyber threat using Red Team testing.
Step 3 Cyber Kill Chain
Evaluation
Focus is assessment of Cybersecurity of the system under test, in a realistic mission and cyber environment, using exploitation testing techniques, post-CDR.
Step 2 Cybersecurity
System Integration Evaluation
Focus is assessment of Cybersecurity in component and system integration vulnerability testing, between MS B and C.
![Page 12: Dr. Steven J. Hutchison Principal Deputy Developmental Test and …itea.org/.../hutchison_firstpresentation_shiftleft.pdf · 2012-12-01 · Shift Left Nov 2012 Page-1 DISTRIBUTION](https://reader034.fdocuments.us/reader034/viewer/2022043005/5f8a3f5632aaf37ff50d5ba3/html5/thumbnails/12.jpg)
Shift Left Nov 2012 Page-12
DISTRIBUTION STATEMENT A – Cleared for Open Publication by OSR on November 21, 2012 – SR case number 13-S-0395
Cybersecurity Testing in the Acquisition Lifecycle
AS TDS
JCIDS Process
Full Rate Production
Decision Review
CBA Joint
Concepts (COCOMs)
MS C MS B
Strategic Guidance (OSD/JCS)
MS A
ICD Technology Development
CDD Engineering & Manufacturing Development
Production and Deployment O&S MDD
Materiel Solution Analysis
AoA
CPD
TEMP
SEP
SRR SFR PDR CDR TRR SVR
*TEMP
*SEP
ASR
PPP
TRA *PPP STAR OTR
TEMP
IOT&E
* STAR * STAR
* SRD
AOTR
Cyber Test Step 1
Cyber Test Step 1 Step 2
Cyber Test Step 1 Step 2 Step 3
Cyber Test Step 1 Step 2 Step 3 Step 4
Reduce the Cyber Attack Surface
![Page 13: Dr. Steven J. Hutchison Principal Deputy Developmental Test and …itea.org/.../hutchison_firstpresentation_shiftleft.pdf · 2012-12-01 · Shift Left Nov 2012 Page-1 DISTRIBUTION](https://reader034.fdocuments.us/reader034/viewer/2022043005/5f8a3f5632aaf37ff50d5ba3/html5/thumbnails/13.jpg)
Shift Left Nov 2012 Page-13
DISTRIBUTION STATEMENT A – Cleared for Open Publication by OSR on November 21, 2012 – SR case number 13-S-0395
Conclusion
• DT&E in mission context
• Improve Interoperability
• Improve Cybersecurity
• Reduce discovery in IOT&E
• Improve Acquisition Outcomes
To ensure rapid fielding of enhanced capabilities to the Warfighter …
Shift Left!
![Page 14: Dr. Steven J. Hutchison Principal Deputy Developmental Test and …itea.org/.../hutchison_firstpresentation_shiftleft.pdf · 2012-12-01 · Shift Left Nov 2012 Page-1 DISTRIBUTION](https://reader034.fdocuments.us/reader034/viewer/2022043005/5f8a3f5632aaf37ff50d5ba3/html5/thumbnails/14.jpg)
Shift Left Nov 2012 Page-14
DISTRIBUTION STATEMENT A – Cleared for Open Publication by OSR on November 21, 2012 – SR case number 13-S-0395
Questions?
![Page 15: Dr. Steven J. Hutchison Principal Deputy Developmental Test and …itea.org/.../hutchison_firstpresentation_shiftleft.pdf · 2012-12-01 · Shift Left Nov 2012 Page-1 DISTRIBUTION](https://reader034.fdocuments.us/reader034/viewer/2022043005/5f8a3f5632aaf37ff50d5ba3/html5/thumbnails/15.jpg)
Shift Left Nov 2012 Page-15
DISTRIBUTION STATEMENT A – Cleared for Open Publication by OSR on November 21, 2012 – SR case number 13-S-0395
T&E Plan – Test – Report cycle can exceed six months!
•Multiple Test Orgs – DT, OT, Iop, IA
•Multiple Decision Makers – MDA, CIO, DAA
Pilot Record OTRR
60 days
OTRR Full Deployment Decision Review
60 days
Eval Report
DIACAP
Interop Testing
OT&E
Operational Test Plan
Test Concept Brief
60 days
Test Plan Approved
User Training Support Implemented
Interop Cert
IAC&A
Tester Training DT&E
14 days
DoD Test, Evaluation, & Certification
AOTR