UNCLASSIFIED

505th Test and Evaluation Group

Challenges in Integrating Testing

into Training and Exercises

Susana V. McKee

505 TEG Test TA

UNCLASSIFIED

UNCLASSIFIED

Delivering C2 Combat Capability

It always seems sooooo obvious….

“No question is so difficult to answer as that to which

the answer is obvious.” -- George Bernard Shaw

If it seems obvious…it ain’t! -- Susi

“All things are difficult before they are easy.”

-- Thomas Fuller

Point # 1

UNCLASSIFIED

UNCLASSIFIED

Delivering C2 Combat Capability

Common Training/Experimentation activities:

- Data collection – what really happened?

- AAR/analysis products

- Real time scenario monitoring

Capabilities & tools before LVC DMO for Test:

- Stand and watch the tactical systems

- Take notes on paper

- Manually cut and paste data from

C2 system logs

Capabilities and tools after LVC DMO for Test:

- Persistent automated, archived, objective

data capture

- Real time scenario monitoring

- Real time comparison of range TSPI to

tactical tracks

- Automated AAR products & analysis reports

Test

Training

Experimentation

Tactics

LVC

DMO

SIMPLE , RIGHT?

Event Support Overlap

UNCLASSIFIED

UNCLASSIFIED

Delivering C2 Combat Capability

CNCTT Big Picture

Persistent capability to capture, reconstruct, analyze and evaluate end-

to-end data during tests, experiments & training events to support

Joint warfighters

Example events:

• Operational Tests (including Tactics Development & Evaluations)

• Flag Exercises (Red, Green, Blue, Virtual)

• Weapon School Mission Employment

• Joint Expeditionary Force Experiment

• AGILE Fire – JT&Es

Key system elements:

• Cost/time-effective way to collect and analyze decision-quality data

• Integrates Joint, Live-Virtual-Constructive (LVC) into events

• Facilitates conducting and collecting data from distributed events

• Testing conducted in conjunction with training exercises

• Multi-medium capture & playback (end-to-end tactical & operational data)

• Event reconstruction to analyze the process and data exchanges

• Mission performance- Did we accomplish our stated objectives

• Ready access to USAF and USA tactical & operational systems/feeds

UNCLASSIFIED

UNCLASSIFIED

Delivering C2 Combat Capability

WOC &

98 RW

RF 11-3

Non-kinetic

Effects

Tanker

Contested Ops

Patriot

CAS/DT

Aircraft

JSTARS & AWACS

Airborne C2

CAOC-N Combat Ops

Division (& LNOs)

624 Cyber OC

Connectivity & Exercise

Response Cell

614 JSpOC

Exercise Response Cell

Contested Ops

(CNA/CND

OCS/DCS)

Contested Ops

(OCS/DCS)

RC-135

Space & IO

Ranges

HC-130P

DGS

US/UK/AU

SOF

UNCLASSIFIED

Delivering C2 Combat Capability

TEG/CAOC-N Core Competencies

Focus

Advanced Combat ID---capes/lims

CDO Advanced Training and TTP Development--capes/lims

LVC/DMO Exercise Planning/Execution (O-Plan flyout)--capes/lims

Non-kinetic Effects Integration--capes/lims

5th Gen/Advanced Program Integration--capes/lims

C2 & ISR Test/Eval & Integration--capes/lims

Air-Sea Battle (Joint C2, IAMD)--capes/lims

Radar System Optimization-capes/lims

C2-ISR Distributed Ops-capes/lims

UNCLASSIFIED

UNCLASSIFIED

Delivering C2 Combat Capability

Point # 2

Establish your Roadmap—and start

driving

Stick with the concept

“If you don't know where you are going, any road will

get you there.” -- Lewis Carroll

UNCLASSIFIED

UNCLASSIFIED

Delivering C2 Combat Capability

AOC Weapons System Inc 10.1

Distributed Ops (EPO 12-001C) 1) Assess capability to perform AOC processes in

a distributed and reach-back operational scenario

using remotely hosted apps, services, and data.

2) Assess operational suitability impacts to

distributed and reach-back ops using remotely

hosted applications, services, and data.

3) Assess impacts to current AOC WS TTP based

on executing scoped AOC processes in a

distributed and reach-back environment enabled

by virtualization technologies.

.

Conducted August 2012 between CAOC-N &

CAOC-X

Conducted in concert with the Joint

Expeditionary Force Experiment (JEFX) limited

objective experiment

JEFX focused on distributed virtual training

MUA assessed RE 11 virtualization capability to

support distributed and reach-back operations

AOC Distributed Ops/Control in exercises

Multiple AOCs concurrently working

processes such as Dynamic Targeting,

Target Nomination, & Global Strike Ops

Utilizes real-time network connection (SIPR,

JIOR)

UNCLASSIFD

UNCLASSIFIED

Delivering C2 Combat Capability

Develop best practices for AOC mission

in a cyber-degraded / denied

environment

AFTTP 3-1.AOC & tactics bulletins

Cyber effects integrated into major

USAFWC exercises

Integrates red cyber aggressors and

blue cyber defense in-depth

CDO environment on AOC floor

Evaluate the capability of the Combat Ops

Division to perform its AOC mission in a

cyber-degraded/denied environment

Evaluate AOC and CRC collaboration

capability in cyber degraded/denied

environment

Develop TTPs for continued AOC operations

in a cyber degraded/denied environment

Phase I Execution 18-22 NOV 2013

Phased cyber approach, affecting various

communication methods, ranging from

deception to degradation to denial

Results could impact JP 3-30 (C2 for Joint

Air Operations), AFTTP 3-1.GP, &

AFTTP 3-3.AOC

1st time CRC (729 ACS) experienced

operations in a cyber-degraded/denied

environment

AOC Operations in a Cyber-Denied

Environment: Phase I (EPO 13-122C)

UNCLASSIFIED

UNCLASSIFIED

Delivering C2 Combat Capability

Distributed Control Demonstration

CAOC-N / RF 14-1

1) Assess the capability to distribute control

to maintain continuity of operations when

communication degradation or denial

inhibits or prevents centralized control of

forces

2) Assess impacts to current AOC WS TTP

based on executing scoped AOC processes

in a distributed environment enabled by

collaborative technologies

505 TS conducted demo in constructive

and live training scenario at CAOC-N

during Red Flag 14-1 training activities

from 27 Jan to 14 Feb 2014

Collection efforts were scoped to

capture operator interactions with

system and supporting C2 functions

and existing TTP impacts

Conduct demos in operationally relevant

degraded/denied env for in-depth TTP dev

and eval (Longer vuls; Planning)

Need for detailed CAOC checklist to ID

procedures for “graceful reintegration” for

CAOC to resume op C2 once distributed

control activities conclude

UNCLASSIFIED

UNCLASSIFIED

Delivering C2 Combat Capability

DMOC Test Support Capability

Employment concepts within existing training events

AOC/C2 process testing within existing training events

DMO (virtual & constructive) stds certification testing

Legacy sim and constructive environment

interoperability testing

Controlled interface dev, performance, ruleset testing

Provide UAV (virtual) video feeds and multi type

datalink messages across live – virtual

Provide constructive environment feeds to add

density/complexity to test environment

UNCLASSIFIED

UNCLASSIFIED

Delivering C2 Combat Capability

Point #3

Ask for Directions—YES, even you men!

Seek out mentors, other Services, Industry…

“My ancestors wandered lost in the wilderness for

forty years because even in Biblical times, men would

not stop to ask for directions.” -- Elayne Boosler

UNCLASSIFIED

UNCLASSIFIED

Delivering C2 Combat Capability

UNCLASSIFIED

UNCLASSIFIED

Delivering C2 Combat Capability

UNCLASSIFIED

UNCLASSIFIED

Delivering C2 Combat Capability

UNCLASSIFIED

UNCLASSIFIED

Delivering C2 Combat Capability

UNCLASSIFIED

UNCLASSIFIED

Delivering C2 Combat Capability

Point #4

Detours and Roadblocks are expected

(it’s rarely a straight path)

“If you are going through hell, keep going.”

-- Winston Churchill

“Kilometers are shorter than miles. Save gas, take your

next trip in kilometers.” -- George Carlin

“On a New York subway you get fined for spitting, but

you can throw up for nothing.” – Lewis Grizzard

UNCLASSIFIED

UNCLASSIFIED

Delivering C2 Combat Capability

How to Establish a Repeatable

Approach NEEDED REALITY

PEOPLE On-site certified testers (6) Distributed certified testers/only 3 on-site

Team to operate test tools (8) 2 SMEs

Team to integrate/plan tests

concurrently with training plans

Tests added-on after-the-fact

PROCESSES Test Discipline Training Discipline/TTP

Robust data collection/analysis Notepads/who’s not doing anything?

Change/Self-assessment Institutionalized Resistance to T&T

TOOLS Instrumentation Some Instrumentation

Representative Ops Sys Outdated Sys

High Fidelity Test Models Training models

$$$ $5.3M Setup/Baseline test funding

(people/sys)

Set up $3M; sustainment $0

UNCLASSIFIED

UNCLASSIFIED

Delivering C2 Combat Capability

Law of Diffusion of Innovation

* Graphic from Wikipedia

UNCLASSIFIED

UNCLASSIFIED

Delivering C2 Combat Capability

KNOWLEDGE PERSUASION DECISION IMPLEMENTATION CONFIRMATION

REJECT

ACCEPT

Five Stages in the Decision

Innovation Process

Slide by Colonel M.V. Coyote Smith, PhD

UNCLASSIFIED

UNCLASSIFIED

Delivering C2 Combat Capability

Technology Cometh…

1700

1800

1900

2000

2000-2025 {

1900-2000 {

1800-1900 {

>1700 {

TRUTH

Tech change is inevitable and accelerating

Am

ou

nt

of

new

tech

no

log

y i

ntr

od

uced

Infusion of Technology Slide by Colonel M.V. Coyote Smith, PhD

UNCLASSIFIED

UNCLASSIFIED

Delivering C2 Combat Capability

11 World Changing Technologies

Airpower -- RPVs

Spacepower – high fidelity cluster satellites

Cyber – defense/attack; most ISR in future

3D Printing –since 1984; originally medical/dental app

Robotics-Soldiers/Marines robotized; autonomous bots

Currency – Bitcoin in 2008 as Internet crypto-currency

Nanotechnology–5-10x (50-100x) more powerful explosives

Biotechnology–By 2025 many genetically engineered cures

Nuclear Weapons – 9 currently have (Iran/Myanmar next?)

Directed Energy – lasers as wpns; EM Pulse against grid?

Cognition Harvesting-brain-to-prosthetic interface; psych

warfare; manufactured memory?

Slide info by Colonel M.V. Coyote Smith, PhD

UNCLASSIFIED

UNCLASSIFIED

Delivering C2 Combat Capability

3D Printing

From dental replacements/prosthetics – 1984

To potential terror applications (i.e., bomb casings, firearms)–present

Commercial/Military sectors developing

Europeans concentrating on “humane” meat, replicate food, stem cells,

virology, replacement organs, quality of life items

Able to print a house in a day (build a base remotely?)

Print out replacement organs (i.e., liver)

AF logistics – print resources in forward areas

“Just in time” vs “As needed”; acq / sustainment impacts

Dramatic shift in manufacturing facilities and trade skills

Slide info by Colonel M.V. Coyote Smith, PhD

UNCLASSIFIED

UNCLASSIFIED

Delivering C2 Combat Capability

It’s Inevitable…

Technology is changing rapidly

Private sector applications

Military/Federal applications

Must be thoroughly evaluated

Rapid fielding cycles require faster/cheaper testing

LVC DMO offers solutions

UNCLASSIFIED

UNCLASSIFIED

Delivering C2 Combat Capability

What’s Next?

Institutionalize threads in all exercises

Air/Space/Cyber Integration (i.e., NKO; GPS jamming, etc)

Add Space/Cyber threads in every VF

Continue to expand at CAOC-N

Add instrumentation, as appropriate

Work within new guidelines for IATT/ATOs

Headway made in Phase I at DMOC—intended to make DMOC

more responsive to CNAF requests

Implement needed Infrastructure upgrades to support

next Phase

Focus on RFs/WSMEs as near-term targets of

opportunity

UNCLASSIFIED

UNCLASSIFIED

Delivering C2 Combat Capability

What’s Next?

Determine suitability of training models for test

Determine C&A/VV&A feasibility for non-accredited sys/models

Develop separate test & training virtual battlespaces

Investigate feasibility of bringing in appropriate threat models

from other distributed sites (such as SIMAF/NASIC)

Develop guards/firewalls to required real world connects

Develop gateways to support TENA to M&S data exchange

Explore feasibility of updating systems/virtual sims to support

test version of aircraft/system software

Sustain JMETC nodes to enable test support opportunities

UNCLASSIFIED

UNCLASSIFIED

Delivering C2 Combat Capability

Point #5

Take the road less travelled…

Innovate

Originality may breed interest/recognition

Create the trend (be a leader/innovator)

“Failure” is temporary…

Pay attention to the details—sweat the small stuff or fail!

“Any man who can drive safely while kissing a pretty girl is simply not giving

the kiss the attention it deserves.” -- Albert Einstein

"Sometimes the road less traveled is less traveled for a reason" -- Jerry Seinfeld

UNCLASSIFIED

UNCLASSIFIED

Delivering C2 Combat Capability

Point #6

It’s not just WHAT you do but HOW You Get There

Others are watching you

Integrity in all you do

Build strong relationships….trust builds commitment

“Delight” your customers

Love what you’re doing – enthusiasm is contagious!

“Oh, you hate your job? Why didn't you say so?

There's a support group for that. It's called EVERYBODY,

and they meet at the bar.” -- Drew Carey

UNCLASSIFIED

UNCLASSIFIED

Delivering C2 Combat Capability

Point #7

Call to Action!

Still need to prove that LVC DMO routinely has a cost/

benefit advantage

Call to action for DoD Partners—not just a DoD challenge

Different: Approach? Location? Venues? Models?

“The path to success is to take massive, determined action.”

-- Anthony Robbins

“I never worry about action, but only about inaction”

-- Winston Churchill

UNCLASSIFIED

UNCLASSIFIED

Delivering C2 Combat Capability

Questions?

UNCLASSIFIED

UNCLASSIFIED