GUIDEx is that robust experimentation methods from … and Analysis Group (JSA) to establish Action...

The thesis of GUIDEx is that robust experimentation

methods from the sciences can be adapted and applied

to military experimentation and will provide the basis

for advancements in military effectiveness

in the transformation process.

Electronic versions of GUIDEx and its pocketbook, Slim-Ex, can be downloaded from the following site:

http://www.dtic.mil/ttcp/

Guide for Understanding and Implementing

Defense Experimentation

GUIDEx

Pocketbook Version of GUIDEx (Slim-Ex)

The Technical Cooperation Program

Guide for Understanding and Implementing

Defense Experimentation

GUIDEx

This document contains information authorized under the auspices of The Technical Cooperation Program (TTCP) for

unlimited release and distribution.

GUIDEx does not present the official policy of any participating nationorganization. It consolidates principles and guidelines for improving the

impact of science-based experimentation on defense capability development.

All organizations are invited to use the guidance it provides.

Pocketbook Version of GUIDEx (Slim-Ex)

Australia Canada United Kingdom United States

SUBCOMMITTEE ON NON-ATOMIC MILITARY RESEARCH ANDDEVELOPMENT (NAMRAD)

THE TECHNICAL COOPERATION PROGRAM (TTCP)

JOINT SYSTEMS ANALYSIS (JSA) GROUP

METHODS AND APPROACHES FOR WARFIGHTING EXPERIMENTATIONACTION GROUP 12 (AG-12) (TTCP JSA AG-12)

Electronic copy compiled in February 2006,Ottawa and available at http://www.dtic.mil/ttcp/

© TTCP

ISBN 92-95046-12-9

Guide for Understanding and Implementing DefenseExperimentation (GUIDEx)

March 2006, Version 1.1

KEYWORDS:experiment, trial, test, hypothesis, analysis, causal,cause-and-effect, defense, warfighting, simulation, wargame,training, exercise, ethics, valid, requirement, capability,development, measure, instrument, unit, joint, force, campaign.

CONTACT:[email protected]

Art Direction by ADM(PA) DMCS CS05-0513-B

T TC P G U I D E x Po c k e t b o o k

CANADIAN FORCESEXPERIMENTATION CENTREOTTAWA, CANADA

Funds for printing this document were provided by the

vT TC P G U I D E x Po c k e t b o o k

iv

Defense Experimentation (GUIDEx). GUIDEx describes 14 Principles leading to valid (good) experimentation thatare amplified through 8 Case Studies drawn from theactivities of the participating nations and coalitions. Part Iof GUIDEx is reproduced here with appropriate additionsas a standalone pocketbook on defense experimentation(known as Slim-Ex). Parts II and III, the main body ofGUIDEx, is for those people who design, execute, analyzeand report on such experiments. These experimenters arethe backbone of the community and should benefit fromthe full detail of the 14 Principles and 8 Case Studies.

GUIDEx is intended to be a guide for clients, people whoask the questions that lead to experiments and campaignsand for whom reports are prepared. It is also for those whodecide how the question will be addressed and approvethe methods that will be applied. It is hoped that thispocketbook will act as an introduction to the full GUIDExand so help stimulate better communication amongmilitary officers, government officials and the defensescientific community of the allied nations on all mattersassociated with defense experimentation.

Paul Labbé Chair, TTCP JSA AG-12

Foreword

The development of allied forces has always been adifficult and complex process. However the need for rapidforce development to respond to asymmetric andunpredictable threats, the demands of coalitionoperations, the perceived need for information supremacy,combined with the availability of new transformationaltechnologies and concepts, have caused this task tobecome even more challenging over the past few years.Experimentation offers a unique means to support thedevelopment and transformation of allied forces byadvancing our knowledge of complex networked systemsand capabilities likely to be fielded in the near future.

“Anything we use today arrives through a process of organizedexperimentation; over time, improved tools, new processes,and alternative technologies all have arisen because they havebeen worked out in various structured ways.” (Thomke 2003: p. 1)

The importance of experimentation motivated TTCP’s JointSystems and Analysis Group (JSA) to establish ActionGroup 12 on Methods and Approaches for WarfightingExperimentation in 2002. The work of AG-12 culminated in a 350-page guide for defense experimentation - theTTCP Guide for Understanding and Implementing

Considerations for Successful Experimentation . . . . . . . . . . . . . .36

Human Variability . . . . . . . . . . . . . . . . . . . . .37Exploiting Operational Test and Evaluation and Collective Training Events . . . . . . . . . . . . .40Modeling and Simulation Considerations . . . . . .48Experiment Control . . . . . . . . . . . . . . . . . . . .51Data Analysis and Collection . . . . . . . . . . . . . .55Ethics, Security and National Issues . . . . . . . . .57Communication with Stakeholders . . . . . . . . . .59

GUIDEx Experiment and Campaign Planning Flowchart . . . . . . . . . . . .64

21 Threats to a Good Experiment . . . . . . . . . .69

GUIDEx Case Studies . . . . . . . . . . . . . . . . . . . .71

Epilogue . . . . . . . . . . . . . . . . . . . . . . . . . . . . .75

Acronyms, Initialisms and Abbreviations . . . . . . . . . . . . . . . . . . . . . .76

References . . . . . . . . . . . . . . . . . . . . . . . . . . .80

Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .82

Acknowledgements . . . . . . . . . . . . . . . . . . . . .86

vii

Table of Contents

Foreword . . . . . . . . . . . . . . . . . . . . . . . . . . . . .iv

Table of Contents . . . . . . . . . . . . . . . . . . . . . . .vi

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . .1Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2Experiments and Science . . . . . . . . . . . . . . . . .4

Designing Valid Experiments . . . . . . . . . . . . . .6Experiment Hypotheses . . . . . . . . . . . . . . . . . .7Components of an Experiment . . . . . . . . . . . . . .8What Is a Good Experiment? . . . . . . . . . . . . . .10Experiments during Capability Development and Prototyping . . . . . . . . . . . . . . . . . . . . . . .15

Integrated Analysis and Experimentation Campaigns . . . . . . . . . . . . .19

Why use a Campaign . . . . . . . . . . . . . . . . . . .21Iterating Methods and Experiments . . . . . . . . . .23Integration of Scientific Methods . . . . . . . . . . .26Different Methods Offer Different Strengths . . . .27Different Methods during Capability Development and Prototyping . . . . . .31Employing Multiple Methods to Increase Rigor . . . . . . . . . . . . . . . . . . . . . .33

T TC P G U I D E x Po c k e t b o o kvi

Introduction

Increasingly, nations such as the United States, GreatBritain, Canada, Australia, New Zealand and indeed NATOitself are relying on experimentation to assist in thedevelopment of their future military forces. For example,the United States Department of Defense stresses theimportance of experimentation as the process that willdetermine how best to optimize the effectiveness of itsjoint force to achieve its vision of the future (US Joint Staff2000). Is this confidence, in the ability of experimentationto support the military transformation process,appropriate? Certainly, experimentation has proven itselfin the sciences and technology by producing dramaticadvances. Can the methods of experimentation that haveso expeditiously and radically developed science andtechnology be applied to the military transformationprocess to achieve similar advances in militaryeffectiveness?

The thesis of this guide is that robust experimentationmethods from the sciences can be adapted and applied tomilitary experimentation and will provide the basis for

1T TC P G U I D E x Po c k e t b o o k

viii

experimentation”, “defense experimentation” and“military experimentation”. GUIDEx has settled on a singleterm, Defense Experimentation in order to present itsideas in a consistent manner. Defense Experimentation isdefined here as “the application of the experimentalmethod1 to the solution of complex defense capabilitydevelopment problems, potentially across the fullspectrum of conflict types2, such as warfighting, peace-enforcement, humanitarian relief and peace-keeping”.GUIDEx also presents the idea of Integrated Analysis andExperimentation Campaigns, in which experiments arecombined with other analytical techniques; both to tacklelarger problems that would not be possible with singleexperiments, and to exploit the strengths of differenttechniques.

3

advancements in military effectiveness in thetransformation process. The authors have structured therelevant experimentation material under 14 Principles,which ensure that defense experimentation programspositively impact coalition organizations’ ability to evolveforce capabilities of the future. Also, they have provided anexperimentation-planning flowchart that in one pageshows what needs to be done, together with a set of CaseStudies that demonstrate the value of the principles inpractice.

GUIDEx is not meant to duplicate information alreadyavailable in other documents and textbooks onexperimentation such as those referenced here, [ABCA2004; Alberts and Hayes 2002, 2005; Dagnelie 2003;Radder 2003; Shadish et al. 2002] or on command andcontrol (C2) assessment [NATO 2002], but organizes andexpands this detailed information under 14 Principles toguide successful defense experimentation.

Scope

GUIDEx is about the use of the experimental method in thedefense domain. A number of terms are used by the TTCPnations to describe such activities, including “warfighting

T TC P G U I D E x Po c k e t b o o k2

1 The major focus of GUIDEx is experiments based upon field events andhuman-in-the-loop virtual simulations, but the principles of GUIDEx arealso applicable to experiments based on analytic wargames andconstructive simulations.

2 Most of the examples available to this guide have been based onwarfighting scenarios, simply because of the legacy of the primary focusof defense experimentation to date.

“If I do this, what will happen?” The key tounderstanding experimentation, and the characteristic thatseparates experimentation from all other researchmethods, is manipulating something to see what happens.The scientific aspect of experimentation is themanipulation of objects under controlled conditions whiletaking precise measurements. In its simplest form[Shadish et al. 2002: p. 507], an experiment can bedefined as a process “to explore the effects ofmanipulating a variable.”

5

Experiments and Science

In about 400 B.C., philosophers Socrates and Platoinvestigated the meaning of knowledge and methods toobtain it using a rational-deductive process, or pure logic(logic), without reference to the real world. Aristotle was atransitional figure who advocated observation andclassification, bridging to later scientists like Ptolemy andCopernicus who developed empirical-inductive methodsthat focused on precise observations and explanation ofthe stars. These early scientists were not experimenters. Itis only when later scientists began to investigate earthlyobjects rather than the heavens, that they uncovered anew paradigm for increasing knowledge.

In the early 1600s, Francis Bacon introduced the termexperiment and Galileo moved from astronomicalobservations to conducting earthly experiments by rollingballs down an inclined plane to describe bodies in motion.The realization that manipulating objects would yieldknowledge spawned a new research paradigm, oneunimagined in the previous 2000 years of exploring theout-of-reach heavens. The basis of this new scienceparadigm called experimentation (the empirical-deductiveapproach) was a simple question [Feynman 1999]:


Experiment Hypotheses

To understand cause-and-effect relationships betweencapabilities and increased warfighting effectiveness is tounderstand experiment hypotheses. Any national orcoalition capability problem may be stated as: Does Acause B? An experimental capability or concept–a newway of doing business– is examined in experimentationto determine if the proposed capability A causes theanticipated military effect B. The experiment hypothesisstates the causal relationship between the proposedsolution and the problem.

Hypothesis

If… “proposed change”Then… “improved warfighting capability”

It is an “If...then...” statement, with the proposedcause–innovative concept– identified by the if clause,and the possible outcome– the problem resolution–identified by the then clause.

7

Designing Valid Experiments

Principle 1. Defense experiments are uniquely suited toinvestigate the cause-and-effect relationshipsunderlying capability development.

Principle 2. Designing effective experiments requires anunderstanding of the logic of experimentation.

Principle 3. Defense experiments should be designed to meetthe four validity requirements.

Improved capabilities cause improved future warfightingeffectiveness. Experimentation is the unique scientificmethod used to establish the cause-and-effectrelationship of hypothesized capabilities. If experimentersdesign the five experiment components to meet the fourexperiment validity requirements, defined later, thedefense experiment will provide the scientific evidence toproceed. Defense experiments are essential to developempirical- and concept-based capabilities that yieldimplementable prototypes. The use of a“develop–experiment–refine” approach ensures that arigorous methodology relates new capabilities towarfighting effectiveness. The development and deliveryof defense concepts and capabilities is thus supportedthrough experimentation.


These five components are useful in understanding alldefense experiments including large field experiments.Some field experiments are grand exercises with multipleexperimental initiatives (possible causes), sometimes asmany as 20 to 30 different initiatives in one experiment. Tobe useful, each individual experimental initiative should beconfigurable as a unique mini-experiment with its own

9

Components of an Experiment

All experiments– large or small, field or laboratory,military or academic, applied or pure–consist of fivecomponents3 [Shadish et al. 2002: p. 2]:

1. The treatment, the possible cause A, is a capability orcondition that may influence warfighting effectiveness.

2. The effect B of the treatment is the result of the trial, anincrease or decrease in some measure of warfightingeffectiveness.

3. The experimental unit 4 executes the possible cause andproduces an effect.

4. The trial is one observation of the experimental unit undertreatment A or under the alternative ~A to see if effect Boccurred, and includes all of the contextual conditions ofthe experiment.

5. The analysis phase of the experiment compares theresults of one trial to those of another.


3 For application of these concepts to test and evaluation, see [Kass 1997].

4 An experimental unit includes all operators with their gear, procedures,and concept of operations. In experimentation, the apparatus includesthe experimental unit and necessary conditions for effecting changesand observing effects.

Five Components of any Experiment

TRIAL 4

TREATMENT APossible Cause AIndependent VariableExamples

– new sensor– new C2 process– new JTF organization

1 EFFECT BPossible Effect BDependent VariableMeasure of Performance (MoP)Examples

– targets detected or not– time from sensor to shooter– percent objectives met

2

EXPERIMENTAL UNITSmallest Unit Assigned to TreatmentExamples

– sensor operator– sensor management cell– Joint Task Force

3ANALYSIS

Document CHANGE in BExamples

– Outcome B compared to:• different treatments• different conditions

5

effectiveness is the number of targets detected. Theexperiment hypothesis could be: “If new sensors areemployed, then target detections will increase.”

1 Ability to use the new capability A

Developing and generating the new experimentalcapability for the experiment is often a major resourcecommitment. In an ideal experiment, operators employ theexperimental capability, in this case the new sensors, to itsoptimal potential; thereby allowing the new capability tosucceed or not succeed on its own merits. Unfortunately,this ideal is rarely achieved. A lesson repeatedly learnedfrom defense experiments is that new experimentalcapabilities are frequently not fully realized in theexperiment.

A number of things can go wrong with an experimentalsurrogate. For example, the hardware or software does notwork as advertised or anticipated. The experiment playersmay be undertrained and not fully familiar with itsfunctionality. Because the experimental treatmentrepresents a new capability, the trial scenario andpotential outcomes may not be sensitive to the newcapability’s enhanced performance.

11

subset of the five components. Each initiative is aparticular treatment with its own experimental unit(operators in one area of the task force), its own set ofoutcome measures, and its own set of trial conditions.However, in practice it is very difficult to maintainindependence among these many experiments within thelarge exercise, which makes it difficult to isolate specificcausal influences.

What Is a Good Experiment?

A good, or valid, experiment provides information toascertain whether A caused B [Shadish et al. 2002: p. 3].Four logically sequenced requirements are necessaryto achieve a valid experiment.5 A simple experimentexample will illustrate these four requirements. Aproposed concept postulates that new sensor capabilitiesare required to detect future targets. An experiment toexamine this proposition might employ current sensors onthe first day of a two-day experiment and a new sensorcapability on the second day. The primary measure of


5 Many detailed good practices developed by experiment agenciesthrough experience (and described in recent books such as [Alberts andHayes 2002, 2005]) can be organized under these four requirementsand the 14 Principles.

stimuli presentations, and a controlled externalenvironment, mitigates experiment-induced error. Inaddition, since the computation of variability in statisticsdecreases as the number of repetitions increases, a largersample size increases the signal-to-noise ratio making iteasier to detect change.

Analysts measure change in effectiveness by comparingthe results of one experiment trial to those of another.Typically, different experiment trials represent differentlevels of applications of the same capability, alternativecompeting capabilities, or the same capability underdifferent conditions. A change in military effectivenessmay also be detected by comparing the results of anexperiment trial to a pre-existing baseline, a taskstandard, or a desired process.

3 Ability to isolate the reason

for change in the effect B

If an experimenter employed a useable capability thatproduced a noticeable increase in the number of targetdetections, was the observed change in detections due tothe intended cause–changing from old sensors to new–or due to something else? In the sensor-experimentexample, an alternative explanation for the increase in

13

A valid experiment design ensures that the new capabilityworks under relevant conditions prior to execution, thatthe operators are adequately trained to employ itappropriately, and that the scenario is sufficiently sensitiveto determine the capability’s effectiveness. Experimenterscontinually monitor these aspects during experimentexecution. If the experimental sensors A do not functionduring the experiment, the new capability will most likelynot affect the military unit’s ability to detect targets B,which is the next experiment validity requirement.

2 Ability to detect a change in the effect B

When the player unit correctly employs a new capability,does it result in any noticeable difference in the effect Bduring the experiment trial? Ideally, a change in thenumber of detections accompanies a transition from old tonew sensors. If this is not the case, this may be becausethere is too much experimental noise6– the ability todetect change is a signal-to-noise ratio problem. Too muchexperimental error produces too much variability,hampering detection of a change. Reduction of experimentvariation, through data collection calibration, limited


6 Experimental noise interferes with the observation of the desiredvariable at a required degree of precision.

Experiment design issues that support operational realismrevolve around the representation of surrogate systems,the use of operational forces as the experimental unit, andthe use of operational scenarios with a realistic reactivethreat. To ensure the operational robustness, theexperiment should examine multiple levels of threatcapabilities under various operational conditions.

Experiments during CapabilityDevelopment and Prototyping

Nations employ a variety of processes to supportdevelopment of improved empirical- and concept-basedcapabilities and are, increasingly, employing defenseexperimentation to support the delivery of this improvedwarfighting effectiveness. These capability developmentand prototyping processes are not the same across thedifferent nations (in some nations these processes arereferred to as concept development andexperimentation, CD&E). However, in most cases theyfollow similar develop–experiment–refine stages. Forthe purposes of GUIDEx, therefore, a generic description ofthese stages is presented with the hope that the idealsembodied can be mapped onto each nation’s own way ofdoing business.

15

detections on the second day could be that of a learningeffect. That is, the sensor operators may have been moreadept at finding targets because of their experience withtarget presentations on Day One and, consequently, wouldhave increased target detections on Day Two, whether ornot different sensors were employed. An increase inoperator experience coincidental with a change in sensorswould dramatically alter the interpretation of the detectedchange in effectiveness. An experiment outcome withalternative explanations is a confounded result. Scientistshave developed experimentation techniques to eliminatealternative explanations of the cause of change:counterbalancing the presentation of stimuli to theexperimental unit, the use of placebos, the use of a controlgroup, random assignment of participants to treatmentgroups, and elimination or control of external influences.

4 Ability to relate the results

to actual operations

If the player unit ably employed the capability, and if anexperimenter detected change and correctly isolated itscause, are the experiment results applicable to theoperational forces in actual military operations? The abilityto apply, or generalize, results beyond the experimentcontext pertains to experiment realism and robustness.


implementation. For example, during refinement,experiments quantify the extent to which proposedcapabilities solve military problems. Experiments alsoexamine capability redundancies and tradeoffs and revealcapability gaps. Prior discovery stage activities onlyspeculate whether proposed further capabilities wouldsolve identified gaps in military effectiveness, whereasexperimentation during refinement empiricallysubstantiates and quantifies the extent proposedcapabilities increase effectiveness in specific caseexamples. In some instances, experimentation maysuggest prototypes for early implementation, or identifyareas needing future investigation. Experiments duringassessment, on the other hand, investigate the robustnessof the solution developed during refinement for possiblefuture military operations. These experiments examinedifferent future contingencies, different multinationalenvironments, and different threat scenarios to ensurethat the refinement stage solution is robust; that it isapplicable to a wide range of potential operationalrequirements in an uncertain future.

Prototypes derived from the earlier stages are often notready for immediate operational use. Experiments duringprototype refinement can transition concept prototypes

17

Experiments are required throughout a capabilitydevelopment and prototyping process. They provide anempirical method to explore new capabilities, to refineconcepts, and to validate new prototypes for


Discovery To clarify future warfighting problemsand to seek potential solutions.

Refinement To examine and refine the extent towhich proposed capabilities or conceptssolve military problems.

Assessment To ensure that solutions from refinementare robust; that they are applicable to awide range of potential operationalrequirements in an uncertain future.

PrototypeRefinement

To transition capability surrogates intopotential operational capabilities bydeveloping complete prototype packagesfor front line commands.

PrototypeValidation

To provide the final demonstratedevidence that the prototype capabilitycan operate within theater and willimprove operational effectiveness.

Stage Aim

Integrated Analysis andExperimentation Campaigns

Principle 4. Defense experiments should be integrated into acoherent campaign of activities to maximize theirutility.

Principle 5. An iterative process of problem formulation,analysis and experimentation is critical toaccumulate knowledge and validity within acampaign.

Principle 6. Campaigns should be designed to integrate allthree scientific methods of knowledge generation(studies, observations and experiments).

Principle 7. Multiple methods are necessary within acampaign in order to accumulate validity acrossthe four requirements.

Experimentation is a necessary tool in addressing largecapability development problems, but this should beembedded in an integrated campaign of experiments,studies and analytical activities. Such Integrated Analysisand Experimentation Campaigns would typically also havean integrated analytical and management process, anduse a variety of techniques to ensure that weaknesses inone technique can be mitigated by others.

19

into potential operational capabilities by developingcomplete prototype packages for front line commands.These experiments develop the detailed tactics,techniques, procedures (TTPs), and organizationalstructures for the prototype as well as developing thetasks, conditions, and standards to facilitate training. Theycan also examine the latest hardware and softwaresolutions and their interoperability with existing fieldedsystems. Experiments during prototype validation providethe final demonstrated evidence to the combatantcommander that the prototype capability can operatewithin theater and will improve operations. Often theseexperiments are embedded within exercises or othertraining events and are used to validate the predictedgains in effectiveness of the force.


• at the technological level: helicopter operations withina combined arms team, surface and sub-surfaceplatforms for maritime operations, and the JSF withinthe air control system;

• at the tactical level: amphibious and airmobile taskgroups;

• at the operational level: the capability balance requiredto achieve the Future Warfighting Concept; and finally,

• at the strategic level: the Effects Based Operationsconcept is being developed in conjunction with manygovernment agencies.

Why use a Campaign

An integrated analysis and experimentation campaign willbe required for a variety of reasons. There may beresource or political reasons why a campaign is preferredto a single activity, or more likely it will be necessarybecause without a coordinated campaign, the problem orissue under investigation simply cannot be satisfactorilyresolved. A campaign allows the problem to beapproached in a coordinated, manageable manner with avariety of analytical techniques and allows a degree ofiteration and synthesis between activities that help ensure

21

Campaigns use a mix of defense experiments and parallelstudies to understand the problem’s context, theassociated warfighting concept and the capabilitiesrequired. The product of the campaign is advice todecisionmakers on the utility, versatility and maturity ofthe concept and the capabilities required to implement theconcept. Campaigns can address issues at all levels fromjoint and combined operations to platforms andcomponents.

An integrated campaign using a variety of techniquesensures that weaknesses in one technique can bemitigated by others. Where results (e.g., inferences)correlate between activities, it increases confidence andwhere they diverge, it provides guidance for furtherinvestigation. It is only when all activities are broughttogether in a coherent manner and the insightssynthesized, that the overall problem under investigationis advanced as a whole.

Such campaigns can address force development issues atany level, for example: technological (e.g., systems ofsystems), tactical, operational, as well as strategic.Instances of activities at each of these levels in Australia,for example, are as follows:


• Synthesis of Military and Analytical Skills. Acampaign, by integrating different techniques, providesimproved opportunity for analytical and military skills tobe applied to the problem.

• Problem Formulation. When the strategic environmentis uncertain and unprecedented, and the impact oftechnology unknown, the experience base is usually toonarrow to conduct the problem formulation confidently.Within the campaign we must therefore build asynthetic experience base and the process ofscientific inquiry is used to increase our confidence inthe problem formulation.

Iterating Methods and Experiments

The initial stage of any campaign is problem formulation.Effective problem formulation is fundamental to thesuccess of all analyses, but particularly at the campaignlevel because the problems are normally ill-defined,complex and adversarial, involving many dimensions anda rich context. Problem formulation involvesdecomposition of the military and analytical aspects of theproblem into appropriate dimensions. Decompositioncannot normally be achieved without detailed analysisusing a matrix of tools such as seminars and defense

23

that the overall problem is sufficiently addressed. Theproblem may initially be ill-defined and a campaign ofactivities will allow assessment and adjustment as theproblem is refined. Some of the analytical reasons forusing a campaign approach are described in the followingsub-sections.

• Problem Characteristics. Military capabilitydevelopment problems are generally complex andcoercive. The socio-technical nature of the system andthe interaction between the components and theenvironment characterize the system as complex. Theimportance of an opposing force, itself a socio-technical system, means the system is coercive. Manyproblems that might be explored through defenseexperimentation are simply too complex to be dealtwith in a single activity.

• Increased Confidence. An integrated campaign ofexperiments and other activities allows a gradual build-up of the knowledge surrounding the problem or issueunder investigation, leading to a more refined androbust concept. This increases confidence that thefindings are valid and creates a systematic body ofknowledge to inform and investigate capabilitydevelopment.


Wargames, and in particular seminar wargames, have animportant role in problem formulation. In wargaming it ispossible to balance the physical and psychological aspectsof the problem by using warfighters as the players whileadjudicating their actions using models or rulesets. Mostimportantly, wargaming introduces an adversary early inthe problem formulation process, providing a stressfulenvironment in which to explore the concept and developthe hypotheses for subsequent analysis. Although human-in-the-loop simulations and live simulations also introducea human adversary, they are frequently too expensive andunwieldy for the problem formulation phase.

25

experiments supported by analytical studies andoperational experience. Detailed analysis also assists inthe reconstruction of the problem segments andinterpretation of results.

In dealing with fuzzy or uncertain interactions, the problemformulation process needs to explore and understand thesignificance of each interaction before making (or seekingfrom customers) assumptions about it. This involveskeeping an open mind, during the early stages of problemformulation, about where the boundaries lie and theirdimensional nature. This is difficult because it makes theprocess of modeling the problem more complicated. A callfor hard specification too early in that process must beavoided. In the end, of course, the problem must beformulated in order to solve it, but formulation should be anoutput from the first full iteration, not an early input to it.

As shown in the following illustration, the problem is beingformulated and refined throughout the entire campaign inan iterative cycle that never really completes until thecampaign itself completes. The process of problemformulation and analysis is undergoing constant review toreshape the direction of the campaign and to ensure thatthe real issue or concept is being addressed.


Coherent Management and Communication Framework

Problemformulation

Full range of underpinning techniques, e.g.:Seminar wargaming; analytic wargaming; constructive simulations; HITL virtual experiments; field experiments; analysis of real operations

Analysis Problemformulation Analysis

On-going campaign analysis

to support and help inform the experimenters who willultimately address the overall question. The campaignplan process must take these other activities into accountwithin its design phase. The ultimate aim is to synthesizethe outputs from all activities into coherent advice to thedecisionmakers.

Different Methods Offer Different Strengths

All experiments must strike a balance among the fourexperiment validity requirements. Attempts to satisfy onework against satisfying the other three. Consequently, 100percent-valid experiments are unachievable. Precision andcontrol increase the ability to detect change and to isolateits cause, but decrease the ability to apply the results toimprecise, real-world situations. Experiments designed toidentify change emphasize strict control of trial conditionsand feature multiple repetitions of similar events;experiments designed to relate results emphasize free-play, uncertainty, and reactive threats. Each individualexperiment design must consider requirement tradeoffs inorder to minimize the loss of one requirement due to thepriority of another.

27

Integration of Scientific Methods

The aim of a campaign is to integrate a range of methods:experiments (observations with manipulation–empirical-deductive); observational studies (observations withoutmanipulation–empirical-inductive) and analytical studies(rational-deductive) into a coherent package thataddresses a complex capability development problem. Thephases of campaign design are the same as for anyevaluation, that is, problem formulation and analysis. Thecomplexity arises because after the completion of eachactivity the problem formulation is reassessed andadjusted and subsequent activities may be redesigned. Asa result a campaign plan is a flexible instrument, with asupporting risk-management framework and an iterativeapproach to constantly review and reshape the remainderof the campaign to ensure that the overall goals areachieved.

In all likelihood, seminars, workshops, historical analysis,and the like, will also be required as part of the campaign


Experiment n

Campaignplanning

anddesign

Experimentdesign

Implementation

Campaignplanning

anddesign

Experimentdesign

ImplementationExperiment Analysis

Questions

Insight andanswers

Experiment

Experiment n+1

Analytic wargames typically employ command and staffofficers to plan and execute a military operation. At certaindecision points, the Blue players give their course of actionto a neutral, White cell, which then allows the Red playersto plan a counter move, and so on. The White celladjudicates each move, using a simulation to helpdetermine the outcome. A typical analytic wargame mightinvolve fighting the same campaign twice, using differentcapabilities each time. The strength of such wargames forexperimentation resides in the ability to detect any changein the outcome, given major differences in the strategiesused. Additionally, to the extent that operational scenariosare used and actual military units are players, analyticwargames may reflect real-world possibilities. A majorlimitation is the inability to isolate the true cause of changebecause of the myriad differences found in attempting to play two different campaigns against a similar reactive threat.

29

Most defense experiments use some form of simulation,which can be grouped into one of four general methods:constructive simulation, analytic wargames, human-inthe-loop simulation, and live (field) simulation. Each ofthese four methods has its own strengths and weaknesseswith respect to the four experiment validity requirementsdiscussed previously. Since one particular method cannotsatisfy all four requirements, an integrated analysis andexperiment campaign requires multiple methods.

Constructive simulations are those in which no humanintervention occurs in the play after designers choose theinitial parameters and then start and finish the simulation.Constructive simulations are a mainstay of militaryanalytical agencies. They allow repeated replay of thesame battle under identical conditions, whilesystematically varying parameters– the insertion of a newweapon or sensor characteristic, the employment of adifferent resource or tactic, or the encounter of a differentthreat. Experiments using constructive simulations withmultiple runs are ideal to detect change and to isolate itscause. Because modeling complex events requires manyassumptions, including those of variable human behavior,critics often question the applicability of constructivesimulation results to operational situations.


Live simulation is conducted in the actual environment,with actual military units and equipment and withoperational prototypes. Usually only weapon effects areactually simulated. As such, the results of experiments inthese environments, often referred to as field experiments,are highly applicable to real situations. Good fieldexperiments, like good military exercises, are the closestthing to real military operations. A dominant considerationhowever, is the difficulty in isolating the true cause of anydetected change since field experiments include much ofthe uncertainty, variability, and challenges of actualoperations; but they are seldom replicated due to costs.

Different Methods during CapabilityDevelopment and Prototyping

As potential capabilities advance through capabilitydevelopment and prototyping stages, the followingconsiderations are useful in selecting which of the fourexperiment validity requirements to emphasize. Forexample, finding an initial set of potential capabilities thatempirically show promise is most important in therefinement stage. Experiments in this early stage examineidealized capabilities (future capabilities with projectedcharacteristics) to determine if they lead to increased

31

Human-in-the-loop simulations represent a broadcategory of real-time simulations with which humans caninteract. In a human-in-the-loop defense experiment,military subjects receive real-time inputs from thesimulation; make real-time decisions, and direct simulatedforces or platforms against simulated threat forces. Theuse of actual military operators and staffs allows this typeof experiment to reflect warfighting decisionmaking betterthan experiments using purely constructive simulation.However, when humans make decisions, variabilityincreases, and changes are more difficult to detect andconsequently to attribute to the cause.


Rigorous experimentation requires multiple methods to meet the four validity requirements.

Hybrids are also possible

ConstructiveSimulationUsually fasterthan real time

simulated forceswith no human

interaction duringexecution.

AnalyticWargames

Human Plannerswith intermittentinteraction with(usually faster than real time)

simulated forces.

Human-in-the-Loop Simulation

Humans withcontinuous, realtime interactionwith simulated forces and/orequipments.

Live Simulation

Actual Forces in a live (field)environment

with simulatedweapon effects.

Requirements for a Good Experiment

Employ Capability +++ ++ +Detect Change in Effect +++ ++ ++ +Isolate Reason for Effect +++ ++ +Relate Results to Operations + ++ +++

Capitalizeon StrenghtsUse combination for most rigorous

conclusions

realistic experiment environment. Isolating the real causeof change is still critical when improving prototypes. Theexperiment must be able to isolate the contributions oftraining, user characteristics, scenario, software, andoperational procedures. As previously described, human-in-the-loop and field experiments provide the opportunityfor human decisionmakers to influence development. Inprototype validation, human decisionmakers ensure thatthe new technology can be employed effectively. Prototypevalidation experiments are often embedded within jointexercises and operations.

Employing Multiple Methods to Increase Rigor

Since experiments using the four main simulationmethods emphasize the four validity requirementsdifferently, an integrated analysis and experimentationcampaign must capitalize on the strengths of each methodto accumulate validity. For example, the model–exercise–model paradigm integrates the strengths of, on the onehand, the constructive simulation (i.e., “model”) and, onthe other, any of the methods that involve humaninteraction (i.e., “exercise” in a generic sense). Thistechnique is especially useful when resource constraints

33

effectiveness, and are dependent on the simulation-supported experiment, using techniques such asconstructive simulation, analytic wargames and human-in-the-loop simulation. Accurately isolating the reason forchange is not critical at that stage, as the purpose is onlyto apply a coarse filter to the set of idealized capabilities.However, during the assessment stage, quantifyingoperational improvements and correctly identifying theresponsible capabilities is paramount in providingevidence for concept acceptance. This is also dependenton experiments with better-defined capabilities acrossmultiple realistic environments. Experiments conductedusing constructive simulation can provide statisticaldefensible evidence of improvements across a wide rangeof conditions. Human-in-the-loop and field experimentswith realistic prototypes in realistic operationalenvironment can provide early evidence for capabilityusability and relevance. Early incorporation of the humandecisionmaker in this way is essential, as the humanoperators tend to find new ways to solve problems.

In prototype refinement experiments, one shouldanticipate large effects, otherwise its implementationmight not be cost effective. Accordingly, the experimentcan focus on the usability of working prototypes in a


ability to understand fully the implications of theexperiment results by conducting “what if” sensitivitysimulation runs. Experimenters examine what might haveoccurred if the Red or Blue forces had made differentdecisions during the experiment.

The model–exercise–model method increases overallexperiment validity by combining the contrasting strengthsof the following methods:

1. experiments using constructive simulation, which isstrong in detecting differences among alternative treat-ments, and

2. experiments using either human-in-the-loop simulation,analytic wargame, or field experiments, which arestronger in incorporating human decisions that betterreflect the actual operating environment.

This paradigm also helps to optimize operationalresources by focusing the exercise event on the mostcritical scenario for useful results, and by maximizing theunderstanding of the event results through post-eventsensitivity analysis.

35

prohibit conducting side-by-side baseline and alternativecomparisons during wargames and field experiments.

In the model-exercise-model paradigm, the earlyexperiments using constructive simulation examinemultiple, alternative, Blue-force capability configurationsand baselines. Analysis of this pre-exercise simulationallows experimenters to determine the most beneficialBlue-force configuration for different Red-force scenarios.An analytic wargame, human-in-the-loop or fieldexperiment can then be designed and conducted, whichprovides independent and reactive Blue- and Red-forcedecisionmakers and operators. One can then re-examinethis optimal configuration and scenario.

Experimenters use the results of the exercise to calibratethe original constructive simulation for further post-eventsimulation analysis. Calibration involves the adjustment ofthe simulation inputs and parameters to match thesimulation results to those of the experiment, thus addingcredibility to the simulation. Correspondingly, rerunningthe pre-exercise alternatives in the calibrated modelprovides a more credible interpretation of any newdifferences observed in the simulation. Additionally, thepost-exercise calibrated simulation improves analysts’


experiments. These considerations relate to the need torecognize and accommodate the human element inexperiment design, and they also provide advice on how tomake the best use of operational test and evaluationevents or training exercises. They also give guidance onsome issues relating to modeling and simulation, on theimplementation of good experiment control and highlightnational regulations, security rules and practices that mayneed special consideration; and finally, there are alsosome practical steps that can be taken to achieve goodcommunications.

Human Variability

The implications arising from using human subjects indefense experimentation are often overlooked. Most, if notall defense experiments examine impacts on socio-technical systems but experiment designs rarely catersufficiently for the human element. Because humans areunique, highly variable and adaptable in their response toan experimental challenge, they are more than likely tointroduce a large experimental variability. In addition,humans will have different experiential baselines in termsof, for example training and aptitude and, unliketechnology, will become tired and possibly demotivated.

37

Considerations for Successful Experimentation

Principle 8. Human variability in defense experimentationrequires additional experiment designconsiderations.

Principle 9. Defense experiments conducted during collectivetraining and operational test and evaluationrequire additional experiment designconsiderations.

Principle 10. Appropriate exploitation of modeling andsimulation is critical to successfulexperimentation.

Principle 11. An effective experimentation control regime isessential to successful experimentation.

Principle 12. A successful experiment depends upon acomprehensive data analysis and collection plan.

Principle 13. Defense experiment design must considerrelevant ethical, environmental, political,multinational, and security issues.

Principle 14. Frequent communication with stakeholders iscritical to successful experimentation.

This guide identifies a number of considerations that areintended to support the practical implementation of


possible to measure learning effects within eachtreatment, and thus estimate any confounding effect oflearning between treatments. Of course, this may increasethe complexity of the experiment design as the dataanalysis will then also need to control for human variabilitymeasures and assess their impact upon the mainvariables.

Although objective measures of variables are favored byexperimenters, subjective measures are important forascertaining the mental processes underlying observedbehaviors. This information may be important, especially ifa subject adapts to using a capability in a way notconsidered by the experimenter. Asking subjects why theyhave changed their behavior can enhance understandingof maladaptive ways of using of a new capability.Consideration needs to be given to the timing of subjectiveinterviews, particularly whether they should take placesoon after the action occurs or at the end of theexperiment. The former may be obtrusive to the subjectsand may impact the results, with the latter being affectedby factors such as memory decay and motivation.

39

They may also learn during experiments. The experimentdesign and the data analysis and collection plan mustrecognize and accommodate human variability, which willbe much larger than would be predicted if the socio-technical system were treated solely as technology. Whatis sometimes overlooked is that this variability providesimportant information on why a socio-technical systemresponds to a challenge in a particular way. Indeed thereis an argument that human variability should not beminimized, as this would lose important information. Highvariability may indicate a fault in the system underexamination, or in the experiment design. Anunderstanding of the impact of human variability onexperiment design and outcome is a fundamental skillrequired by all experimenters.

Regardless of the experimenter’s ability to control humanvariability, it is important, if possible, to measure it. This isdone mainly to see if detected effects can be explained interms of human variability rather than the experimentaltreatments. For example, where a single group is thesubject for all the treatments, then learning by that groupduring and between the treatments may have aconfounding effect on the whole experiment. It may be


Exploiting collective training (exercises) has a range ofbenefits as well as disadvantages and a variety of factorsmust be taken into account in both planning andexecution. The principal one is that training always hasprimacy and the experimenter has little control overevents, thus the skill is in understanding the constraintsthat the exercise opportunity will present and knowinghow to work within them. Exploiting training exercises forthe purposes of experimentation is most achievable duringthe prototype validation phase of a capability developmentprogram when functional prototypes exist.

The potential to include experimentation within OT&Eprograms is very high. This is so in part because many ofthe components of OT&E events are the same as theircounterparts in experiments. They are well supported bythe technical/engineering community and valued by theoperational community as a component of the operationalreadiness process. The operational community willtherefore generally be engaged in OT&E events and thepotential to include experiments in these events as wellcan be very good. An important benefit to experimenters is the OT&E infrastructure, which includesengineering/technical staffs and facilities; planningsupport; test support during execution and evaluation

41

Exploiting Operational Test andEvaluation and Collective Training Events

Opportunities to conduct experimentation may be found intraining exercises and in operational test and evaluation(OT&E) events. Operational assessments, in particular,provide an opportunity for conducting experimentationwith substantial technological and expert staff support.The drive to conduct experimentation activities duringtraining exercises and OT&E events is almost entirely dueto the difficulty of acquiring the resources (equipment,estate, human) to undertake defense experiments of anysignificant size. Arguably, the equipment programs thatrequire most support from experimentation are thoseintended to enhance collective rather than team orindividual effectiveness, and thus collective groups ofpersonnel (which may comprise command teams withhigher and lower controllers) are required to undertakethat experimentation. It is a simple fact of life in the early21st Century that most nations generally do not have unitsand formations available to dedicate to experimentation,except for the most limited-scale activities. Thereforeexploiting routine training exercises and other collectiveevents should be given serious consideration.


exercise enhances the usability of the experimentalcapability and should ensure that it will function correctlyduring the exercise trials. This is less of an issue for OT&E,as this activity is generally for validating the performanceof new operational systems and the testing is implicit.Additionally, to address the second experiment validityrequirement in training exercises, i.e., the ability to detecta change in the effect, establishing a pre-exercisedefinition of expected performance and comparing theprototype’s actual performance during the exercise to itsexpected performance provides the necessary ability todetect change. For OT&E, the performance of newoperational systems is typically documented in manualsand validated computer models may exist. Therefore, thebaseline system performance should be well establishedand the potential for detecting change should be good.

While the ability to isolate the reason for the observedchange effect, i.e., the third experiment validityrequirement, is the most problematic in experimentationembedded in training exercises, experimenters cannevertheless achieve some level of satisfaction here aswell. When examining different capabilities during a singleexercise, the experimenter should conduct different

43

support for the after-action review or report (AAR). Thebenefit from the use of OT&E staffs and facilities isrealized because of the strong overlap between the twoprocesses. An important benefit to the OT&E community isthat the prototypes from experiments may soon beoperational systems. In such circumstances, there is asignificant advantage to be obtained by the inclusion ofOT&E staffs in experimentation on these systems.

Although training exercises and OT&E events do not allowexecution of elaborate experiment designs because itwould impede training and impact operational readiness,scientific methodology and the four experiment validityrequirements can be applied to such embeddedexperiments. Experimentation in these situations naturallyprovides the strongest venue for meeting the fourthexperiment validity requirement, i.e., the ability to relateresults to actual operations. While operational necessityrestricts the ability to meet the first three experimentvalidity requirements in training exercises, and to a lesserextent in OT&E events, the experimenter can amelioratethe limitations to some degree. With respect to the firstexperiment validity requirement, i.e., the ability to use thenew capability, prototype testing prior to the training


45

prototype trials at different times so the effects of oneprototype do not influence the effects of the other. It isprudent to have an experienced exercise “observer-controller” view the prototype trial to assess the extentthat any observed results were the results of theexperimental capability instead of unintended causes.Additionally, showing that the rigorous experiment dataaccumulated during the concept development phase ofthe prototype is still relevant to the exercise conditionsalso supports GUIDEx third experiment validityrequirement. Experimentation embedded in OT&E eventsalso creates considerable challenges for meeting the thirdexperiment validity requirement. The best approach in thiscase is through comprehensive, detailed data collection,which is typically the case in OT&E events anyway.

Finally, for both the use of training exercises and OT&Eevents, a Model-Exercise-Model paradigm that wassuccessfully calibrated to the event results would allowfollow-on sensitivity analysis to demonstrate that inclusionand exclusion of the experimental capability accounted fordecisive simulation differences.



46

Training Exercises

Benefits• Availability of experimental subjects in large numbers• High level of engagement of experimental subjects• Use of training infrastructure • Moderate sample sizes, for repeated exercise series• Ability to use repeated exercises as a control group,

or baseline• They rate highly in terms of relating any detected

change to real operations.

Constraints• Exercises are designed to stimulate various training

points that may not satisfy an experiment design• Training has primacy– can a genuine experiment

design be fitted around training?• Scenarios and settings designed for training purposes• Limited opportunities to make intrusive changes to the

exercise or collected data intrusively• Can results be published without breaching the

anonymity of the training audience? • Interventions by Exercise Control for training reasons,

e.g., the training force is winning too easily• Exploitation of an exercise too early in a unit’s training

cycle can yield poor results, e.g., the collective skillsmay be too low.

OT&E Events

Benefits• Availability of operational staff and platforms• High level of engagement of technical community• Use of OT&E infrastructure • Moderate sample sizes, for repeated test series• Ability to use repeated tests as a control group,

or baseline• Strong potential for relating any detected change to

real operations.

Constraints• OT&E events are designed to quantify aspects of

equipment performance or to determine if a standard isbeing met that may not satisfy an experiment design

• OT&E has priority and the experiment may not interferewith test objectives

• Scenarios and settings designed for OT&E purposes• Limited opportunities to make intrusive changes to the

test or collected data intrusively• Can results be published without breaching the

anonymity of the test audience?

simplifying it. In “The Lanchester7 Legacy” [Bowen andMcNaught 1996: Vol. III, Ch. 9], the authors wrote: “It haslong been understood by Operational Analysts that, indealing with complicated situations, simple models thatprovide useful insights are often to be preferred to modelsthat get so close to the real world that the mysteries theyintend to unravel are repeated in the model and remainmysteries.” We can therefore imply an axiom that M&Sshould be as simple as possible while remaining adequatefor the task in hand.

M&S definition

It is a key principle that the definition of the M&S to beused in an experiment should be derived from theexperiment design, and not the other way around.However, rarely will practitioners have the luxury ofcompleting their experiment design and then movingthrough a user requirements and subsequently systemrequirements definition process in sequence. Usually aconcurrent process is necessary, with the processesbeginning in the order given above. A spiral developmentprocess can then take place. There are several well-

49

Modeling and Simulation Considerations

This guide presents modeling and simulation (M&S) asintrinsic to conducting most defense experiments. There isnow a wide range of M&S techniques available and thismakes the innovative use of M&S cost effective for manydefense experimentation applications. However, there aresome significant issues associated with selecting both thetypes of M&S to be used and the specific elements of theexperiment federation.

A balanced view of fidelity and validity

For many years, as rapidly increasing computing powerled to many new modeling possibilities, there was agenerally held view that greater fidelity, or accuracy, wasalways better. Indeed, many took the term “validity” to bealmost synonymous with fidelity and detail. The modernview is that validity actually means “fit for purpose,” withthe purpose being to execute the desired experimentdesign. This means that we should consider the mainmeasure of merit for M&S to be adequacy, not fidelity. Theexperiment design should effectively define what level offidelity is adequate. Furthermore, the main point ofmodeling is to rationalize the complexity of real life by


7 F.W.Lanchester was one of the pioneers of military operationalresearch.

regardless of whether a strict model-exercise-modelparadigm is being followed. In particular, architecturalframeworks such as Zachman [Zachman 1987] andDoDAF8 represent an excellent and increasingly popularmeans to describe military problems and potentialcandidate solutions in a variety of different ways. When amodel-exercise-model paradigm is being followed,process models based on these frameworks can often bepreferable to complex constructive combat simulations.

Experiment Control

Experimentation is intrinsically a controlled activity,although the degree of possible and required controlvaries from case to case. The experiment design should beexplicit in describing which variables must be controlled inorder to prevent rival explanations for the findings, andwhich variables can be allowed to remain uncontrolledthough usually recorded. It should also describe thecontrol regimes to be put in place to ensure that thisoccurs in practice. The identification of interveningvariables and learning effects must be well understood.However, simply outlining the required measures in the

51

established processes for achieving this, e.g., the USFederation Development and Execution Process (FEDEP)and the European Synthetic Environment Developmentand Exploitation Process (SEDEP).

Modeling the process to be examined by the experiment

Experiments and observational studies (where a conceptis subjected to objective observation, but withoutmanipulation) are intrinsically connected to the idea ofhypotheses. The hypothesis is simply a plausibleproposition about either causal or associativerelationships. Thus in a general sense there is alwaysimplicitly a model of the process being experimented withby virtue of there being one or more hypotheses. However,it is possible, and in most cases desirable, to model theprocess in advance in a much more tangible way,


Experiment Design

Define User (or Experiment) Requirements

Define System (M&S) Requirements

8 DoD Architecture Framework, see [DoDAF Working Group 2004]

Experiment Planning

The planning of major defense experiments requires amanagement team, which takes the decisions required tosettle high-level issues, has oversight on the activities ofthe various teams, and ensures that the experimentplanning and organization develops toward the objectivesin a timely manner. A series of reviews throughout theplanning period is usually necessary to ensure that theprocess of preparing for the experiment is remaining ontrack. For larger experiments, e.g., joint or coalition ones,it is common to employ conferences for this purpose,organized and run by the management team; typicallythree or four might be used.

Experiment Execution

The experiment management team usually transforms intothe control staff during execution. The controller’s role is toensure that the experiment is progressing according toschedule or to be on top of the situation if it is not. Thecontroller observes the players and collects their inputdaily and works closely with the analysts in monitoring theprogress of the experiment. The controller providesfeedback to the experiment director and implementschanges as required to ensure the event achieves the

53

experiment design document is not sufficient. Theexperiment director and his team must actively seek toimpose the required controls throughout the planning andexecution phases of the experiment.

Experiment Design

The experiment design process is a logical journey fromthe questions to be answered, or hypotheses to be tested,to the detailed definition of the experiment. Thus theexperiment design is the cornerstone of the control regimethroughout the life of the experiment, since it sets out inbroad terms what needs to be done. Success in designingexperiments is rooted in early stakeholder engagement toestablish objectives and intent. An integrated analysis andexperimentation campaign goes a long way towardproviding the framework for detailed stakeholderguidance. Furthermore, nothing allows for the control ofvariables during experiment design more than early, firmdecisionmaking. The longer decisions on scenario,participation, funding, technical environment, and studyissues are allowed to linger, the more options theexperiment designers must keep open and the harder it isto control the variables that can affect the outcome of theexperiment.


Data Analysis and Collection

Data collection is designed to support the experimentanalysis objectives that in turn rely on a conceptual modelunderlying the experiment. The data analysis offers theopportunity to revisit the underlying conceptual modelidentified for the experiment and determines cause-and-effect relationships. A data analysis and collection plan isan essential part of an experiment.

A significant part of the experiment consists of gatheringdata and information. Interpreting the information intofindings and combining them with already knowninformation to obtain new insights tends to be challenging.Once it is determined what needs to be measured, adecision is required to identify the data necessary and toanalyze it using appropriate (usually statistical) analysistechniques. The plan ensures appropriate and valid dataare generated and that the key issues of the experimentare addressed. When determining analytical techniques touse, an estimate for the number of observations must beconsidered, depending on the expected variability in thedependent variables and the number of them. It isessential to prioritize and ensure there are sufficientobservations for all objectives, measures of performance,and measures of effectiveness requiring analysis. There

55

experiment objectives. In doing so, the controller mustdeal with military judgment (observations from theplayers) and scientific objectivity (input from the analysts).

Experiment Analysis

The analysis or assessment team for an experimentshould ideally be derived at least partly from theexperiment design team, and they should work closelywith the team responsible for the concept underexperiment and the team responsible for providing theexperiment’s technical environment. Initially, they shouldreview the concept and approach planned to conduct theexperiment and prepare an analysis plan to meet theneeds of the experiment design. During the course of anexperiment, analysts compare observations and resultsand begin to integrate their views of what is being learnedfrom the experiment. As sufficient data is collected,analysts begin to form preliminary insights. However, thetemptation to announce some startling finding (especiallyone that it is believed the experiment sponsor will like)should be resisted at all costs, because it is quite likelythat when the analysis is complete, that finding will at bestneed to be modified, and at worst, changed altogether.Thus, first impressions should generally be conservative;this is an important control consideration.


documentation about what happened during theexperiment and can be used to explain why certain resultsoccurred.

Ethics, Security and National Issues

This guide describes a number of different aspects ofdefense experimentation. However, in addition, distinctivenational regulations, security rules and practices shouldnot be underestimated and proper consideration must begiven to them in planning experiments.

Environmental considerations

Wherever there is live activity, there will be some level ofenvironmental impact. In particular, great care must betaken regarding proximity to historical or cultural sites. Aswell as legal and multinational environment issues,environmental constraints generally will have an impacton the scope of any live experiment or exercise. It isessential that results be interpreted in the light of allenvironmentally imposed artificialities. The test andtraining communities have been working withenvironmental issues for years and there is no reason forthe experimentation community to deviate from thevarious protocols that already exist.

57

exist various types of collection mechanisms used inexperiments.

Questionnaires (also referred to as surveys) are often usedin data collection. They can be used to gather numeroustypes of information. The participants’ background can beobtained through this means. This can be done before thestart of the experiment. The participants can also bequestioned about aspects of the experiment such as theirperceptions about the systems and processes tested, theirview on others participating, strengths and weaknesses ofthe systems and processes as well as recommendedimprovements.

With information systems becoming more crucial,Automated Collection Systems to collect data are nowmore important. It is important to determine what clockeach system that is used to collect data is synchronized toin order to facilitate analysis.

Observers have an important part in the experiment bycapturing interactions between participants. For instancethey take notes about what is going on, crucial eventstaking place, notable behaviors and other such activities.Observers can also be used to provide a chronologicalnarrative of the events that occurred. This provides


issues. By recruiting subjects to undertake an experiment,or by exposing the data collector to a potentially hazardousmilitary environment the experimenter is expecting themto operate outside their normal working practices.Although ethics is a complex field, its fundamentalconcerns in professional contexts can be defined.Research that lacks integrity is considered to be ethicallyunacceptable, as it not only misrepresents what it claimsto be but also misuses resources. In addition, there is anobligation for defense experiments to comply with relevantnational Health and Safety legislation and to provideworking conditions that would ensure, as far asreasonably practicable, a healthy and safe workingenvironment for experimenters and subjects alike.

Communication with Stakeholders

The final product of any defense experiment must be theevidence that the right question has been addressed andthe evidence required for its findings to be exploitedeffectively. This will also provide the experimenter with thenecessary foundation for advising on the applicability andfeasibility of advancing an evaluated concept, or elementsof a concept, toward eventual realization as actualoperational capabilities. Good and continuouscommunication is central to achieving such a successful

59

Security considerations

Even within single-nation experiments, security issues cangive rise to real practical problems. In particular, the rise ofsecure digital command, control, communications,computers and intelligence (C4I) and sensitiveintelligence, surveillance, target acquisition andreconnaissance (ISTAR) sources (which are oftenthemselves at the centre of the experiment purpose) hasresulted in security considerations becoming much moreprominent in the design and execution of defenseexperiments than hitherto. As a general rule, the lower thesecurity classification of these elements, the lower thecost and risk of the experiment and thus experimentsshould be run at the lowest classification level possible.This is not to say, of course, that undue efforts should bemade to make everything unclassified or artificially low inclassification. As previously discussed, all experiments arecompromises, and the experimenter needs to decidewhere the benefits of (for example) higher classificationand therefore higher fidelity representations ofequipments or scenarios outweigh the benefits of usinglower classification analogues.

Ethics considerations

Any experiment, which involves human subjects andhuman data collectors, could potentially pose ethical


who need to be influenced. However, the question mayarise from many sources and it may not always bepossible to directly engage or even identify the originalsource. For example the question may have arisen from astrategic plan, which states that “there is a need toenhance interoperability with our allies to a level whichwill allow us to undertake concurrent medium scaleoperations.” This will reflect a political imperative, andwhoever is responsible for the strategic plan may haveappointed intermediaries whose task is to implement thisdirective. In this case, these are all key stakeholders, andit is essential to determine their relationships and howthey work together. Intermediaries will have formed theirown understanding of the question being posed anddefined a campaign to implement the directive.

Communicating in the run up to the experiment

Although this will be a particularly busy period, it isessential that regular dialogue be maintained with thestakeholder community prior to the experiment. Bymaintaining this regular dialogue, changes in priorities canbe quickly identified and accommodated.

61

outcome; and yet it is still possible to find an experiment,or integrated analysis and experimentation campaign,which does not have a rational plan for communicatingwith stakeholders.9 A communications plan must considerhow the different stages in running an experiment mayrequire different approaches to good communication;stages such as determining the right set of questions andissues to be addressed, maintaining the confidence of keystakeholders that the potential changes to their prioritiesare being considered, ensuring all stakeholders haveappropriate access during the experiment and makingsure that they understand the output.

Determining the right set

of question and issues

A key prerequisite to a single experiment or campaign isthe identification of the origins of the question to beaddressed and identification and commitment of keystakeholders. One difficulty is that the obvious stakeholderis often not the person that originally posed the question.Therefore an initial step must be to chase down the originsof the question, and from that define the key stakeholders


9 Stakeholders are defined as persons who have a vested interest in theproduct from the experiment or campaign.

A far better approach is to continue the dialogue with thekey stakeholders to determine how the work has beenreceived, to assist in interpreting results and, moreimportantly, to advise on how it should be exploited. Wherethe experiment is part of a wider campaign supportingconcept or capability development, the experimenter mayalso have the opportunity to advise on the consequencesfor the over-arching concept of the particular experimentfindings.

63

Communicating during the experiment

In most cases the major interaction with stakeholdersoccurs during the visitor day. Visitors should beencouraged to view the entire experimentation processfrom the pre-brief to the post exercise wash up, andinvited to observe and interact with the subjects in a waythat does not interfere with the experiment. Additionalattendance outside the specific visitor day of stakeholderswith a direct involvement in the campaign implementationimproves communication in that they are then briefed atregular intervals.

Communicating after the experiment

A well-written report will contain a one-page abstract, anexecutive summary and a full report. The traditionalapproach to dissemination of results has been to producea paper that is sent to key stakeholders, with or without apresentation. While this has obvious merits the generalexperience is that this approach tends to produce “shelf-ware.”10 It should be remembered that these are busypeople who will wish to gain quick appreciation of the keyissues and findings, in order to exploit the information.


10 A UK term, which means that the report is produced but never read in full.

(in purple) from the specific individual experiment stages(in orange). The grey areas indicate the products of theexperimentation process, while green shows the customeror stakeholder interactions. The flowchart itself beginsfrom the green cloud at the top-left hand corner,representing the initial problem, as posed by the customer.

The campaign of integrated analysis and experimentationthen commences with a number of iterations around thecampaign problem formulation and campaign design loopin order to develop with the customer an agreedcampaign-level problem statement. During this processthe campaign designer begins to identify the analyticalmethods and experiments that might be used to answerthe problem. Once a required experiment is identified, themore detailed process of experiment problem formulationcan begin. Again, the flowchart suggests that the problemformulation should iterate and overlap with the experimentdesign in order to ascertain the problem scope suitabilityfor experimentation imposed by real-world considerations.A number of potential experimental questions may requiresome initial design work to be undertaken before anacceptable, workable and useful problem defined can thenbe submitted to a complete experiment design anddevelopment. The lesson is “be prepared for exploratory

65

GUIDEx Experiment and Campaign Planning Flowchart

In order to help practitioners in applying the GUIDExprinciples to address their specific problems, the followingflowchart was developed. This is by no means aprescriptive recipe for perfect experimentation, but anattempt to lay out the chronological sequence forexperiment and campaign related activities and to showthe iterations and linkages between various stages of theexperimentation process. Indeed, GUIDEx encourages thatthe specific application of Principles to a given problemshould be tailored according to the scale and nature of theissue under investigation. There is no single “best” way toundertake experimentation, rather the skill of thepractitioner is to use a degree of artistic license in applyingthe science advocated within GUIDEx in order to maximizewhat can be achieved for a given problem under real-world constraints of resources, time, expectation andunderstanding.

The color code of the flowchart separates the integratedanalysis and experimentation campaign activities



66

activities or false starts before one can move forward witha good concept for detailed design.”

The flowchart outlines some of the products needed forsuccessful experimentation, such as analysis and datacollection plans, technical development requirements,ethics and safety plans and finally joining instructions forthe participants. The practitioner’s role at this stage is tomanage the competing demands of technicaldevelopment, customer and player expectation, legislativerequirements, rehearsal and training requirements whilestill maintaining overall control of the scientific andanalytical rigor. Finally the experiment itself is executedand the process of analysis and reporting can begin.

In general as the individual experiment is being planned,designed and undertaken, the campaign analysiscontinues and once the results from the experimentemerge from the collected data, the campaign itself mayevolve to take account of the knowledge gained. Lessonsmust be assimilated. If necessary, further experimentationor analytical activities can be undertaken and the cyclerepeats. Throughout this entire process, the interactionwith the customer is key to ensuring that the answersgenerated do indeed answer the questions posed.


68

21 Threats to a Good Experiment

Building on the work of Cook and Campbell [Cook andCampbell 1979], one can identify the things that can gowrong in an experiment. Cook and Campbell call thesethreats to validity, in other words, they are identifiedproblem areas that can cause one to not meet any one ofthe four experiment requirements presented from page 10to 15 of this Pocketbook. While Cook and Campbellidentified 33 threats to validity, they have been combinedand distilled down to 21 potential threats to defenseexperiments. Moreover, they have been rearranged into atwo-dimensional matrix to better systematically illustratehow the threats to experiment validity can be understoodand treated with respect to each of the four requirementsand the five experiment components. Additionally, manynames of their threats to validity have been changed toreflect military experiment terminology. For example,learning effects is the substitute of Cook and Campbell'smaturation.

All good experiment practices are then ways to eliminate,control, or ameliorate these threats. A good experimentplan would show how each has been accounted for andcountered.

GUIDEx Case Studies

The following is a high-level overview of the results of theeight Case Studies offered by GUIDEx.

1. Testing Causal Hypotheses on Effective Warfighting:This was a series of experiments for a commonoperational picture (COP) experimental treatmentcondition using a Persian Gulf air/sea scenario where allparties—higher echelon and lower echelon—had boththe national intelligence supported big picture and thelocal tactical picture. This combination was experimentallyproven to be superior technology for such operations,resulting in greater shared situation awareness and betterbottom line combat effectiveness.

2. UK Battlegroup Level UAV Effectiveness: Thisexperiment supported a major UK unmanned air vehicle(UAV) acquisition program in demonstrating the hugeinformation gathering potential of UAVs at the tacticallevel, compared to existing ISTAR assets. However, equallyimportantly, it showed that if integration into thesupported HQs is not achieved effectively, then theresulting information overload can have a hugelydetrimental effect on mission success.

71

The two-dimensional framework of this Table provides asubstantial advantage over the traditional “laundry list” ofgood practices. The framework associates different goodpractices with each of the four experiment requirements.This facilitates understanding why particular goodpractices are important and the impact on experimentvalidity if the threat is not properly attended to. Forexample, it is impossible to implement all of the goodpractices in any particular experiment. Thus, anunderstanding of the impact of unimplemented goodpractices is critical to designing the “best available”experiment. Furthermore, associating good practices withthe different experiment components allows theexperiment designer to see the interaction of goodpractices across all aspects of the experiment. Fortunately,when developing an experimentation campaign, one canachieve a higher level of fulfillment of the good practicesby using the particular power of complementaryexperimentation approaches.


73

3. UK NITEworks ISTAR Experiment: The UK, like othernations, is presently investing heavily in ISTAR sensorsand systems. However, it is widely recognized thateffective information requirements management (IRM) isvital to the efficient use of those systems. This experimentinvestigated both technological and procedural means ofimproving IRM. It showed conclusively that a collaborativeworking environment with appropriate working practiceswould have a major beneficial effect on IRM effectiveness.This assisted the development of ISTAR managementpriorities in the UK.

4. Pacific Littoral ISR UAV Experiment (PLIX): This CaseStudy provides insights difficult to capture withoutexperimentation; the strong hypothesis of identifying andtracking all targets proved not to be attainable eventhough sensor coverage was nominally complete, pointingto integration requirements for an effective ISRarchitecture.

5. An Integrated Analysis and ExperimentationCampaign: Army 21 / Restructuring the Army 1995-99:This campaign demonstrated the importance of detailedproblem definition and an iterative approach based onwargaming, field trials and analytical studies. Thewarfighting concept under test was found to fail underrealistic environmental constraints. However, the resultsled to an alternative concept, which is the basis for currentAustralian Army force development.


6. The Peregrine Series: a Campaign Approach to Doctrineand TTP Development: This on-going campaign ofexperiments and studies is contributing directly to thedevelopment of the doctrine for employment of theAustralian Army’s new Armed Reconnaissance Helicoptersand demonstrates how experimentation can be used toinform capability development questions at unit level and below.

7. Multinational Experiment Three (MNE 3): Despite thecomplexity of the MNE 3 effects-based planning (EBP)experiment and the findings that the concept andsupporting tools require further development, the eventdemonstrated the potential for EBP to make a coalitiontask force a more effective instrument of power. It alsoshowed the benefits for collaboration to produce the bestideas from a collective thought process in a coalition,which included a civilian interagency component.

8. Improved Instruments Increase Campaign Values:While improved experimentation instruments provided theopportunity to generalize some results, they alsoincreased the validity of campaign’s results andknowledge generation synthesized for future informationmanagement systems.

Epilogue

The thesis of GUIDEx is that, while it is true that defenseexperiments are not like some highly abstracted andinanimate laboratory experiments, the logic of science andexperimentation can be applied to defense experiments toproduce credible tests of causal claims for developingeffective defense capabilities. An overview of that thesishas been presented in this pocketbook version of GUIDEx.

This guide presents experimentation practices andexamples resulting from the deliberation of the AG-12participants, who have all had experience in their owncountries’ defense experimentation efforts. The reader isencouraged to apply and adapt the 14 Principles laid outin GUIDEx to improve experimentation across the TTCPnations, although they do not express national positions.Many examples within the guide are based on the specificperspective and experience of different lead-nationauthors with contributions from other participants: theymay require supplementary effort to relate them tonational perspectives. It is anticipated that as GUIDEx isused, practitioners will develop additional good practicesand examples, and this will stimulate an update to GUIDExin the future.


74

EBP effects-based planning

GCCS Global Command and Control System

GIG Global Information Grid

GUIDEx TTCP Guide for Understanding and InterpretingDefense Experimentation

HITL human-in-the-loop

HQ headquarters

HUM TTCP Human Resources and Performance Group

HW/SW hardware/software

IRM information requirements management

ISR intelligence, surveillance and reconnaissance

ISTAR intelligence, surveillance, target acquisition andreconnaissance

JFCOM Joint Forces Command

JSF Joint Strike Fighter

JTF Joint Task Force

MAR TTCP Maritime Systems Group

MBM model-based-measures

M-E-M model-exercise-model

MNE Multinational Experiment

MoD Ministry of Defence (UK)

MoE measure of effectiveness

MoM measure of merit

MoP measure of performance

MSEL master scenario event list

77

Acronyms, Initialisms and Abbreviations

AAR after-action review or report

ABCA American, British, Canadian, Australian Armies

ACT Allied Command Transformation (NATO)

AG Action Group

AU Australia

C2 command and control

C4I command, control, communications, computersand intelligence

CA Canada

CCRP Command and Control Research Program

CD&E or CDE concept development and experimentation

CFEC Canadian Forces Experimentation Centre

COBP code of best practice

COP common operational picture

CPX command post exercise

CS Case Study (With capitals for GUIDEx CSs, casestudy otherwise)

DISA Defense Information Systems Agency

DoD Department of Defense

DRDC Defence Research and Development Canada

Dstl Defence science and technology laboratory

DSTO Defence Science and Technology Organisation


79

NAMRAD Non-Atomic Military Research and Development

NATO North Atlantic Treaty Organisation

NCO network centric operations

NCW network centric warfare

NEC network enabled capability

NITEworks Network Integration Test and Experimentationworks

NL National Leader

OT&E operational test and evaluation

OTH-T over-the-horizon targeting

PLIX Pacific Littoral ISR Experiment

TP Technical Panel

TRADOC US Army Training and Doctrine

TTCP The Technical Cooperation Program

TTPs tactics, techniques and procedures

TUAV tactical unmanned air vehicle

UAV unmanned air vehicle

UK United Kingdom

US/USA United States of America

USV uninhabited surface vehicle


DoDAF Working Group. 2004. “DoD ArchitectureFramework, Version 1.0.” p. 87.

Feynman, R.P. 1999. The Meaning of It All: Thoughts of aCitizen Scientist. USA: Perseus Books Group. 133 p.

Kass, R.A. 1997. “Design of Valid Operational Tests.”International Journal of Test and Evaluation(June/July):51-59.

Radder, H. 2003. The Philosophy of ScientificExperimentation. Pittsburgh, PA: University of PittsburghPress. 311 p.

Shadish, W.R., T.D. Cook, and D.T. Campbell. 2002.Experimental and Quasi-experimental Designs forGeneralized Causal Inference. Boston: Houghton Mifflin.623 p.

Thomke, S.H. 2003. Experimentation Matters; Unlockingthe Potential of New Technologies for Innovation. Boston:Harvard Business School Press. 307 p.

US Joint Staff. 2000. “Joint Vision 2020.” edited by USDepartment of Defense: US Government Printing Office.

Zachman, J.A. 1987. “A Framework for InformationArchitecture.” IBM System Journal 26(3):276-292.

81

References

ABCA. 2004. “American, British, Canadian, and AustralianArmies' Standardization Program Analysis Handbook(draft for unlimited distribution).” 66 p.http://abca.hqda.pentagon.mil/

Alberts, D.S. and R.E. Hayes. 2002. Code of Best Practicefor Experimentation. Washington, DC: CCRP. 436 p.

—. 2005. Campaigns of Experimentation; Pathways toInnovation and Transformation. Washington, DC: CCRP.227 p.

Bowen, C. and K.R. McNaught. 1996. “Mathematics inWarfare: Lanchester Theory.” p. 141-156 in TheLanchester Legacy, Volume 3 - A Celebration of Genius,edited by N. Fletcher. Coventry, UK: Coventry UniversityPress.

Cook, Thomas D. and Donald T. Campbell. 1979. Quasi-Experimentation: Design and Analysis Issues for FieldSettings. Boston: Houghton Mifflin. 405 p.

Dagnelie, Pierre. 2003. Principes d'expérimentation:planification des expériences et analyse de leurs résultats:Electronic edition. 397 p. http://www.dagnelie.be


Eempirical-deductive … 4, 26empirical-inductive … 4, 26environment … 13, 22, 25, 30, 35, 50, 54, 57, 58ethics … 57, 58, 59, 66exercises and OT&E events … 40, 42, 44experiment control … 37, 51experiment design … 12, 15, 27, 36, 42, 46, 51, 54,

65, 70experimentation campaign … 3, 19, 21, 52, 60, 64, 70experiments and science … 4

Ffield experiment … 9, 25, 31, 32, 33, 34, 35five experiment components … 6, 69four experiment requirements … 10, 69, 70

ability to detect a change … 12ability to isolate the reason for change … 14ability to relate the results to actual operations … 15ability to use the new capability … 11

83

Index

Aanalytic wargame … 3, 28, 29, 32, 34, 35Aristotle … 4

BBacon, Francis … 4

Ccapability development and prototyping … 15, 16, 31case study … 71cause-and-effect … 6, 7, 55communication … 25, 36, 37, 59communication plan … 60confounding … 14, 39constructive simulation … 3, 25, 28, 30, 32, 33, 34Copernicus … 4

Ddata analysis and collection plan … 36, 38, 55data collection mechanisms … 55defense experiment … 2, 6, 9, 19, 28, 36, 40, 48, 53,

57, 69


Rrational-deductive … 4, 26

Sscenarios … 3, 15, 17, 29, 34, 46, 47, 58security … 36, 37, 51, 58Shadish … 2, 5, 8, 10stakeholder … 36, 52, 59, 60, 61, 62, 63, 65subjective measures … 39

Tthreats to a good experiment … 68

Vvariability … 12, 13, 30, 31, 36, 37, 38, 39, 55visitor day … 62

WWarfighting Experimentation … 69wargaming … 25, 72written report … 62

85

Hhuman element … 37human-in-the-loop … 3, 30, 32, 34, 35hypotheses … 7, 11, 25, 50, 52, 71

Iintegrated analysis and experimentation campaign … 3,

19, 21, 52, 60

LLanchester Legacy … 49learning effects … 39, 51, 69live simulation … 25, 31

Mmodel-exercise-model … 34, 44, 51modeling and simulation … 37, 48Multinational Experiment … 73

PPeregrine Series … 73Persian Gulf … 71problem formulation … 19, 23, 24, 65Ptolemy … 4


An electronic copy of this report is available at thefollowing URL: http://www.dtic.mil/ttcp

National reviewers: AU, Dr Paul Gaertner; CA, ChrisMcMillan, UK, George Pickburn; and US, Dr Paul Hiniker.

Copy editor: France Crochetière

87

Acknowledgements

The preparation of this document would not have beenpossible without selected collaborative activitiesconducted under the TTCP umbrella that includedmeetings, conferences and workshops with participationfrom JSA, HUM and MAR (group, technical panel andaction group members), interactions with ABCA, NATO RTOand ACT (Allied Command Transformation), and the directand indirect contributions by participating country experts.

The participants of TTCP JSA AG-12 with the collaborationof several experts produced this document. They are listedbelow by alphabetical order of family name.


Bowley, Dean Defence Science & TechnologyOrganization (DSTO)

AU

Comeau, Paul Canadian Forces ExperimentationCenter (CFEC)

CA

Edwards, Dr Roland, NL11

Defence Science and TechnologyLaboratory (Dstl)

UK

Hiniker,Dr Paul J.

Defense Information SystemsAgency (DISA)

US

11 UK National Leader until June 2004, then Dr Geoff Howes joined AG-12 as UK NL.

Howes, Dr Geoff,NL

Defence Science and TechnologyLaboratory (Dstl)

UK

Kass, Dr RichardA., NL

Joint Forces Command (JFCOM),Experimentation

US

Labbé, Paul,Chair

Defence Research & DevelopmentCanada

CA

Morris, Chris NITEworks UK

Nunes-Vaz, DrRick

Defence Science & TechnologyOrganization (DSTO)

AU

Vaughan, Dr Jon,NL

Defence Science & TechnologyOrganization (DSTO)

AU

Villeneuve,Sophie

Canadian Forces ExperimentationCenter (CFEC)

CA

Wahl, Mike Joint Forces Command (JFCOM),Experimentation

US

Wheaton, DrKendall, NL

Canadian Forces ExperimentationCenter (CFEC)

CA

Wilmer, Col Mike US Army Training and Doctrine(TRADOC)

US

TTCP Document Feedback

The aim of TTCP is to foster cooperation within the scienceand technology areas needed for conventional (i.e., non-atomic) national defense. The purpose is to enhancenational defense and reduce costs. To do this, it provides aformal framework that scientists and technologists canuse to share information among one another in a quickand easy fashion. Its structure is illustrated below:

More information on TTCP can be found on its publicWebsite at http://www.dtic.mil/ttcp/

For the purpose of maintaining and updating TTCPunlimited distribution documents (publications that, due totheir value to the academic, scientific and technologicalcommunities, are widely distributed) readers and users ofthese documents are invited to email their appreciation,comments and suggestions for future editions [email protected]

This address is administered by the TTCP WashingtonStaff, who will pass feedback onto the appropriatedocument point of contact. For more information on TTCPdocument feedback, please see the TTCP guidancedocument ‘POPNAMRAD’, which can be found on thepublic website.


88

National Science and Technology Communities and Programs

Technical Panel• Chair• National Leaders• Team Members

Action Group• Chair• National Leaders• Team Members

Project• Project Leader• Project Officers• Team Members

Group• Executive Chair• National Representatives

TTCP PrincipalsWashington Deputies

Washington Secretariat

LEVEL 1

LEVEL 2

LEVEL 3


Notes

GUIDEx is that robust experimentation methods from … and Analysis Group (JSA) to establish Action...

Documents

Transcript of GUIDEx is that robust experimentation methods from … and Analysis Group (JSA) to establish Action...