Toward Formative Measures for Instructors in Outcomes Based Training & Education

8/12/2019 Toward Formative Measures for Instructors in Outcomes Based Training & Education

1/39

Marceau, R., Jackson, C., Dean, C., Diedrich, F., Artis, S., Wiese, E., & Riccio, G. (2010). Formative Measures for Instructors. In: Riccio, G.,

Diedrich, F., & Cortes, M. (Eds.).An Initiative in Outcomes-Based Training and Education: Implications for an Integrated Approach to

Values-Based Requirements (Chapter 2). Fort Meade, MD: U.S. Army Asymmetric Warfare Group. [Cover a rt by Wordle.net represents word

requency in text.]


2/39

Development of Instructor Measures 20

Asymmetric Warfare Group

Chapter 2. Formative Measures for Instructors

Ryan Marceau, Cullen Jackson, Courtney Dean, Fred Diedrich, Sharnnia Artis, Emily Wiese

Aptima, Inc.

Gary E. Riccio

The Wexford Group International

2.1 Development of Formative Measures

2.1.1 The COMPASS Methodology

In order to assess training effectiveness and provide developmental feedback, it is important to

assess instructor performance in relation to the knowledge, skills, and attitudes that are being

developed. A bottom-up process for developing performance measures grounds the assessment in

behavior illustrated by experts and individuals who have attained proficiency in a particular job(MacMillan, Garrity, and Wiese, 2005). This helps ensure that behavior-based measures are

meaningful and pertain to constructs for which data can be collected reliably.

Our methodology for measure development combines subject matter expertise with established

psychometric practices to produce measures of observable behavior. This methodology, originally

developed by Aptima with the Air Force Research Laboratory, is referred to as COmpetency-based Measures for Performance ASsessment Systems (COMPASS

SM) (MacMillan, Entin,

Morley, & Bennett, in press). The COMPASS methodology was used to operationalize the

principles and practices of OBTE into observable behavior that can be measured in evaluation of

a training program and that can provide formative feedback to instructors.

The COMPASS methodology employs an iterative series of three workshops with subject-matter

experts to develop and validate observation-based performance measures (for complete details,

see MacMillan et al., in press). In the first workshop, working as a group with a facilitator,subject-matter experts identify behavior upon which the performance measures are based. These

performance indicators refer to observable behavior that allows an individual to rate the quality ofindividual or team performance. For this workshop, the objective is to identify behavior that can

be observed rather than inferred. The performance indicators allow identification, at a high level,

of behavior for which it is most important to develop specific measures. In addition, a workflow

or a series of tasks typically is addressed to provide context for the performance indicators and

associated measures. This context helps identify behavior that is diagnostic or critical to qualityperformance.

While a few performance indicators are readily translated into performance measures, more

detailed information is generally needed to create behaviorally anchored performance measures

that coincide with the performance indicators from the first workshop. For the anchors, it is usefulto consider specific behavior that is related to effective and ineffective performance for each

performance indicator. Therefore, the second COMPASS Workshop consists of a series of one-

on-one interviews with subject matter experts to identify explicit behavior that illustrates

superior, average, or poor performance for each of the performance indicators. Conducting this

workshop with individual experts allows for documentation of multiple opinions and, hence,

constant comparison and a more thorough examination of the various assumptions and

considerations for the behaviorally anchored measures.


3/39

21 Marceau et al.


The goal of the third workshop is a detailed review and modification of a set of draft performance

measures. The development of an observer-based rating instrument involves concerns regarding

observability (i.e., Will there be an opportunity to observe this behavior?), rating scale (i.e., How

much variability in behavior will be observed?), and wording of the behavioral anchors to reduce

ambiguity and promote inter-rater agreement. In order for performance measures to be

informative, the third workshop also includes a review of relevance and a confirmation that the

performance measures capture the behavior described in the performance indicators derived infirst workshop. Based on the results of the third workshop, the set of performance measures is

further refined, subsequently reviewed, and tested for feasibility of data collection.

2.1.2 Development of Measures for OBTE

The three COMPASS workshops were conducted to develop performance measures for assessing

the extent to which OBTE principles and practices are represented in instructor behavior.

Accordingly, the first COMPASS workshop was conducted in a group setting with the

progenitors of OBTE from the Asymmetric Warfare Group (AWG). These subject matter experts

developed the Combat Applications Training Course (CATC) that the AWG offers tostakeholders in Army training and education who are interested in OBTE. This train-the-trainer

course presents OBTE in the context of marksmanship training. CATC was used as a point ofdeparture in the first workshop for most of the discussion about specific behavior and broader

instructional strategies that exemplify OBTE. The performance indicators developed in this

workshop were categories of behavior that reflect OBTE.

The second COMPASS workshop consisted of individual interviews with a selection of the

subject matter experts who were instructors for CATC. The interviews focused on identifying

examples of instructor behavior that would illustrate the novice, average, and expert level of

performance for each performance indicator. The topic of conversation focused on what

instructors should do to be consistent with OBTE. It addressed behavior indicative of effectiveperformance and behavior indicative of ineffective performance. The information gathered in this

workshop informed the creation of three behavioral anchors to develop a given performance

indicator into a behaviorally anchored measures for instructor performance. Anchors weredeveloped for most of the performance indicatorsall of the ones for which a rating scale would

be appropriate.

The third and final COMPASS workshop presented the participants from workshop with the final

list of behaviorally anchored measures as they relate to each performance indicator. Performance

measures were reviewed to ensure that the items were seen as meaningful, relevant, and

observable. Subject matter experts reviewed the measures for the performance indicators withrespect to the following criteria: relevance, observability, measurement type (e.g., scale, yes/no,

checkboxes), measure wording, scale type, and scale wording. Consequently, the third workshop

resulted in a complete set of observation-based performance measures for assessing instructors

with respect to OBTE.

2.2 Description of Formative Measures

2.2.1 Results of the COMPASS Process

The product of the COMPASS effort was a set of 65 measures that reflect the principles andpractices of OBTE (see Appendix A for the complete set). Many of the measures also possess

behaviorally anchored rating scales that describe observable instructor behavior at the novice,

average, and expert levels. These measures operationalize OBTE in terms of a formative


4/39



assessment, that is, assessments that are actionable in the continuing improvement of training and

education. The COMPASS process yielded a set of measures that have meaning within a

community of practice. Disciplined use of the measures provides the opportunity for the user to

become increasingly oriented to the values and best practices of the community. This provides

additional meaning and validity to the endeavors of anyone who uses the measures for formative

feedback. It thus is an important source of confidence for instructors. Given that the measures

refer to observable behavior, they also tend to ground confidence in a sense of competence; thatis, sources of confidence are verifiable by oneself and others.

These formative measures for instructors are intended to be general. For some instructor

measures, a particular focus such as rifle marksmanship training helped reduced ambiguity. In

such cases, it should be relatively straightforward to translate the particulars into those of another

skill or knowledge domain. We believe the measures can be employed to assess instructor

behavior in any domain to provide feedback with respect to the principles and practices of OBTE.

Many of measures developed for planning and execution of instruction are described below. They

are grouped under high-level headings that were instrumental in the development of the measures.These headings were intermediate products of the initial COMPASS workshop. They resulted

from initial discussions for particular performance measures that eventually became focused andrefined to a level of detail sufficient for identification of specific behavior that exemplifies

OBTE. Figure 1 shows an example of a behaviorally anchored performance measure (Does the

instructor articulate the why to the Soldiers?) in the context of its nested superordinate

headings (in gray). The twelve highest-level headings are used below to group and describe themeasures. In subsequent development of a grounded theory for OBTE, the measures have been

regrouped into practical categories that facilitate their selection and use with respect to broad

outcomes around which OBTE is organized (see Chapter 3 and Epilogue).

4 Communicate the parameters of learning ("why are we out here" / "what, why, and how")

4.1 Communicate the "right" problem (i.e. what is the real problem they're trying to solve)

4.1.1 Combat or mission applications vs. meeting the minimum standard

1. Does the instructor articulate the why to the Soldiers?

Relies only on tasks,

conditions and

standards; focus is on

completing the event

Explains the why but not

in the context of mission

success/problem solving;

states solution in the

context of the problem

Lays the foundation of

why at the beginning of

training; states the

problem, then guides the

Soldiers to discovery of

the tactical relevance

c N/A Comments/Notes:

c N/O

Figure 1. Sample Performance Measure. N/A refers to not applicable and N/O refers to not

observed.


5/39

23 Marceau et al.


2.2.2 Elaboration on the Description of Measures

In the description of measures below, we utilized two techniques that are not part of the

COMPASS methodology but that help in utilizing the products of COMPASS. In particular, the

description of each group of measures includes an instructional vignette that characterizes the

thinking and purposeful behavior of notional instructors in planning and executing a learning

event. The vignettes are based on first-hand observations of instructor behavior in CATC and ourdiscussions with the instructors. Thus they generally refer to marksmanship training. It is

important to reemphasize, however, that the measures are not limited to marksmanship training.

They can and should be adapted for other skills and knowledge sets (see e.g., Chapters 12 and

13). The notions about the thinking and purpose of instructors were influenced by discussions

with CATC instructors and the progenitors of OBTE (see Chapter 3).

For each group of measures described below, we elaborate on the practical and scientific

implications of OBTE based on our interactions with subject matter experts in OBTE and given

our theoretical dispositions and experience as scientists and educators. The descriptions reflect

our constant comparison between themes from our discussions with experts over many months

and well-established lines of thought in relevant scientific disciplines (Camic, Rhodes, &

Yardley, 2003; Denzin & Lincoln, 2003). The citations make the comparisons explicit; they pointfrom our representations of OBTE to the relevant literature as opposed to being representations of

the literature. The citations represent interrelated bodies of workmost of which evolved over

many decadesthat can provide anchors and verifiable foundations for a grounded theory of

OBTE. In Chapters 4 and 5, we took greater license in exploring the reciprocal relationships

between our inquiry into OBTE and the relevant literature. Together, the connections described in

Chapters 2 through 5 reveal opportunities for continuing improvement of the measures and the

associated practices of OBTE. This is important given the raison dtre for the formative

measures and the empirical investigations that utilize them (see Chapter 1).

As in our development of a grounded theory for OBTE, identifying implications and personal

meaning in the measures is an important part of how they should be used. To be consistent with

OBTE, particular measures should not be imposed on the user, and no particular measure shouldbe considered mandatory. OBTE is not prescriptive; the measures and associated vignettes are not

a script. The user should take ownership of the continuous quality improvement that the measures

enable and adapt them as appropriate for the situation at hand.

2.3 OBTE Performance Measures: Planning for Training

2.3.1 Define Outcomes

Defining training outcomes is the first and most critical step in building any program of

instruction. OBTE emphasizes that instructional system design should reflect and respond to the

most urgent needs of Full Spectrum Operations. OBTE focuses on developing Soldiers over time

by exposing them to experiences designed with respect to attributes such as confidence,accountability, and initiative as well as associated capabilities such as awareness, discipline,

judgment, and deliberate thought. OBTE does not emphasize meeting standards alone; it does not

dismiss them either. Rather, it seeks to shape instruction to promote continuing development of

Soldiers who are agile overall and proficient in particular skills under unpredictable and changingconditions (Cornell-dEchert, 2009b; see also, Chapter 1).

An outcome is different from a learning objective but ultimately should be integrated with

learning objectives associated with a near-term sequence of learning events (see Cornell-dEchert,


6/39



2009a). Outcomes can be viewed as a broader purpose for training or education. They should

relate to execution of military missions and development of individuals to ensure mission success

(Cornell-dEchert, 2009b). As design goals for learning objectives, longer-term outcomes can

provide a framework for linking of learning events in curriculum level design. If a learning

objective specifies the end state of a learning event, outcomes provide guidance and constraints

on the means to the end. An important implication is that, in principle, learning objectives and

developmental outcomes can be convergent or divergent. Without explicit consideration ofdevelopment outcomes that are always being influenced, one way or another in a learning event, a

narrow focus on efficiency and near-term objectives can lead to an unacceptable level of risk of

divergence and intertemporal interference between objectives and outcomes (cf., B. Glaser,

2002).

Designers who understand the developmental role they have with cadre will

refrain from narrowly defining every aspect of training or education and instead

focus on principles. The science of the Designers craft must support the art of

the instructors craft; it must be vigilant in never suffocating the art of

instruction. Designers who desire to create a learning environment for studentswill first seek to create a learning environment for cadre. Designers who

recognize the importance of an adaptive and thinking Army will first seek tocreate an outline of instruction that fosters adaptation and thinking within the

cadre. [M. Darwin, personal communication, November, 2008]

Example Measures

1. Is the training designed to emphasize the importance of combat applications?

Training focuses on

tasks/events; goal is to

pass training (e.g.

qualify)

Combat applications are

described, but training

focuses on tasks/event

Tasks/Events resemble

combat application and

mission success

2. Do the instructors incorporate development of intangible attributes (Judgment,Adaptability, Accountability, Problem Solving, Confidence, Initiative, Awareness,

Thinking Skills) into their vision for achieving the Commanders intent?

Vision is focused on

apparently efficient

and correct proceduralaccomplishment of the

Vision focuses on correct

Soldier performance of

the task but notdevelopment of the

Vision focuses on

effective development

of the individual andcorrect Soldier


7/39

25 Marceau et al.


event/TSP individual performance of the task

Example Vignette:

An instructors training objective for a certain day is to practice grouping. An expert instructor

might try to schedule a 25-meter range for this day so that the Soldiers can easily walk downrange, see their groups and attempt to self-diagnose before a Drill instructor helps them. An

immediate result is Soldiers who have a better appreciation of factors affecting their shot groups.

More importantly, progress is made toward longer-term developmental outcomes in that Soldierscome to understand the value of various forms of feedback in learning. A novice instructor, with a

focus on throughput, might plan only to walk down range and then make the changes to the rifle

themselves. The outcome for the novice instructor, whether intended or not, are Soldiers who

believe that practice without feedback is a reasonable approach to learning and who have little or

no awareness that bad habits can be developed as easily as good habits. Expert instructorsunderstand that there is no such thing as an event in which no learning occurs, they understand

that learning is either good or bad, they understand that time constraints and priorities such as

throughput can lead to instructional shortcuts that have unintended consequences and

counterproductive outcomes. During planning, expert instructors consider these critical influenceson the quality of instruction.

Outcomes and training events should always focus on learning that is relevant to unpredictable

environments rather than the accomplishment of a scheduled event in more or less expedient

instructional settings. The novice instructional behavior in the vignette above might still result in

the Soldiers meeting the grouping standard but very little Soldier development would occur. Theexpert instructor allows the Soldiers to view and evaluate the effect they have on their shot

groups. This increases understanding and encourages deliberate thought on behalf of the Soldier.

Though the Soldier may not be able to accurately diagnose the problem at first, guidance from the

instructor will help them to discover the solution to their problem and lead to increased Soldier

confidence and accountability.

2.3.2 Create a Positive Learning Environment

In OBTE, instructors utilize and exemplify leadership. Arguably, the terms instructor and

leader are synonymous in this approach to learning and development. Every instructor is afacilitator, advisor, and mentor. This means that some mistakes will be allowed, that students will

learn lessons from mistakes instead of being derogated for them, and the instructors role will be

to guide students toward discovering a solution. Time should be built into the schedule for such

guidance and for students to engage in the self-discovery and collaborative reflection that are

especially important when learning events are not completely scripted and controlled.

Instructors should plan to frequently assess how the training is progressing. This assessment doesnot have to take any time away from training; it can be as simple as lagging behind the Soldiers

as everyone walks up range. During this short time, instructors can assess if they are achieving

their intent for the day and make changes as necessary. This way, the instructor can adapt the

training to the evolving needs of the students without sacrificing progress toward long-term

developmental objectives.


8/39



Example Measures

1. Do the instructors plan to scale down from an authoritarian approach to a mentoring

leadership style as appropriate for training events?

!Yes !No

2. Do the Drills design the training events to gradually increase in difficulty?

Unaware of stress

management; Plans to

increase difficulty of

training according to a

set schedule/process

Plans to introduce stress,

but not at the appropriate

level or time

Balances stress and

difficulty to the capability

of the Soldiers

Example Vignette:

Drill instructors are planning out how they will have the Soldiers practice shooting from different

positions. Due to the high Soldier to instructor ratio, there will not be enough instructors to ensure

that all the Soldiers are building proper positions. A novice instructor would likely plan to first

demonstrate the positions to all the Soldiers and then spread instructors evenly across the lanes

and run all the Soldiers through in cycles. The instructors would monitor multiple lanes to ensure

safety and provide assistance when necessary. An expert, in this case, might plan to first split the

Soldiers into small groups, one for each instructor. The instructors will facilitate discussions

among their groups on the how to properly build a stable position, guiding the Soldier towards anunderstanding of the fundamentals of a stable position.

The experts decision to break the Soldiers into groups will allow instructors to facilitate the

individual development of the Soldiers. By ensuring that each Soldier becomes acquainted with

the fundamentals of how postural configuration and stability relate to marksmanship, they are

increasing the likelihood that Soldier will be successful in shooting not only from the particular

positions experienced but from other positions as well. As a student of combat-relevant

marksmanship, the Soldier will develop perceptual and motor skills of stabilization in addition to

appreciating the implications of stability (cf., Riccio & Stoffregen, 1988;Riccio, 1993). This will

increase a Soldiers confidence in situations where factors of enemy or terrain, for example,require nonoptimal or unusual postures. Though the initial small group exercises may delay the

start of the shooting portion of the training, the increased understanding and proficiency will

increase the rate of learning in novel or more difficult situations.

The balance between task difficulty and student capability is critical to student development.

Increasing difficulty or applying stress too quickly will lead to failure and be detrimental to the

students confidence. Going too slowly will result in inefficient training and students becoming

disengaged. A proper balance will result in efficient and effective training that also builds


9/39

27 Marceau et al.


students confidence as they learn to learn and perform under nonoptimal conditions. Instructors

should employ a building block approach that increases complexity, for example, as students

learn and adapt new skills (see, e.g., Merrienboer & Kirshner, 2007;Tobias & Duffy, 2009). Such

scaffolding can be planned in terms of starting simpler as well as ending more complex.

2.3.3 Create the Parameters of Learning

It cannot be assumed that the students will understand why a training exercise is relevant to their

mission. It is important to design exercises that aid the student in making these connections.

Understanding the why behind training will contextualize students learning and presumably

will lead to better transfer to the operational environment. In OBTE, instructors plan for

opportunities to provide explanations or otherwise to reveal the broader context of a learning

event.

While large blocks of unused time (e.g., hours or days) are generally nonexistent in a training

program, there is a multitude of brief moments (e.g., seconds or minutes) that are wasted in

almost any instructional situation. Much can be communicated during the frequent interstitialmoments between telling and doing. Value can be added in both the content of communication

(e.g., explanation) and the occurrence of communication (e.g., modeling). Instructors shouldprepare for such opportunities by having teaching points that can be expressed concisely at

moments that, while not precisely predictable, are reasonably likely to occur. There are

opportunities for education during training.

The inextricable linkage between training and education, in practice, calls for different

approaches in design and development. While training produces a change in behavior, education

produces a change in thinking. Training Soldiers to be agile requires consideration of cognitive

skills even when the focus of a learning event is on fundamental perceptual and motor skills. In

fact, cognition is separable from perception and action only under the most artificial andcontrived situations (cf., Neisser, 1976; Shaw & Bransford, 1977; Winograd, Fivush, & Hirst,

1999). The linkages can be so basic that little or no explanation may be necessary for students to

come to a better understanding of them. The right kind of experience may be sufficient.

Example Measures

1. Do the instructors plan to discuss the tactical relevance of the task with the Soldiers?!Yes ! No

2. [Follow-up] As part of the discussion, will the instructors ask the Soldiers to describe thetactical relevance of the events?

Only plans to lecture to

Soldiers on the task; nocontext

Plans to engage the Soldiers

on why the event istactically relevant, but



Plans to state the problem,

then guide the Soldiers todiscovery of the tactical

relevance (i.e. problem

solving exercise)


10/39



Example Vignette:

Soldiers in Army basic training sometimes use sandbags to support their rifles in the prone-

supported position. However, it is unlikely that there will be sandbags laid out for them in theater.The skill that is being taught is how to establish a solid, supported position, not how to shoot

using a sandbag. Expert instructors would plan to ask the students to think about or discuss waysthey could create a stable prone-supported position in theater. What would they use for support?

How would this change their position? Why is this important in combat? This discussion could

happen during the down time in between shooting cycles while Soldiers are rotating or waitingtheir turn to shoot.

Creating the parameters of learning helps explain the why the experience and skills acquired in a

learning event relate to a mission application of a current task. To the extent possible, the learning

environment should have operational relevance and realism. Situation-specific resource and

safety constraints often make it difficult for the training environment to look like the field or

operational environment. The tendency in Army training and education is to think that, absent

obvious realism, operational relevance can be addressed through vignettes or other supplementary

materials that tell the student about a mission and set a conceptual context for instruction. To be

sure, such a conceptual context can be useful to the student but, by itself, it has many of the

limitations of telling without doing. The connections are not likely to be understood deeply or

retained by the student.

A learning environment need not appear superficially like an operational environment to have

operational relevance or, more specifically, to develop skills that transfer to an operational

environment (cf., Warren & Riccio, 1985). A learning environment can have fidelity with respect

to perception and action if it replicates constraints on what an individual can achieve and how.

Generally this requires a context that is sufficiently rich to include observable cause-effect

relationships such as linkage among tasks and concurrent or downstream consequences of ones

actions (cf., J. Gibson, 1977, 1979). In the context of marksmanship, for example, the relationshipbetween constraints imposed by ones momentary biomechanics, the environment, and

perceptual-motor tasks can be manipulated and revealed in operationally relevant ways without

the physical realities of combat. Training can be designed to give individuals experience,

sometimes in unusual settings, that sensitizes them to the interrelationships between postural

control, stabilizing and destabilizing elements of the environment, and visibility in general (cf.,

Riccio, McDonald, & Bloomberg, 1999). The broader implication is that basic research in the

behavioral sciences can inform the design of a training environment (cf., Gagne & Gibson, 1945;

Riccio, 1995; Riccio & McDonald, 1998).

2.4 OBTE Performance Indicators: Training Execution

2.4.1 Communicate the Parameters of Learning

As indicated in section 2.3.3, OBTE instruction should seek to lay the foundation of the why at

the beginning of training. Students should be exposed to the problem and guided to discovering

the tactical relevance of a particular training activity. OBTE instruction emphasizes how a

particular activity applies to the overall mission or desired outcome.


11/39

29 Marceau et al.


Example Measures

1. Does the instructor articulate the why to the Soldiers?

Relies only on tasks

conditions and

standards; focus is on

completing the event

Explains the why but not

in the context of mission

success/problem solving;



Lays the foundation of

why at the beginning of

training; states the

problem, then guides the

Soldiers to discovery of

the tactical relevance

Example Vignette:

Soldiers typically arrive at the range early in the morning following chow. They get intoformation and wait for instruction from the Drill instructor. Often, the Soldiers do not know or

have very little information regarding training content for that day until they are told by a Drill

instructor or the tower to complete a task. Even at this point, they still do not know the reason or

combat relevance of the task, just that they have to complete it. An expert in OBTE would start

the day by bringing everyone in and discussing the goals for that day. The expert does not simply

recite passages from the Training Support Package for the day but presents the goals in the

context of a combat related problem that the Soldiers must overcome in order to achieve mission

success. The instructor takes opportunities to reveal, not simply to lecture about, the critical

interrelationships among barricades, defilade postures, muzzle awareness, location of other unit

members, and characteristics of enemies and noncombatants in terms of the capabilities and

requirements for postural stability and visibility of targets in a task that otherwise is ostensibly

only about marksmanship.

The vignette above describes one way in which an instructor can increase learning by providing

context around a training task. The impact of training presumably is increased to the extent that

Soldiers are sensitized to contextual relationships not limited to the task at hand (cf., J. Gibson &

E. Gibson, 1955; Bransford & Schwartz, 1999). In this case, the context is not just the learning

objectives for the event, but also their relationship to lethality and survivability in combat and

perhaps even to strategic implications of small unit action in the context of FSO. By discussing

why particular activities and props are included in a learning event, the instructor can educate the

attention (cf., J. Gibson, 1979) of the student and help reduce workload due to an unnecessarilyconfusing or inexplicably cluttered setting. Through this deeper understanding, Soldiers become

better equipped to apply the same concepts towards solving problems in different situations theymay encounter in the operational environment.

2.4.2 Training Emphasizes Broad Combat or Mission Success

Training should emphasize problem solving irrespective of environment or resources. Many

training environments do not accurately reflect all aspects of the operational environment. This

need not be the case for operational relevance, and it may be desirable for learning (see section

2.3.3.2). In these situations it is important that the instructor ensure the students are making


12/39



connections in the training task that are similar to the connections on which mission success

depends in an operational environment.

In the case of OBTE, effectiveness wont truly be known until the Soldier performs in combat,

but evidence of learning and development of efficacy (the foundation or capacity for

effectiveness) will manifest itself in ways that allow meaningful assessment of both the Soldier in

training and the trainer in execution. It also provides for explicit or implicit self-assessment andthe development of a sense of self-efficacy (cf., Bandura, 1977, 1997; see also, Ryan & Deci,

1985) in both trainers and their Soldiers (I know when I am performing well and I know that I

can perform well). This is a sense of self-efficacy that is grounded in demonstrable competence

and an appreciation of what ones actions and capabilities for action in a particular environment

afford for good or ill (cf., J. Gibson, 1977; Shaw, Turvey, & Mace, 1982; Turvey, 1992; see also,

Endsley, 1995; Neisser & Jopling, 1997).

Example Measures

1. Does the training emphasize broad combat/mission success?

Train to a specific task

only

Train to examples or

experience only; reflects

specific environments orresources only

Creates training that

emphasizes problem

solving irrespective ofenvironment or resources

Example Vignette:

A unit who knew they would be deploying to an urban area in Afghanistan trained extensively in

simulated urban environments. They practiced shooting around buildings, cars and other urban

terrain. They were proficient in room clearing tactics and close quarter marksmanship. Upon their

arrival they were sent to an area of operation that consisted of partial urban and wooded terrain.

The Soldiers did not feel confident in their ability to operate in the wooded terrain because they

had trained in urban terrain only. Consequently, they would be hesitant to follow the enemy into

the woods. Subsequently, in the role of an instructor in a simulated urban environment, one of the

Soldiers made sure that his trainees understood general principles of individual movement,

collective maneuvering, and situation awareness in the context of general characteristics of the

environment such as paths, obstacles, barriers, margins, brinks, footing, partial enclosures,

camouflage, concealment, vistas, and vantage points. Some of this was accomplished by exposing

Soldiers to variation in these constraints amid invariant tasks involving individual perception andaction as well as collective coordination. Soldiers also were asked to consider contingencies in

the way various characteristics of the environment could be utilized.

What the Soldiers described in the initial part of the vignette did not realize was that all the

fundamentals upon which the urban tactics were built could be applied to the wooded terrain. In

contrast, being exposed to variation in the environment and in behavior with respect to it can help

them avoid being distracted by the concrete facts of buildings and roads. They can differentiate


13/39

31 Marceau et al.


the essential from the incidental; they can see various aspects of environment in a different way

(cf., J. Gibson & E. Gibson, 1955; E. Gibson, 1991). The implication is that learners can free

themselves from old habits and assumptions carried from different occupations and purposes.

They can come to understand the essential interdependence of task and the way the environment

is perceived (cf., J. Gibson, 1979; Riccio & Stoffregen, 1988). They can become better prepared

for a new occupation and a new purpose in an otherwise familiar environment. Thus they become

more adaptable to novel tasks and conditions. The instructor can facilitate this learning throughjudicious and well-timed guidance (cf., Bransford & Schwartz, 1999; Tobias & Duffy, 2009).

This is not to imply that the Soldier understands these things in an abstract way. The implication

is that he simply knows what to do and is not frozen in his tracks by superficial novelty.

2.4.3 Customize Instruction When Possible Based on Constraints/Conditions

OBTE is learner-centered (cf., Bransford, Brown & Cocking, 2000; see Chapter 3). Instructors

should try to balance training difficulty with the students momentary capability to learn. It is

important to recognize the point of diminishing return (i.e. student is exhausted or burnt out) and

adapt the training accordingly. People learn differently and at varying rates. Instructors shouldcoach individual students or address the individual training predicaments of students who may

have unique needs. This should not be taken to imply that instructors must provide every studentwith individual attention all the time or even provide this to most students most of the time.

Sometimes a little individual attention is all a student needs to get back in the game. Other times,

there might be an individual difference that is unnecessarily hindering a students ability to learn

(e.g. a small Soldier with a large rifle). Even if the instructor cannot always provide guidance toall the students that could benefit from it, it is better to help some students that none of them.

Moreover, the visible act of providing guidance to a student provides a model even for

bystanders. Instructors can be influential as role models for students who are not the object of the

instructors immediate attention (cf., Lave & Wenger, 1991).

In OBTE, it is desirable to provide students with the right amount guidance at the right time. An

implication of this principle is that it is important to know when not to intervene or to instruct

explicitly. Encouraging students to take ownership of their own learning can have valuable effectson student motivation and engagement in learning (cf., Guay, Ratelle, & Chanal, 2008).

Instructors are not passive during these opportunities for student initiative. They are vigilant inmonitoring the situation and ready to show signs of approval when students demonstrate

discipline and awareness in exercising such initiative. Timely approval shows that instructor was

willing to trust the student and not merely inattentive or uncaring. It reinforces accountability.

Example Measures

1. Does the instructor adapt the training to the audience/environment?

Sticks to set schedule;unaware of diminished

learning (e.g. ignores

indicators of

exhaustion)

Reactively balancestraining difficulty to the

capability of the Soldiers

(e.g. reacts after

performance has

diminished significantly

Proactively balancestraining difficulty to the

capability of the Soldiers;

recognizes point of

diminishing return


14/39



2. Do the instructors successfully address individual training predicaments?

Doesnt recognize

individual problem

Recognizes individual

problem, doesnt help the

Soldier find a suitable

solution

Identifies the issue and

helps the Soldier find a

work around (e.g. shorter

weapon for smaller

person)

Example Vignette:

A Soldier is struggling during a grouping exercise; the Soldiers groups continue to stay large

despite direction from the drill instructor on proper breathing, positioning, and trigger squeezing.

A novice instructor may grow frustrated in this situation and insists that the Soldier is notlistening to instructions. An expert, whose focus is on Soldier development, would look deeper to

examine if something else is going on. Upon further inspection the instructor noticed that this

Soldiers earplugs do not fit correctly. The Soldier is wearing them correctly, but they are not

sealing the ear properly causing to Soldier to flinch every time a shot goes off. The instructor

finds a different style of earplug for the Soldier and the groups start to become remarkably

tighter.

In this situation, an OBTE instructor recognized a Soldiers individual difference that was

preventing him from being successful. The instructor was able to then identify a solution to the

students problem (changed earplugs). Doing so then allowed the student to master the

fundamentals and become more confident in his or her ability as a rifleman. Had this goneunnoticed, that student might think that he simply was not skilled enough to shoot a solid shot

group, leading to low confidence as a rifleman. Additionally, the instructor acted as a mentor in

this case by demonstrating effective problem solving skills in addressing the Soldiers individual

training predicament.

2.4.4 Facilitates Learning of Concepts

OBTE promises to develop deep understanding and habits of learning that help Soldiers

recognize and exploit similarities between their prior experiences and novel situations. This

competency generally will result in the perception of what a unique situation affords for actionand in the associated readiness for expeditious action that is efficacious and that reveals

additional information that reduces ambiguity (cf., E. Gibson, 1988; J. Gibson, 1979). Suchdevelopment of perception and action enables Soldiers and their units to be agile and effective in

Full Spectrum Operations.

Instructors structure the problem and coach the student towards the solution. Not only does thisapproach to training increase learning by leading the student to discover the solution, it also

encourages students to take ownership of their own learning. From the perspective of a mentor

and facilitator, the focus of the instructors attention when students exercise initiative is on

whether the students response is adaptive in the sense of being task directed and oriented (i.e.,


15/39

33 Marceau et al.


efficacious in principle), not whether it is momentarily effective. Such efficacy is the difference

between adaptability and mere flexibility. Flexibility is better than inflexibility but not much.

Efficacy in general eventually leads to effectiveness in particular situations but not in every

instance. Mistakes will be made. The role of instructors is to ensure that the consequences of

mistakes are bounded (Cornell-dEchert, 2009b). It is to identify a students zone of proximal

development (see Cole, John-Steiner, Scribner, & Souberman, 1978), the margin between their

momentary range of efficacy and their potential range of efficacy. This is the zone within whichstudents are challenged without being frustrated and within which there is the greatest capacity

for semi-autonomous learning without deleterious consequences.

Example Measures

1. Does the instructor guide Soldiers to self-discovery of how to achieve desired results?

Dictates a specific pathto the solution to a task Structures the problem; butstill dictates a specific path

to the solution

Structures the problemand guides Soldier to

efficient self discovery of

solution path

2. Does the instructor use safety as a training enabler?

Focuses on SOP and isregimented; safety is

disconnected from its

real purpose

Explains safety in thecontext of accomplishing

the training events, but not

as a combat and training

enabler (i.e. safety isrestrictive)

Explains safety as acombat and training

enabler (e.g. weapons

awareness allows for more

independent or complexscenarios)

Example Vignette:

Marksmanship training is heavily structured and regimented in order to maintain safety.

However, Soldiers do not learning to be safe on their own because the tower is always tellingthem exactly what to do. They appear to be afraid of their weapons. A different approach would

be to start off regimented and pull back after Soldiers demonstrate safe behavior. Without

constant direction from the tower, Soldiers can eventually take on the responsibility for being safe

and become more confident in their ability to be safe with a weapon.


16/39



Transferring the responsibility for safety to the student increases the students awareness and

accountability. The instructor demonstrates trust in the Soldiers ability to be safe.

Soldiers also become more confident in their ability to be safe with their weapons in future

situations when a drill instructor is not present. In Army training, safety and instructional

methodology (i.e., nature, extent, and timing of guidance provided to students) typically are

viewed as the responsibilities of different groups of people if not wholly incommensurate

considerations. The juxtaposition of instructional guidance and safety in OBTE is unusual. Aswith most practices in OBTE, this juxtaposition is due to exigencies and realities of the

operational environment, such as the critical interrelationships between lethality and survivability,

especially amid the ambiguities of Full Spectrum Operations. An assumption of OBTE is that,

instead of finding ways to integrate such fundamental capabilities, they should never be separated

in the first place. Accordingly, there should be a persistent coupling between initiative and

accountability from the beginning and at every stage of a Soldiers training, education, and self-

development.

This does mean expanding our own circles of interaction to include, for example,

the range, ammo, and training developers of the Army who rarely see a Soldiertrain, but do influence decisions on how, why, and to what level his training will

be supported by the Army Outcomes are in so many ways influenced by theavailability of inputs - resources, including trainers and leaders - and we should

start capturing the inputs required for training more effectively. [K. McEnery,

personal communication, February, 2009]

2.4.5 Creates a positive learning environment

A positive learning environment is foundational in OBTE. A balance of authoritarian and

collaborative styles of instruction is an important part of creating such an environment. One

manifestation of this balance is to avoid being directive about content (what to think) whilebeing somewhat directive about process (how to think) (HQDA, 2003; cf., Freeman et al.,

2008; Roberto, 2005, 2009). Instructors understand and embrace their role as mentors. They

serve as guides along the Soldiers path of success by providing clear left and right limits, bynoting milestones and decision points, but also by allowing some exploration (cf., Cole, John-

Steiner, Scribner, & Souberman, 1978; Feldbaum, 1965; Filatov & Unbehauen, 2004, E. Gibson,1988; Henle, 1971; Thelen & Smith, 1994; Safonov, 2001; Tschacher & Dauwalder, 2003). They

facilitate the developmental process by which Soldiers take ownership of their own pursuit of

success. One strategy in this role is for instructors to provide opportunities for Soldiers to think

critically and be problem solvers through training events that require assessment, judgment,

decision-making and execution. As Soldiers try to solve the problem at hand, instructors guidethem through directive questioning and discussion. In the end, the Soldiers solve the problem.


17/39

35 Marceau et al.


Example Measures

1. Does the instructor reinforce the importance of problem solving?

Event is scripted;

training is based on

correct performance of

predetermined task

Training provides

opportunities for Soldiers to

be problem solvers but

instructors give the solution

Training provides

opportunities for Soldiers

to be problem solvers;

events require assessment,

judgment, decision

making and execution

2. Does the instructor ask questions and allow/encourage the Soldiers to answer?

Provides answers for

the Soldiers

immediately afterquestion is asked

Encourages Soldiers to

answer questions, but

provides an answer if one isnot immediately offered

Allows time for Soldiers

to think of an answer and

guides them to appropriateresponses

Example Vignette:

Instructors, during ARM, have set up a range with different stations every five lanes. Soldiers

start at one end and make their way to the other end in pairs, performing different tasks and

exercises at each lane. These exercises include target discrimination, malfunction drills and other

complex tasks. The goal for the day is to develop Soldiers to perform under high stress, combat

like situations. One lane includes a wounded Soldier drill in which the pair of Soldiers has to

react to an injured Soldier, get him out of the line of fire and treat the injury accordingly. Upon

reaching this scenario, two Soldiers are struggling to lift a very large Soldier on stretcher. They

try twice to get him off the ground, but he is simply too heavy. These Soldiers, recognizing that

they have to get the wounded Soldier out of the line of fire, decide to each lift one end of the

stretcher and drag the Soldier out of harms way. Upon seeing this, the drill instructor makes surethat the Soldiers see that he has noticed their solution while he lets them carry on. Later he

questions the Soldiers with interest about their decision to deviate from standard procedure.

The vignette describes a situation in which the instructor has a choice to intervene or not. He

might have, for example, yelled at the Soldiers, criticizes them for not carrying stretcher the

correct way, and make them start over and do it again. This would have failed to maintain a

positive learning environment. Moreover, overly focused on completing the task correctly

would have punished the Soldiers for not following the prescribed methods for carrying a

wounded Soldier even though what they did was more combat effective given the situation.

Instead, by letting the Soldiers exercise initiative, the instructor allowed them experience

problem-solving under stress. By engaging in collaborative reflection with them after the event,


18/39



the instructor found out that their goal was to get the wounded Soldier out of harms way. He thus

could acknowledge that they found the most effective way to accomplish that goal given the

constraints of the situation. He would be able to recognize the deliberate thought and sound

judgment of the Soldiers and positively reinforce their behavior.

It is important to understand that, in OBTE, the deliberate thought and judgment required in

mission-relevant problem solving does not necessarily imply cognition that is mentally effortfulor temporally protracted (e.g., on the order of tens of minutes to hours) as we might imagine

based on years of experience with taking tests in learning environments. Deliberate thought is as

much about disciplined awareness, attunement to things that matter in ones surroundings and in

the flow of events, appreciating the consequence of ones behavior for others and vice versa, and

about the attendant purposeful coupling of perception and action. Such awareness can unfold and

inform decisions on time scales as brief as seconds to minutes (cf., Salas & Klein, 2001).

2.4.6 Instructors Utilize Measures of Effectiveness & Self-Evaluation

OBTE is assessment-centered in the sense that assessment is a component of learning not merely

an assessment of learning (cf., Bransford et al., 2000; see Chapter 3). It is important thatinstructors are constantly reflecting on the progression of learning. Instructors ensure that

students are getting the training that they need and decide when to advance to the next event or to

revisit a learning objective. OBTE is focused on the student instead of the schedule. Schedules

should not dictate how the training progresses; the students momentary capabilities and potentialshould. Instructors should make in-stride adjustments to training as necessary. The informational

basis for these decisions should be explicit to students whenever possible and appropriate. The

point is not to justify changes to the students but to reveal that training is principled, that changes

are event-driven, and that there should be constant vigilance about the conditions in light of

assumptions that may be violated and thus necessitate a change. If this is occasionally madeexplicit to students, it will become implicit to them in future decisions of their instructors and

leaders. Student will become more likely to look for conditions that may have motivated a

change; they will become more aware of the situation (cf., Endsley, 1995).

Example Measures

1. Are the instructors discussing the effectiveness of the training?

Instructors

automatically advancedifferent groups

without considering

training progress

Instructors check on

Soldiers progress but donot effectively adjust the

training

Check on Soldiers

progress and discusseswhen to advance to next

event or when to revisit a

learning objective


19/39

37 Marceau et al.


2. Do the instructors use unexpected conditions to provide tactically relevant trainingevents?

Allow the unexpected

conditions to interfere

with training

Make in-stride adjustments

to continue training but do

not leverage unexpected

conditions as a training

opportunity

Make in-stride

adjustments to incorporate

unexpected conditions

into the training design if

appropriate

Example Vignette:

A schedule calls for several complex ARM exercises to occur on a particular day. However, it is

over 100 degrees out today and the Soldiers are getting burnt out fast. Instructors notice that body

positions and movements of the Soldiers are becoming progressively worse, fundamentally, as theday goes on. Concentration wanes and shooting accuracy deteriorates. The instructors decide that

the students are reaching the point of diminishing returns. Thus, they make an in-stride

adjustment with the Soldiers by rolling back the complexity of the tasks to ensure the Soldiers do

not fail.

The vignette describes a situation in which the instructors put the emphasis on learning and

development and not on task completion or following the schedule. Though Soldiers will have to

perform under high stress situations in combat, they first need to adequately master the basics

before they can be expected to perform well under more demanding conditions. Had the

instructor decided to continue the training at the higher complexity level, the Soldiers would have

pressed on, but their confidence would have declined as they continued to get poor results andeventually failed (cf., Merrienboer & Kirshner, 2007). By adjusting the training complexity to the

current ability of the Soldiers to perform, the instructor ensured that the Soldiers confidence in

their ability to perform would remain high. The adjustment would not have been considered if the

instructors were not vigilant about the state and progress of the students.

In OBTE, instructors do not depend unduly on explicit communication with Soldiers to assess

their physical, physiological, cognitive, or emotional states. It can be a useful tool, however, and

one with additional benefits. By occasionally checking with students about their self-perception

and their perception of the situation, instructors reveal to students the importance of situation

assessment in principled approach to training and, more generally, in a principled approach todecision making in the field (cf., Endsley, 1995). Instructional situations can be dominated by

planned events or facts and still allow students to have opportunities to consider possibilities andalternatives when assumptions or expectations are violated. These opportunities can be brief or

small but if they are common, they encourage a certain vigilance and anticipation that

automatically engage students more deeply in the situation. Seizing such opportunity doesnt

require instructors or developers to throw out the lesson plan.


20/39



2.4.7 Uses scenarios to facilitate learning

Scenarios are potentially powerful tools for emphasizing the reasons for a learning event, why the

event is structured the way it is, and what are the essential versus the incidental aspects of the

event with respect to the operational environment. Our use of the term scenario thus is broader

than its connotation in common military usage. In our usage, a scenario is a situation that is

sufficiently rich to reveal the consequences of ones actions beyond the task at hand. If a trainingscenario focuses on superficial similarity to its operational analog, it may not be effective and

may even distract students from the key lessons to be learned. There are several contextual

variables that can be manipulated effectively in a scenario-based learning event and that are not

especially dependent on superficial similarity to an operational environment. They include but are

not limited to stressors, linking of basic Soldier tasks, and linking of individual activities with

collective activities (cf., section 2.4.2 above).

Stress can be an excellent facilitator of training through scenarios. Instructors should incorporate

stress (mental and physical) into training in ways that challenge the student but are proportional

to the student capabilities. Stress should not cause the student to fail, but should be challenging.The goal of stress application is to give the student a sense of accomplishment and confidence in

their ability to perform under adverse conditions. Students should be allowed to discover howstress affects their performance, and how to mitigate it. Stress can be manipulated easily through

time constraints or by task linkage.

Example Measures

1. Do the instructors group tasks into collective behaviors?

Does not group tasksinto collective

behaviors; results in

incorrect performance

of linked tasks

Does not group tasks intocollective behaviors (no

observable negative

consequences)

Groups task in a way thatsimulates the combat

application and reinforces

correct performance of

linked tasks

2. Do the instructors effectively incorporate stress (mental and physical) into training eventsto benefit the development of the Soldier?

No stress resulting in

apathy or too much

stress resulting in

chronic failure

Some stress resulting in

some learning; Soldier

unchallenged or overly

challenged

Stress is proportional to

the task and Soldier

capabilities resulting in a

sense of accomplishment


21/39

39 Marceau et al.


Example Vignette:

During a training exercise, Soldiers are practicing their fundamentals while shooting from the

prone position. The instructors see that the Soldiers are all doing well so they decide to add stressby limiting the amount of time the Soldiers have to shoot their ten rounds to one minute. At first

the Soldiers all shoot off their rounds in the first 20 seconds and their shot groups are poor. Theinstructors point this out and ask the Soldiers to think about how the time stress affected their

performance. Next cycle, the Soldiers take more time and their shot groups improve. After a

couple cycles the instructors then tell the Soldiers they now have to take ten shots from the prone,reload their rifle while communicating that they are reloading, and then switch to the kneeling

position and take ten more shots in two minutes total. After a few cycles of this exercise, the

instructors facilitate a discussion with the Soldiers about managing stress in combat.

In the vignette, several simple tasks are gradually combined to create more complex, linked tasks.

Stress is added by incorporating a time constraint. This gradual ramping of stress keeps the

Soldier out of their comfort zone, but not to the point of failure. At the end of the day, the

Soldiers in the vignette will be performing complex, linked tasks that they would have never

thought they were capable of doing at the beginning of the day. More novice instructors may

incorporate the wrong kinds of stress (e.g., verbal intimidation) or may apply too much of one

type of stressor. Consider, for example, an apparently reasonable stressor of having Soldiers run a

mile in full kit or do several strenuous activities prior to participating in a training exercise.

Though this is a kind of stress that is relevant to the operational environment it is applied in a way

that does not allow the Soldier to overcome the effects of stress gradually. Through the gradual

addition of stress and complexity, the Soldiers continuously build up their confidence and become

ready to tackle more complex tasks.

The vignette also shows how the linking of tasks can be relatively simple and straightforward yet

give Soldiers experience with critical interrelationships among basic Soldier skills such as move,

shoot, and communicate. Supplemented with a bit of a backstory provided by the instructor, eventasks that apparently are purely procedural can be learned in ways that vastly improve a Soldiers

readiness for combat. Consider SPORTS, a procedure for correcting a weapons malfunction.

The mnemonic refers to a process to clear a rifle malfunction by slapping upward on the

magazine, pulling the charging handle back, observing the obstruction, releasing charging handle,

tapping forward assist; and squeezing the trigger. The context in which this procedure would be

performed is not incidental and could potentially have profound implications.

On the range, a weapons malfunction is a benign event. Obviously in a firefight, the situation is

quite different. The speed with which a Soldier can get the weapon back into the fight is a

potentially a life or death matter, and inability to do is not simply a matter of a no-go. Returning

fire is the higher-order task, one that could be accomplished to some degree by others in the unit.

Thus it becomes critically important to communicate to others that the weapons malfunction hasoccurred. This could change the priorities of another Soldier or simply just change the direction

of fire or his firing position. At the same time, without an operable weapon, the requirements and

tradeoffs for seeking cover change. The tradeoffs are peculiar to the momentary situation and therelevant factors of mission, enemy, terrain and weather, troops and support available, time

available, civil considerations. The linkage between move, shoot, communicate, and assessment

of the situation could not be more fundamental and the consequences could not be more

immediate.


22/39



In a firefight, an outward orientation and the attendant situation awareness is critical but it is

complicated by the distractions of the unusual, if not unique, psychological and physiological

responses to a lethal threat (cf., Kolditz, 2007; Lerner & Keltner, 2001; Lerner & Tiedens, 2006;

Lukey & Tepe, 2008). Habits become important at such times. It is not the time to perseverate on

a mnemonic and its associated procedure if it isnt working. Good habits are necessary. Habits

require frequent repetition. It is never too early to give Soldiers the opportunities to develop goodhabits of linking move, shoot, communicate, and situation assessment, and to give them these

opportunities often. A premise of OBTE is that such basic combat skills can be acquired outside

of combat, that they can be acquired in training. These opportunities will be missed if it is

assumed that there are procedures Soldiers can learn in the absence of deliberate thought and

broader awareness.

2.4.8 Instructors exhibit intangible attributes in own actions

OBTE doesnt require that instructors interact with or even be aware of every student all the time.

The instructor is a role model (cf., Bandura, 1977). Instructors should demonstrate behavior andattitudes their students can model because, more likely than not they will model the instructors

behavior (cf., Lave & Wenger, 1991). It thus is important for the instructor to be aware that hisbehavior may be noticed and influential whether or not the behavior was planned or intentional.

One should be aware, for example, that an authoritarian or intimidating style is not necessarily

perceived as flowing from competence. Similarly, evidence that the instructor does not know it

all, or as much as a student knows in some areas, is not necessarily perceived as incompetence.The expertise with which the instructor should most be concerned is as a leader, mentor, and

facilitator. Their impact is similarly profound whether they accept this responsibility or not, and

whether they are prepared for it or not. The assumption of OBTE is not that instructors should

have a greater impact; it is that they can have a better impact.

Even when there are gaps in an instructors knowledge or experience, there is an opportunity to

demonstrate expertise in learning. The instructor can help a student learn to learn. In the end, this

may be the most important influence an instructor has on a student given that adaptability ofSoldiers in the operational environment implies a capacity to learn. The point here is that students

are always learning to learn in addition to learning what they should learn. We can choose toteach students to learn one way or another but we cannot pretend that there is not a choice. In

OBTE, initiative and accountability are underpinnings of confidence. Confidence is grounded in

demonstrable competence, and competence is ensured by the capacity to learn. The capacity to

learn is increased by initiative and taking ownership of ones own learning. Responsible initiative

is grounded in accountability and bounded by it. These intangible attributes are not abstract in theprocess of learning and teaching. They are observable in the behavior of highly motivated and

engaged individuals.

Example Measures

1. Do the instructors effectively exhibit intangible attributes in their own behaviors as theyconduct their training?

!Yes ! No


23/39

41 Marceau et al.


2. Do the instructors demonstrate openness in changing their training progression?

Training will not

deviate from a set

schedule

Instructors determine when

it is time to move on a new

task

Soldiers have input into

the progression of training

Example Vignette:

A Soldier is shooting well from the prone unsupported position. However, a drill instructor sees

that the Soldier is not resting the magazine on the ground and that doing so would create a more

stable position. The drill instructor tells the Soldier to rest the magazine on the ground even

though this is a controversial practice. After putting the magazine on the ground, the Soldierbegins to shoot poorly and miss his targets. The instructor, realizing that the Soldier is more

effective with the magazine off the ground, tells the Soldier that he can shoot whichever way is

more comfortable and more effective for him. The Soldier then switched back to his original

position and shot more accurately. Later the instructor explained why he was willing to have the

Soldier try a different shooting position.

A novice, in the situation above, might have not been as flexible or confident enough to reverse

his prior guidance to the student (cf., Edmonson, 1999). However, the more expert instructor

recognized that placing the magazine on the ground is only one tactic that can be employed to

create a stable position. He was willing to risk his credibility to show the Soldier that there is no

single right solution in most situations. The instructor created several teaching points throughthe guidance he provided and its immediate impact on the Soldiers performance. It created an

opportunity, for example, to talk about the assumptions behind the rule of not resting the

magazine on the ground. This is more than merely interesting trivia; it makes explicit that there

are assumptions associated with any standard procedure and that these assumptions may have

nothing to do with the task at hand (e.g., learning the adaptive skills of marksmanship) and may

not be valid in the contemporary operating environment. Another is that, while there are several

solutions to most problems, the best solution may vary from time to time and from individual to

individual. The instructor also has the responsibility, however, to explain to the Soldier that

neither is the case that anything goes. This provides an opportunity to help the Soldier become

familiar with the notion of left and right limits for most tasks that both allow for and require someautonomy, and that exploration is a reasonable manifestation of such bounded initiative.


24/39



2.4.9 Hotwashes and Mini-AAR

After-Action Reviews and Hotwashes are a persistently high priority in OBTE. These discussions

about lessons learned should be conducted in an open and supportive manner regarding. The

purpose is not to summarize what just happened in the training but to facilitate a discussion

among the student that allows them to evaluate their own behavior and performance in relation to

others. The distinction between self-assessment and peer-assessment evaporates in the AAR (S.Flanagan, personal communication, April 30, 2009). The duality between self and other is

replaced by an extended sense of self (cf., Neisser & Jopling, 1997; Neisser, 1996).

Accountability is developed through a deep and enduring sense of the relatedness within the unit

in the context of shared objectives (cf., Deci & Ryan, 1985, 2008; Lerner & Tetlock, 1999, 2003).

Through these discussions, instructors ensure that students grasp the reasons behind a learning

event and its connection to a mission application. Additionally, this self-critique serves as a

problem solving activity in itself, boosting student confidence as they collaboratively discover

how to perform more effectively.

Example Measures

1. Do the instructors ensure the Soldiers can articulate the consequences of their actions?

Tells Soldiers what

went wrong and why itis important in combat,

but does not discuss

how to mitigate

mistakes next time

Asks Soldiers to explain

what went wrong, why itsimportant in

combat/mission, but does

not discuss how to mitigate

mistakes next time

Guides Soldiers through

explaining what wentwrong, why it is

important in

combat/mission, how they

might improve next time

2. Do the instructors focus the why of training back to the relationship between theindividual and big picture/mission?

Does not go into depth

on why the Soldiers

need the skills thatwere trained (e.g. You

need these skills

Facilitates discussion on the

big picture; less focus on

the individualscontribution

Facilitates discussion on

the big picture; focuses on

the importance ofindividuals (i.e. You can

make a difference)


25/39

43 Marceau et al.


Example Vignette:

After a long day of training in the hot sun, all the Soldiers in the Platoon are exhausted and

hungry. Their drill instructor asks all the Soldiers to sit down on the shaded bleachers for a fewminutes prior to returning to the barracks. He then begins to tell them a story about a time when

his unit was deployed in Iraq. They were spending 36-hour cycles out in the city as part of aneffort to integrate with the populace. On one occasion, the drill instructors unit received sniper

fire from a neighboring building and one Soldier was wounded. One team broke off to go after

the sniper. They followed the sniper into a neighboring area of operations and eventually detainedhim. They decided to secure a building and spend the night there because it was getting late. They

all were all exhausted and did not want to return to their AO in the night without more support.

By the time the Soldiers made it back to the FOB they had spent over 72 hours out in the city on

only 36 hours worth of food and water. The instructor uses this story not only to make

connections with a mission context but also as a point of departure for collaborative reflectionabout the distractions of hardship and its effect on individual and collective decision making.

In the vignette, the instructor leverages the circumstances and effects of a long hot day to make

connections between a mission context and momentary conditions of training that otherwise are

distractions and apparently irrelevant to learning. Without his discussion, the Soldiers would

likely have gone to chow and never thought critically about that days training and its lessons for

performance in theater. Instead, the instructor revealed the operational relevance of their shared

experience in training on that day training. He did this not only by providing a narrative, back

story to which tired and hungry students might not even pay attention. He motivated their

collaborative reflection about the effects of such stressors on individuals and the group. He thus

brought them back into the moment as a learning event, even as they rested.

The AAR serves two purposes. It helps guide the activity of remembering in which the meaning

of prior events is actively reexamined with respect ones current capabilities for knowing and

interacting with the world (Neisser & Hyman, 2000). It also helps participants in the discussiontake ownership of their own learning and teaching (Magolda, 1999). In the context of OBTE and

instructor education, these are tightly interwoven cognitive and social activities (Sidman, Riccio,

Semmens, et al., 2009). The principles and practices of OBTE help focus an AAR on interactions

and interrelationships among tasks and among individuals within a task organized unit (see

Chapter 3). The conversation is more likely to be self-referential, actionable, and empowering

than a conversation about the real or imagined external factors that conspire to impede learning.

Changing the conversation is the easiest way to change the culture of training and education

(Cornell-dEchert, 2009b).

2.5 Uses of the Measures

The COMPASS process yielded a set of measures that have meaning within a community of

practice. Disciplined use of the measures provides the opportunity for the user to becomeincreasingly oriented to the values and best practices of the community. This provides additional

meaning and validity to the endeavors of anyone who uses the measures for formative feedback

(Mislevy, & Riconscente, 2006; Pellegrino, Chudowsky, & R. Glaser, 2001). It thus is animportant source of confidence for instructors. Given that the measures of shared values and best

practices refer to observable behavior, they also tend to ground confidence in a sense of

competence; that is, sources of confidence are verifiable by oneself and others.


26/39



2.5.1 Formative Measures for Instructors

There are three aspects of a good approach to using formative feedback for instructors: (a) it

should relate to the intent of instruction, (b) it should inform actions that can be taken to improve

instruction with respect to the intent, and (c) it should reflect the instructors span of control. The

relationship of the formative measures to the intent of OBTE is explored in this chapter. This

analysis is developed further in Chapter 3 in the context of categories of measures derived fromthe relationship between the measures and the principles of OBTE (see also, Appendix A). For

instructors, the relationship emphasizes that feedback should not be collected willy-nilly. A good

instructor has a plan for what to observe based on the interpretation of intent in terms of the

events of the day and the situation at hand. Measures that support the plan should be given

priority. An experienced instructor may utilize additional measures as needed. In any case, it is as

important to decide what measures not to use, as it is to decide which ones to use, in a specific

learning event. An example of such planning is provided in Appendix A. An implication of

planning for formative assessment is that the measures also can be helpful in planning the

learning event itself. The measures can help an instructor prioritize opportunities and exigencies

of the moment in the context of student learning and development over time. They can helpconnect prior and subsequent learning events, and they can help make connections between

learning objectives of an isolated event with longer-term outcomes for a student.

The very nature of the COMPASS methodology ensures that the resulting measures are

actionable because they are based on actual experience and observable behavior of instructors

that is verifiable in principle and in practice (MacMillan, et al., in press; MacMillan, et al., 2005).Minimal interpretation is required to go from use of the measures to an appreciation of what an

instructor should do differently to improve. In a sense, the measures educate the attention of the

instructor about what is done and what can be done in a learning event with respect to the intent

of OBTE. The COMPASS methodology also tends to isolate actions and outcomes that are within

an instructors span of control. There are nuances in this use of formative feedback, however, inthat the command climate and organizational culture may, explicitly or implicitly, discourage the

behavior emphasized by the measures (Bandura, 1995; Rasmussen, 1997; Rasmussen, Brehmer,

& Leplat; 1991). In such cases, the measures have broader utility (see section 2.5.4 below).

2.5.2 Quality Assurance and Instructor Education

Measures that are informative and actionable to instructors are ideal for an external observer who

would provide feedback to an instructor for the purpose of assuring or improving quality. They

also facilitate the identification and promulgation of best practices among peers and throughout

an organization. One reason for this is that there are many measures that collectively representbest practices for realizing the principles of OBTE and achieving its intent. An instructor could

improve by behaving better with respect to a particular measure or by behaving differently as

suggested by other measures. In a sense, the complete set of measures is a menu from which

instructors can select in the implementation and systematic improvement of OBTE. In any

situation, it will be easier and more effective to implement OBTE through some types of behaviorthan others. Over time, this will become clear in the feedback obtained with various measures by

various instructors. The measures thus facilitate sharing of lessons learned that are of common

interest and relevance.

An interesting use of the complete measures set would be to design an instructor education course

around them (cf., Pellegrino, et al., 2001; Sidman et al. 2009). Categories of measures (see e.g.,

Chapter 3) could be used to design associated teaching modules ranging, for example, from a

half-day to multiple days in length. In a sense, such a course would be a guided version of what


27/39

45 Marceau et al.


instructors do in using the measures to plan, execute, and assess of their own instruction. In a

pedagogical context, it would be extremely valuable for instructor to be exposed to the

application of the principles and practices of OBTE to different skills or knowledge sets. This

would be good use of the collaborative tactical decision games promoted by the Adaptive

Learning Model (Vandergriff, 2006, 2007) that is closely allied with OBTE.

2.5.3 Continuous Improvement of Assessments

While the COMPASS process results in measures that are specific enough to provide formative

feedback to instructors, they are pointedly not as specific as they could be. In general, there are

just too many nuances and appropriate variations in how a particular instructor can interact with a

particular student. It would not be feasible to list them all, and it wouldnt be desirable to do so

because they would be too dependent on a particular course context. Moreover, in OBTE, it

would antithetical to tell instructors exactly what to do. This is especially important for

organizations in which instructors historically are told what to do

Toward Formative Measures for Instructors in Outcomes Based Training & Education

Documents

Transcript of Toward Formative Measures for Instructors in Outcomes Based Training & Education