Development of OBTE as a Service System

8/12/2019 Development of OBTE as a Service System

1/17

Riccio, G., & Diedrich, F. (2010). Implications for Service System Development. In: Riccio, G., Diedrich, F., & Cortes, M. (Eds.).An Initiative

in Outcomes-Based Training and Education: Implications for an Integrated Approach to Values-Based Requirements (Chapter 11). Fort Meade,

MD: U.S. Army Asymmetric Warfare Group. [Cover art by Wordle.net represents word frequency in text.]


2/17

Implications for Service System Development 198

Asymmetric Warfare Group

Chapter 11. Implications for Service System Development

Gary E. Riccio

The Wexford Group International

Fred Diedrich

Aptima, Inc.

11.1 Lessons Learned about Transfer of OBTE

The overall goal of OBTE is to foster the development of Soldiers with respect to personal

attributes and competencies that are critical for success in Full Spectrum Operations (Table 1). It

offers a variety of strategies to enable progress toward such outcomes (Darwin, 2008; Cornell-

dEchert, 2009a,b). More specifically, CATC focuses on the role of instructors in the

development of Soldiers through a field-based marksmanship course. Results from the surveys

indicate that participants are generally positive when asked about their experiences in CATC and

that they hold the instructional approach in high regard. Moreover, their comments tend to focus

on outcomes around which the approach is organized (problem solving, intangibles,understanding and awareness, and combat performance). This suggests that they understand the

intent of OBTE. In addition, when asked about longer-term impact, respondents indicated that the

AWG course had an influence on their behavior when in theater, that it influenced their training

approaches with their own units, and that they experienced growth with respect to core outcomes.

The evidence from observations of actual instruction following CATC reveals both opportunities

and challenges. In particular, our findings suggest that DS do not consistently apply OBTE to the

extent possible in Initial Entry Training. In the particular application of OBTE at Fort Benning

and Fort Jackson, transfer of training from CATC is not complete (see also, Dean et al. 2009).

CATC provides exemplary marksmanship training while developing outcomes consistent with

OBTE. While marksmanship is a valuable motivator, it may distract participants in CATC from

learning as much as they otherwise might learn about OBTE as instructors. Perhaps moreimportantly, as the AWG surveys suggest, a single one-week exposure to OBTE probably is notsufficient to educate instructors about the varied opportunities for detailed implementation. This

has implications for further integration and development of instructor education about OBTE into

the Institutional Army (Cornell-dEchert, 2009a,b). In particular, there is a need for capabilities

that can provide or guide opportunities for continuing education about OBTE beyond field-based

courses like CATC or occasional workshops and seminars.

From the perspective of OBTE for process improvement, our data provided actionable feedback

about gaps in DS application of OBTE in Initial Entry Training. For instance, stress often was

added too quickly due to throughput considerations, such that increasing complexity was not used

as a training enabler. Privates did not come to understand why stress is added and how to manage

it. Instead, as training increased in complexity, Privates failed to execute fundamentals adequatelyin the face of increasing complexity. This worked against development of confidence, initiative,

and combat performance. In addition, the level of one-on-one verbal interaction between DS and

Privates was very low. Privates rarely asked questions, and DS rarely asked Privates questions.

This limited the opportunities for growth in problem solving skills and understanding. Privatesoften were told what to do and told solutions, rather than being guided to develop those solutions

themselves. Similarly, DS rarely stressed explanations of why. A consequence of this seemed to

be that Privates likely did not understand the deeper meaning behind their tasks and had limited

chances for real sense-making (e.g., Bransford et al., 2000). All these gaps and shortfalls should


3/17

199 Riccio & Diedrich


be relatively easy to address without fundamental changes to doctrinally prescribed Programs of

Instruction or Training Support Packages (Cornell-dEchert, 2009b).

Collectively, the evidence suggests considerable potential for OBTE in the Army, especially if it

becomes explicitly more directed toward instructor education (Cornell-dEchert, 2009a,b). It is

noteworthy that, even without explicitly addressing instructor education, our findings suggest that

OBTE can motivate individuals to take ownership of their own learning and development and thatit can increase self-efficacy for teaching and developing others. The approach motivates a greater

interest in instructor-student interactions and how these interactions affect progress toward

developmental outcomes while satisfying course-specific learning objectives (Cornell-dEchert,

2009b). This has clear implications for the kind of future research that could have a practical

impact on training in the Army (Bandura, 1995; Deci & Ryan, 1985, 2008; Ryan & Deci, 2008;

Schwartz, Lin, Brophy, & Bransford, 1999; Sidman, Riccio, Semmens, Dean, & Diedrich, 2009).

11.2 Implications for Service System Development

In the context of CMMI for Services (CMMI Product Team, 2009), our findings lead torecommendations for both instructors and their units. It emphasizes aspects of instruction to

sustain or to improve with respect to the ideals of OBTE (Cornell-dEchert, 2009a,b).

11.2.1 Further Development and Analysis of Stakeholder Requirements for OBTE

The practices associated with requirements in the Service System Development process area ofCMMI are not simply a front-end activity. The needs of end-users and stakeholders can change

over the life cycle of a service system. It also is the case than the understanding of requirements

becomes more refined and elaborated over experience with service system development,

transition and delivery (CMMI Product Team, 2009, p. 440; Swain, 2005). Development and

analysis of requirements should be continuous, for example, in due diligence on service incidentsand institutional constraints, restrictions, and limitations on the delivery of service. Some problem

areas observed in the current investigation warrant further collaboration with end-users and

stakeholders, such as:

Emphasis on throughput, or efficiency over effectiveness, puts understanding andmastery at risk. While it is certainly the case that logistical demands necessitate

efficiency, this criterion is meaningless if knowledge or skills are not retained or not

readily adaptable to Full Spectrum Operations.

To the greatest extent possible, advancement across events and application of increasingcomplexity should be based on leaner needs, implying that the pace of training needs tobe more fully differentiated between learners. From the perspective of OBTE, this

emphasizes mastery as a pursuit that builds confidence and effective combat application.

Where possible, instructors should allow Soldiers to assume responsibility for safety andtheir learning. From the perspective of OBTE, such self-reliance builds confidence,

initiative, and accountability. Moreover, to the extent that Soldiers become accountable,DS may be able to concentrate more on instruction and less on management.

11.2.2 Further Development of OBTE as a Service System

While CATC is focused on marksmanship, it is not about marksmanship per se, and this

distinction can be difficult to grasp given the participants intense interest in marksmanship. The

weeklong course, to which the present sample of DS was exposed, focused almost solely on

marksmanship. There was only a small amount of discussion on application of the methods to a


4/17



different domain (e.g., land navigation). DS had difficulty understanding the instructional

techniques they witnessed (e.g., discussing minutes of angle in ballistics) as exemplars of a more

general instructional strategy (e.g., explaining the why, encouraging deliberate thought, and

awareness of context dependence of skills). This is not surprising given that exposure to only one

exemplar is not likely to promote generalization and abstraction. Hence, we recommend that

instructor education include applications of OBTE across multiple domains of skill and learning.

It also is clear that additional support can be provided to participants in a field-based train-the-

trainer course soon after they complete the course. For instance, elements such as take-home

materials that explore application to different domains could be provided, perhaps employing

multimedia-based vignettes that challenge the instructors to apply lessons learned (Bruny,

Riccio, Sidman, Darowski, & Diedrich, 2006). In addition, such materials and exercises could be

employed in workshops, following marksmanship training for example, that promote student

application to domains outside of marksmanship (Sidman et al., 2009). The critical point is to

provide support that gives instructors opportunities to apply what they have learned about OBTE

to a variety of domains. Consistent with OBTE, the approach to further instructor education

would be to introduce additional domains that provide instructors with general strategies andconsiderations rather than an apparent script for instruction in the new domain. Some general

strategies are reiterated below:

Where possible, instructors should allow Soldiers to solve problems. The emphasisshould be on asking leading questions instead of telling Soldiers the solution. From the

perspective of OBTE, this strategy gradually builds skills of thinking and problemsolving, and reinforces expectations of accountability rather than dependence on their

superiors.

The easiest change with a potential for significant impact is to create an environment thatfosters communication with Soldiers, rather than communication directed at Soldiers orthe absence of communication. From the perspective of OBTE, to the extent that

Soldiers feel that they can ask questions, and make some mistakes, they will come to

better understand combat application, master skills, and grow in confidence.

Where possible, peer coach should be utilized to overcome the limitations of instructor-student ratio. Although peer coaches will no doubt provide inappropriate guidance on

occasion, this may be outweighed by the benefit of Soldiers coming to see themselves as

active participants in problem solving and discovery. This is especially true with the high

student-to-instructor ratios that are likely to be the biggest perceived obstacle to full

implementation of OBTE. To the extent that instruction is viewed not only as the transferof knowledge, but as an opportunity for collaborative problem solving, then the role of

the peer coach becomes critical.

It is important to note that inculcation of a mindset consistent with OBTE requires that the work

environment should be open to and supportive of change (e.g., Bandura, 1995; Burke & Hutchins,2007; Rasmussen, 1997). As noted by Dean et al. (2009), DS indicated that they felt somewhat

constrained by the nature of the programs of instruction and associated resources. It is likely that

gaps in transfer of OBTE were rooted, at least in part, in perceived constraints on rifle

marksmanship training and the extent to which the DS believed they could deviate from the

common practices. A key challenge, therefore, is the extent to which programs of instruction can

be made to be more flexible and DS can be empowered to believe that they can take initiative to

more fully implement OBTE. Hence, there is a critical need to educate commanders and

supporting units about the value of OBTE as a service system. Without a command climate that


5/17



fosters agility in instructional units, an approach like OBTE is not likely to be successful

(Appendix C; Haskins, 2009; Schwitters, 2009).

11.2.3 Further Verification and Validation of OBTE

There is a broad range of best practices for peer review, verification, and validation in Service

System Development (CMMI Product Team, 2009, pp. 454-462). As in the current investigation,it is important to employ a multifaceted and multidisciplinary approach to these activities to avoid

sub-optimized solutions and unintended consequences. Consideration of a portfolio of

methodologies, and even some exploration with alternative methods, also helps an organization

find the right level of effort and detail for its process improvement (Garcia & Turner, 2006). In

the present context, a specific aim was to identify ways in which the behavioral and social

sciences can provide guidance to verification and validation of an instructional service system.

There are two general characteristics of our approach in this respect: (a) develop grounded theory

for OBTE; and (b) promulgate scientific inquiry into OBTE as a process over time conducted

within a diverse community of practice.

On our view, identification and development of theoretical foundations is critically important in

service system development because of the diversity of methodologies that are available and thatmay have to be employed for verification and validation. Theoretical coherence arguably is the

only way to identify convergence among otherwise incommensurate sources of evidence. From

the outset, we intentionally included in the research team, a diversity of subdisciplines and

theoretical commitments, mostly within the discipline of scientific psychology. The groundedtheory that emerged in this investigation came to be dominated by theoretical commitments allied

with the interrelated lineages of social learning theory, situated cognition, motivation and

emotion, ecological psychology, and dynamical systems theory in the social and behavioral

sciences. We are explicit about this bias wherever appropriate. More generally, we believe that

reflection and candor about theoretical biases should be a characteristic of systems engineeringapplied to the integration and development of capabilities that have an impact on behavioral and

social phenomena (Flyvbjerg, 2001; Godfrey-Smith, 2003; Schrim & Caterino, 2006; see also,

Epilogue in this monograph).

We believe that the value of a theoretical commitment (e.g., coherence and directedness)outweighs the potential disadvantage of narrowed vigilance in the development of grounded

theory. At the same time, we recognize that involvement of a broader scientific community can

mitigate the potential problem of narrowness. This is important beyond the value of debate,

skepticism, and alternative sources of evidence. It emphasizes an aspect of science that is

consistent with the intent of peer review, verification, and validation in systems engineering.Scientific and technical understanding becomes more refined and elaborate over time with the

accumulation of evidence. One should avoid presumptive judgments based on point estimates

and isolated comparisons with questionable generalizability in domains as broad as training and

education or in specific applications that are replete with uncontrollable sources of variance.

In this regard, there is much to be learned from the other evidence-based services systems and

associated scientific disciplines about how to use and how not to use scientific evidence in social

decision-making (American Association for the Advancement of Science, 1993/2009; Bailar,

1997; Best, Trochim, Haggerty, Moor, & Norman, 2008; Foster & Huber, 1998; Glasziou &

Haynes,2005; Kohn, Corrigan, & Donaldson, 1999; Mislevy & Riconscente, 2006; National

Research Council, 2009; Pellegrino, Chudowsky, & Glaser, 2001; Sackett, Rosenberg, Gray,

Haynes, & Richardson, 1996; Swales, 2000). A socially aware and scientifically based approach

to verification and validation should strive for theoretical coherence, juxtaposition of


6/17



complementary and opposing perspectives, traceability to programs of research that span decades,

empirical evidence from multiple methods that are replicable, and consideration of sources of

variability and uniqueness in empirical findings to ensure that conclusions are credible,

transferable, dependable, and confirmable (cf., Denzin & Lincoln, 2003; Foucault, 1966/2002;

Godfrey-Smith, 2003; Kuhn, 1962/1970; Popper, 1959). It should not be limited by narrow

conceptions of hypothesis testing (cf., Flyvbjerg, 2001; Henkel & Morrison, 1970/2006; Kline,

2004; A. Ryan, 1959; Schrim & Caterino, 2006).

With respect to verification and validation, as well as scientific influence on these endeavors, an

important contribution of this investigation is to stress the importance of productive dialog about

OBTE within a diverse community of stakeholders. The intent of this dialog should not be to

prescribe the use of particular instructional methods or techniques but to provide some scientific

guidance about the most fruitful topics of conversation and innovation by instructors (cf., James,

1899/1907, pp. 7-11). In particular, we believe this investigation can stimulate productive dialog

because of the definition of OBTE in terms of instructor behavior and instructor-student

interactions and, more specifically, because of the measures of instructor and student behavior

that enable OBTE to be verifiable. They provide one topic of conversation for forums in whichthere is peer-to-peer sharing of information about best practice in training and education (e.g.,

Costanza, Leibrecht, Cooper, & Sanders, 2009). In the context of continuous verification andvalidation, we believe such conversations should include scientists as well as instructors,

instructional designers, course developers, quality assurance personnel, and commanders in

instructional programs. The most radical departure would be to facilitate contributions from

stakeholders in theater (Riccio, dEchert, et al., 2006; Riccio, Lerario, et al., 2006).

There is dialog and debate about OBTE in a diverse community of stakeholders, and it appears to

be growing (AWG, 2009). Figure 6 suggests a challenge in achieving efficient sharing of issues

and lessons learned about OBTE. Stakeholders are widely dispersed. Decentralized collaboration

and the resulting lessons learned thus are not readily apparent in timely fashion to decisionmakers. Potential solutions to this problem are emerging in the nascent Army Training Network

(ATN) that builds on the recently revised Army Field Manual, FM 7-0 Training for Full

Spectrum Operations, and transforms FM 7-1 into a Virtual Field Manual (Davis, 2009). Oneuse of this forum would be for continuous peer review that is central to verification and validation

in CMMI Service System Development (CMMI Product Team, 2009). A scientific approach toverification and validation can help ATN establish a topic of peer-to-peer discussion that is more

likely to be on point, efficient, systematic, and actionable for OBTE in particular and for good

training in general.

Figure 6: Needs for distributed peer-to-peer collaboration about OBTE (after Devens, 2009)


7/17



11.3 References

American Association for the Advancement of Science (2009).Benchmarks for scientific literacy.

New York: Oxford University Press. Retrieved April, 2009, from American Association for

the Advancement of Science, Project 2061, http://tinyurl.com/c9mrcr. (Original work

published 1993)Asymmetric Warfare Group (2009). U.S. Army Asymmetric Warfare Group workshop on

Outcomes-based Training and Education. Applied Physics Laboratory, Johns Hopkins

University, Laurel, MD.

Bailar, J. (1997). The promise and problems of meta-analysis.New England Journal of Medicine,

337(8), 559-561.Bandura, A. (Ed.) (1995). Self efficacy in changing societies. Cambridge, UK: Cambridge

University Press.

Best, A., Trochim, W., Haggerty, J., Moor, G., & Norman, C. (2008). Systems Thinking for

Knowledge Integration: New Models for Policy-Research Collaboration. In L. McKee, E.

Ferlie & P. Hyde (Eds.), Organizing and Reorganizing: Power and Change in Health Care

Organizations(pp. 154-166). New York, NY: Palgrave Macmillan.

Bransford, J. D., Brown, A. L., & Cocking, R. R. (2000).How people learn: Brain, mind,experience, & school.Washington, DC: National Academy Press.

Bruny, T., Riccio, G., Sidman, J., Darowski, A., & Diedrich, F. (2006). Enhancing warrior ethos

in initial entry training. Proceedings of the 50th Annual Meeting of the Human Factors and

Ergonomics Society, San Francisco, CA.

Burke, L.A., & Hutchins, H.M. (2007). Training transfer: An integrative literature review.Human

Resources Development Review, 6, 263-296.

CMMI Product Team (2009). CMMI for services, version 1.2. (CMU/SEI-TR-2009-001; ESC-

TR-2009-001). Pittsburgh, PA: Carnegie Mellon University.

Cornell-dEchert, B. (2009a).An introduction to outcomes-based training and education. Fort

Meade, MD: Asymmetric Warfare Group.

Cornell-dEchert, B. (2009b). Outcomes-based training and education: Implementation guide.

Fort Meade, MD: Asymmetric Warfare Group.

Costanza, M., Leibrecht, B., Cooper, W., & Sanders, W. (2009).Peer-To-Peer Training

Facilitators Guide(ARI Research Product, in press). Alexandria, VA: U.S. Army Research

Institute for the Behavioral and Social Sciences.

Darwin, M. (2008). Outcomes-based training and education: fostering adaptability in full

spectrum operations(Briefing, December 2008). Fort Meade, MD: Asymmetric Warfare

Group.

Davis, J. (2008, January 21). Call for Army Training Network (ATN) training products.Retrieved

March, 2009 from US Army Combined Arms Center, Blog Network, Army Training

Network, http://tinyurl.com/alxknw.

Dean, C., Diedrich, F., Artis, S., Horn, Z., Jefferson, T., & Riccio, G. (2008). Outcomes-based

training and education: Student measures for rifle marksmanship.Report to the Asymmetric

Warfare Group (Contract W9113M-06-D-0005). Vienna, VA: Wexford-CACI, Inc.

Devens, M. (2009, March).Introductory remarks. Presentation at the US Army Asymmetric

Warfare Group workshop on Outcomes-based Training and Education, Applied Physics

Laboratory, Johns Hopkins University, Laurel, MD.

Deci, E.L. & Ryan, R.M. (1985).Intrinsic motivation and self-determination in human behavior.

New York: Plenum.

Deci. E.L. & Ryan, R.M. (2008). Self determination theory: A macrotheory of human motivation,

development, and health. Canadian Psychology, 49, 182-185.


8/17



Flyvbjerg, B. (2001).Making social science matter: Why social inquiry fails and how it can

succeed again. Cambridge, UK: Cambridge University.

Foster, K, & Huber, P. (1998).Judging science: Scientific knowledge and the federal courts.

Cambridge, MA: MIT.

Foucault, M. (2002). Order of things: An archaeology of the human sciencesNew York:

Routledge Classics. (Original work published 1966).

Garcia, S, & Turner, R. (2006). CMMI(R) survival guide: Just enough process improvement.Upper Salle River, NJ: Addison-Wesley.

Glasziou, P., & Haynes,B. (2005). The paths from research to improved health outcomes.ACP

Journal Club, 142(2), A8-10.

Godfrey-Smith, P. (2003). Theory and reality.Chicago, IL: University of Chicago Press.

Haskins, C. (2009, March).Development of outcomes based training. Presentation at the US

Army Asymmetric Warfare Group workshop on Outcomes-based Training and Education,

Applied Physics Laboratory, Johns Hopkins University, Laurel MD.

Henkel, R. & Morrison, D. (Eds.) (2006). The significance test controversy: A reader.

Piscataway, NJ: Aldine Transaction. (Original work published 1970)

James, W. (1899/1907). Talks to teachers on psychology: and to students on some of life's ideals.New York, NY: Henry Holt and Company.

Kline, R.B. (2004).Beyond significance testing: Reforming data analysis methods in behavioralresearch.Washington, DC: APA.

Kohn, L., Corrigan, J., & Donaldson, M. (Eds.) (1999). To err is human: Building a safer health

system.Washington, DC: National Academy Press.

Kuhn, Thomas (1970). The structure of scientific revolutions. Chicago: University of ChicagoPress. (Original work published 1962)

Mislevy, R. J., & Riconscente, M. M. (2006). Evidence-centered assessment design: Layers,

concepts, and terminology. In S. Downing & T. Haladyna (Eds.),Handbook of Test

Development (pp. 61-90). Mahwah, NJ: Erlbaum.

National Research Council (2009). Strengthening forensic science in the United States: A pathforward. Retrieved February, 2009 from National Academies Press,

http://www.nap.edu/catalog/12589.html.

Pellegrino, J., Chudowsky, N., & Glaser, R. (Eds.) (2001).Knowing what students know: Thescience and design of educational assessment. Washington, DC: National Academy Press.

Popper, Karl (1959). The logic of scientific discovery. New York: Basic Books.Rasmussen, J. (1997). Risk management in a dynamic society: a modeling problem. Safety

Science, 27, 183-213.

Riccio, G., dEchert, B.C., Lerario, M., Pound, D., Bruny, T., & Diedrich, F. (2006).Enhancing

Joint Task Force Cognitive Leadership Skills. Report to the Army Research Institute for the

Behavioral and Social Sciences, contract number Army contract no. W74V8H-06-P-0186.Vienna, VA: The Wexford Group International.

Riccio, G., Lerario, M., Cornell dEchert, B., Pound, D., Bruny, T., & Diedrich, F. (2006).

Training a Joint and expeditionary mindset. Report to the Army Research Institute for the

Behavioral and Social Sciences, contract number W74V8H-06-P-0189. Vienna, VA: The

Wexford Group International.Ryan, T. (1959). Multiple comparisons in psychological research,Psychological Bulletin, 56, 26-

47.

Sackett, D.L., Rosenberg, W.M., Gray, J.A., Haynes, R.B., & Richardson, W.S. (1996). Evidence

based medicine: What it is and what it isn't.British Journal of Medicine,312(7023), 71-72.

Schwartz, D.L., Lin, X., Brophy, S., & Bransford, J.D. (1999). Toward the development of

flexibly adaptive instructional designs. In: C.M. Reigilut (Ed.).Instructional Design Theories

and Models: Volume II.Hillsdale, NJ: Erlbaum.


9/17



Schwitters, J. (2009, March). Command imperative for change.Presentation at the US Army

Asymmetric Warfare Group workshop on Outcomes-based Training and Education, Applied

Physics Laboratory, Johns Hopkins University, Laurel, MD.

Schrim, S. & Caterino, B. (2006).Making political science matter: debating knowledge,

research, and method. New York: New York University.

Sidman, J., Riccio, G., Semmens, R., Geyer, A., Dean, C., & Frederick Diedrich, F. (2009).

Reshaping Army institutional training: Current training.Final Report to the Army ResearchInstitute for the Behavior and Social Sciences, contract number W74V8H-04-D-0047 DO

0010.

Swain, R. (2005). Changes in instructional system design (ISD): Improving training product

delivery to United States Army Soldiers. USAWC Strategic Research Project. Carlisle

Barracks, PA: U.S. Army War College.

Swales, J. (2000). The troublesome search for evidence: Three cultures in need of integration.

Journal of the Royal Society of Medicine, 93, 402-407.

Tobias, S. & Duffy, T. (eds.) (2009). Constructivist instruction: Success or failure.New York:

Routledge:


10/17

ix Riccio, Diedrich, & Cortes (Eds.)


TABLE OF CONTENTS

.

page

Prologue: A Programmatic View of the Inquiry into Outcomes-Based Training & Education....... 1Historicity of our Research on OBTE..........................................................................................1The Approach and Lessons Learned from the Research..............................................................3Documentation of the Research ...................................................................................................4

Section I. Development of Stakeholder Requirements for OBTE..............................................6 Chapter 1. Preparation for Full Spectrum Operations ......................................................................7

1.1 Requirements of Full Spectrum Operations...........................................................................81.2 Outcomes-Based Training and Education (OBTE)..............................................................10

1.2.1 Exemplar of OBTE: Combat Applications Training Course........................................111.2.2 OBTE as a Multifaceted Instructional System .............................................................12

1.3 An Appraisal of Instruction with Respect to OBTE ............................................................131.3.1 A Systems Engineering Framework for Integration and Development of OBTE........131.3.2 Preparation for Validation and Verification .................................................................14

1.4 References ............................................................................................................................17Chapter 2. Formative Measures for Instructors..............................................................................20

2.1 Development of Formative Measures ..................................................................................202.1.1 The COMPASS Methodology......................................................................................202.1.2 Development of Measures for OBTE ...........................................................................21

2.2 Description of Formative Measures .....................................................................................212.2.1 Results of the COMPASS Process................................................................................212.2.2 Elaboration on the Description of Measures.................................................................23

2.3 OBTE Performance Measures: Planning for Training.........................................................232.3.1 Define Outcomes ..........................................................................................................232.3.2 Create a Positive Learning Environment......................................................................252.3.3 Create the Parameters of Learning................................................................................27

2.4 OBTE Performance Indicators: Training Execution............................................................282.4.1 Communicate the Parameters of Learning....................................................................282.4.2 Training Emphasizes Broad Combat or Mission Success ............................................292.4.3 Customize Instruction When Possible Based on Constraints/Conditions ....................312.4.4 Facilitates Learning of Concepts ..................................................................................322.4.5 Creates a positive learning environment.......................................................................342.4.6 Instructors Utilize Measures of Effectiveness & Self-Evaluation................................362.4.7 Uses scenarios to facilitate learning..............................................................................382.4.8 Instructors exhibit intangible attributes in own actions................................................402.4.9 Hotwashes and Mini-AAR............................................................................................42

2.5 Uses of the Measures ...........................................................................................................432.5.1 Formative Measures for Instructors..............................................................................442.5.2 Quality Assurance and Instructor Education ................................................................442.5.3 Continuous Improvement of Assessments....................................................................452.5.4 Program Evaluation and Organizational Change..........................................................46

2.6 References ............................................................................................................................46


11/17

Table of Contents x


Chapter 3. Principles and Practices of Outcomes Based Training & Education............................503.1 Multifaceted Inquiry.............................................................................................................50

3.1.1 Interaction with Progenitors of OBTE..........................................................................513.1.2 AWG Documents on OBTE .........................................................................................523.1.3 Collaborative Reflection on Participant Observation in CATC ...................................523.1.4 Interaction with Stakeholders .......................................................................................53

3.2 Essential Characteristics of OBTE.......................................................................................533.2.1 The Meaning of Developmental is a Critical Difference..............................................533.2.2 The Definition of Outcomes is a Critical Difference....................................................563.2.3 The Emphasis on Values and Causally Potent Intangibles is a Critical Difference .....583.2.4 The Meaning of Experience is a Critical Difference....................................................613.2.5 The Emphasis on Instructor-Student Interactions is a Critical Difference ...................623.2.6 The Emphasis on Learning to Learn is a Critical Difference .......................................633.2.7 The Emphasis on Collaborative Design and Development is a Critical Difference.....65

3.3 Toward a Grounded Theory for OBTE................................................................................663.3.1 Need for an Integrated Interdisciplinary Framework ...................................................663.3.2 Formative Measures of Instructor Behavior as Evolving Best Practices of OBTE......67

3.4 Emerging Best Practices in OBTE for a Community-Centered Environment.....................683.4.1 Leadership and Enculturation of Soldiers.....................................................................683.4.2 Robust and Adaptable Plan...........................................................................................703.4.3 Instructors as Role Models ...........................................................................................703.4.4 Collaborative Identification of Outcomes and Measures .............................................71

3.5 Emerging Best Practices in OBTE for a Knowledge-Centered Environment .....................713.5.1 Integrated Understanding of Basic Soldier Skills in Full Spectrum Operations ..........723.5.2 Task Relevance of Planned Instructional Events..........................................................723.5.3 Reveal Operational Relevance of Training...................................................................733.5.4 Incorporate Stress into Instructional Events .................................................................733.5.5 Identify General Lessons Learned and Extrapolate to New Situations ........................74

3.6 Emerging Best Practices in OBTE for an Assessment-Centered Environment...................743.6.1 Collaborative Reflection and Problem Solving ............................................................753.6.2 Communication.............................................................................................................753.6.3 Nature and Extent of Guidance.....................................................................................763.6.4 Establish a Pervasive Mindset of Collaborative Reflection..........................................76

3.7 Emerging Best Practices in OBTE for a Learner-Centered Environment ...........................773.7.1 Soldier Motivation and Development of Intangibles....................................................773.7.2 Plan for Development of the Individual .......................................................................783.7.3 Get Students to Take Ownership ..................................................................................783.7.4 Collaborative Reflection as a Means to Develop Self Efficacy....................................79

3.8 References ............................................................................................................................79Chapter 4. Grounded Theory for Values-Based Training & Education.........................................86

4.1 Exploration of Holistic and Functionalistic Underpinnings for OBTE ...............................864.1.1 Fundamental Units of Analysis.....................................................................................874.1.2 Nested Time Scales and Adaptability...........................................................................884.1.3 Adaptability and Ambiguity .........................................................................................904.1.4 Mechanistic Analogies and Predominant Experimental Paradigms .............................92

4.2 Three Pillars for the Scientific Foundation of OBTE ..........................................................93 4.2.1 Ecological Psychology..................................................................................................934.2.2 Self-Efficacy Theory.....................................................................................................974.2.3 Positive psychology......................................................................................................98

4.3 A More Integrated Scientific Infrastructure.......................................................................101


12/17

xi Riccio, Diedrich, & Cortes (Eds.)


4.3.1 Self Determination Theory .........................................................................................1014.3.2 Situated Learning Theory ...........................................................................................1034.3.3 Existential Psychology................................................................................................105

4.4 Building on the Scientific Infrastructure for OBTE...........................................................1094.4.1 Triadic Frameworks....................................................................................................1094.4.2 Further Development ..................................................................................................112

4.5 References ..........................................................................................................................112Chapter 5. Passion and Reason in Values-Based Learning & Development ...............................118

5.1 The Nested Self ..................................................................................................................1185.1.1 An Alternative to Individual versus Collective ..........................................................1185.1.2 Cognition and Reality .................................................................................................119

5.2 Conscious Experience and the Dynamics of Thinking ......................................................1225.3 Emotion, Information, and Engagement ............................................................................125

5.3.1 Ecological Perspective on Emotion ............................................................................1255.3.2 Emotion as Engagement .............................................................................................1265.3.3 Implications for Training and Education....................................................................129

5.4 Emotion, Decision-Making, and Inter-Temporal Choice...................................................1295.4.1 Toward a More Integrated Theory..............................................................................1295.4.2 Emotion and Decision-Making...................................................................................1305.4.3 Emotion and Nested Time Scales ...............................................................................1315.4.4 Neuroeconomics and Inter-Temporal Reasoning .......................................................1325.5.5 Inter-Temporal Reasoning and Adaptive Dynamical Systems...................................133

5.5 Beyond Science..................................................................................................................1345.5.1 Existentialism..............................................................................................................1345.5.2 The Soldier-Scholar as an Emergent Property of a Collective Pursuit.......................135

5.6 References ..........................................................................................................................137Section II. Verification and Validation of OBTE as a Service System ..................................142 Chapter 6. Initial Impressions of Participation in CATC .............................................................1436.1 Methods..............................................................................................................................143

6.1.1 Participants..................................................................................................................1436.1.2 Procedure ....................................................................................................................1436.1.3 Analyses......................................................................................................................144

6.2 Results ................................................................................................................................1446.3 Implications for Service System Development: Peer Review ...........................................1466.4 References ..........................................................................................................................147

Chapter 7. Local Development of Measures of Effectiveness .....................................................1497.1 What do Instructors Believe Soldiers Should Learn in Initial Entry Training?.................1497.2 Measure Development Process ..........................................................................................1507.3 What do OBTE-Trained DS Believe is Important to Assess in BRM/ARM? ................... 1517.4 Implications........................................................................................................................1567.5 Conclusions........................................................................................................................1587.6 References ..........................................................................................................................159


13/17

Table of Contents xii


Chapter 8. Observations of Behavior and Communication in Rifle Marksmanship Training .....1608.1 Methods..............................................................................................................................160

8.1.1 Participants..................................................................................................................1608.1.2 Procedure ....................................................................................................................1608.1.3 Analyses......................................................................................................................161

8.2 Results ................................................................................................................................1638.2.1 Behavior of DS ...........................................................................................................1638.2.2 Behavior and Performance of Privates .......................................................................1658.2.3 Patterns of Communication ........................................................................................1688.2.4 Potential Influence of Instructor Behavior on Performance of Privates.....................170

8.3 Implications for Service System Development..................................................................1718.3.1 Verification of OBTE .................................................................................................1718.3.2 Validation of OBTE....................................................................................................172

8.4 References ..........................................................................................................................173Chapter 9. Impact on Rifle Marksmanship Training....................................................................174

9.1 Behavioral Data Collection During Basic Rifle Marksmanship ........................................1749.1.1 Method........................................................................................................................1749.1.2 Assessment..................................................................................................................1759.1.3 Results An Overview...............................................................................................1779.1.4 Evidence for Influence of OBTE................................................................................1789.1.5 Behavior of Drill Sergeants after Exposure to OBTE ................................................1809.1.6 Behavior of Privates....................................................................................................1829.1.7 Patterns of Communication ........................................................................................1869.1.8 Summary.....................................................................................................................186

9.2 Attitudes Toward an OBTE in Basic Training...................................................................1879.2.1 Method........................................................................................................................1879.2.2 Results.........................................................................................................................187

9.4 References ..........................................................................................................................191Chapter 10. Influence of CATC in an Operational Setting ..........................................................19210.1 Methods............................................................................................................................192

10.1.1 Participants................................................................................................................19210.1.2 Procedure ..................................................................................................................19210.1.3 Analyses....................................................................................................................193

10.2 Results ..............................................................................................................................19310.2.1 Downstream Impact on Marksmanship ....................................................................19310.2.2 Downstream Impact on Training in the Units ..........................................................19410.2.3 Downstream Impact on Self Efficacy.......................................................................195

10.3 Implications for Service System Development: Validation.............................................19610.4 References ........................................................................................................................197

Chapter 11. Implications for Service System Development.........................................................19811.1 Lessons Learned about Transfer of OBTE.......................................................................19811.2 Implications for Service System Development................................................................199

11.2.1 Further Development and Analysis of Stakeholder Requirements for OBTE..........19911.2.2 Further Development of OBTE as a Service System ...............................................199 11.2.3 Further Verification and Validation of OBTE..........................................................201

11.3 References ........................................................................................................................203


14/17

xiii Riccio, Diedrich, & Cortes (Eds.)


Section III. Further Development of OBTE as a Service System ..........................................206Chapter 12. Development of General Measures for Students ......................................................207

12.1 Intent ................................................................................................................................20712.2 Performance Measure Development Process...................................................................207

12.2.1 Phase One: Define Performance Indicators (PI).......................................................20712.2.2 Phase Two: Translate PI into performance measures...............................................20812.2.3 Phase Three: Measure refinement.............................................................................20812.2.4 Phase Four: Retranslation of Measures ....................................................................208

12.3 Product of Measure Development....................................................................................20912.3.1 Learner Perception of the Instructor and Course......................................................20912.3.2 Learner Engagement.................................................................................................21112.3.3 Student Relationship with Teacher ...........................................................................21212.3.4 Student Results .........................................................................................................21412.3.5 Self-Report Measures ...............................................................................................216

12.4 Conclusion........................................................................................................................21712.5 References ........................................................................................................................217

Chapter 13. Adapting OBTE in a Classroom Environment .........................................................21913.1 Intent ................................................................................................................................21913.2 Observing OBTE in the Classroom Environment............................................................219

13.2.1. Participants...............................................................................................................21913.2.2. Procedure .................................................................................................................22013.2.3. Measures ..................................................................................................................220

13.3 Utility of OBTE Measures in a Classroom Environment ................................................22013.3.1 Generality of Measures.............................................................................................22013.3.2. Implications for Improvement of Measures.............................................................22113.3.3 Implications for improvement of course design .......................................................222

13.4 Use of 360 Reviews for Collaborative Reflection..........................................................22313.4.1 The Role of a 360 Review in OBTE .......................................................................22313.4.2 Narrative of a Participant Observer ..........................................................................22513.5 Learning, cognitive load and motivation..........................................................................22813.5.1 The NASA Task Load Index as a subjective measure of student workload.............22813.5.2 Results.......................................................................................................................22913.5.3 Implications ..............................................................................................................230

13.6 Conclusions......................................................................................................................23013.7 References ........................................................................................................................231

Chapter 14. Organizational Climate and Creation of Durable Change ........................................23314.1 The Need ..........................................................................................................................23314.2 Initial Indications of Possible Resistance to Change .......................................................23414.3 Models and Considerations for Sustainable Change........................................................235

14.3.1 The Change Transition Period ..................................................................................23514.3.2 Organizational Culture..............................................................................................23714.3.3 Clarity of Mission and Shared Understanding..........................................................23714.3.4 Relevant Observations During the Current Investigation.........................................23814.3.5 Organizational Support and Incentives.....................................................................238

14.4 Conclusions......................................................................................................................23914.5 References ........................................................................................................................239


15/17

Table of Contents xiv


Chapter 15. Five ways OBTE can enable the Army Leader Development Strategy....................24215.1 Background ......................................................................................................................24215.2 An Emerging Consensus..................................................................................................244

15.2.1 What Part to Balance?...............................................................................................24415.2.2 Improving Training, by Design ................................................................................24515.2.3 Increased Use of dL and Dependence on Self-Development ...................................24615.2.4 Future Orientation, Unknown Requirements............................................................24715.2.5 The Quality Instructor Challenge .............................................................................24715.2.6 Purpose and Design are Key.....................................................................................24815.2.7 A Natural Advantage ................................................................................................24915.2.8 Task Specialization or Generalized Competency.....................................................249

15.3 Conclusion........................................................................................................................25115.4 References ........................................................................................................................252

Epilogue. Integration of Leader Development, Education, Training, and Self-Development.....254Toward Values-Based Standards for Army Doctrinal Requirements ......................................254Nested Standards and Quality Assurance.................................................................................256Needs and Opportunities for Staff & Faculty Development.................................................... 259

A Role for Science and Measurement .................................................................................259Toward Best Practices in Instructor Education....................................................................260Critical Considerations for Further Scientific Investigation ....................................................263

The Necessity of Long-Term Studies ..................................................................................263False Dichotomy of Objective-Subjective...........................................................................264Clarity About What Is Evaluated.........................................................................................265Next Steps ............................................................................................................................266

References ................................................................................................................................268Section IV. Appendices...............................................................................................................270 Appendix A. OBTE Principles & Practices: Instructor Measures................................................271

A.1 Genesis of Formative Measures for Instructors ................................................................271A.2 Principles of Outcomes-Based Training & Education......................................................272A.3 Guide to Using Measures of Instructor Behavior..............................................................276A.4 Complete Menu of Instructor Measures............................................................................279

Appendix B. OBTE Principles & Practices: Student Measures...................................................318B.1 Guide to Using Measures of Student Behavior ................................................................. 318B.2 Complete Menu of Student Measures ...............................................................................319

Appendix C: A Commanders View of Outcomes-Based Training and Education..................... 340Summary ..................................................................................................................................340

Definition.............................................................................................................................340Description...........................................................................................................................340 Elements of OBTE. ..................................................................................................................341Developing the Outcomes....................................................................................................341Developing the Training Plan ..............................................................................................341Conducting Training............................................................................................................342How Training is Assessed....................................................................................................344

Conclusion................................................................................................................................344


16/17

xv Riccio, Diedrich, & Cortes (Eds.)


Appendix D: Warrior Ethos..........................................................................................................345Analysis of the Concept and Initial Development of Applications..........................................345

Current Understanding of Warrior Ethos.............................................................................345Purpose.................................................................................................................................348 Approach..............................................................................................................................348 Expansion of the Definition of Warrior Ethos.....................................................................348The Tenets of Warrior Ethos ...............................................................................................349Clarifying the Definition of Warrior Ethos..........................................................................351Warrior Attributes Derived from the Tenets of Warrior Ethos ...........................................353

References ................................................................................................................................355Supplementary Work Product from Warrior Ethos Project .....................................................355

Appendix E: Indicators of Warrior Ethos.....................................................................................356Methods....................................................................................................................................356

Participants...........................................................................................................................356 Instruments and Facilities ....................................................................................................356Procedure .............................................................................................................................356

Results ...................................................................................................................................... 358Qualitative Findings.............................................................................................................358Quantitative Findings...........................................................................................................358

Discussion ................................................................................................................................359


17/17

!"#$%&'#( #* &+, '(",-&'./&'#( /- 0,*$,1&,2 '( &+, 1+/3&,0- #* &+'- 4#(#.0/3+5

Development of OBTE as a Service System

Documents

Transcript of Development of OBTE as a Service System