AFOTEC’s Space Test Initiative: Transforming OperationalTesting and Evaluation of Space System Capabilities

Major General Stephen T. Sargeant

U.S. Air Force Operational Test and Evaluation Center,

Kirtland Air Force Base, New Mexico

Colonel Suzanne M. Beers

U.S. Air Force Operational Test and Evaluation Center Detachment 4,

Peterson Air Force Base, Colorado

Historically, the value of the operational test and evaluation (OT&E) data has been limited

during the acquisition and deployment of space systems because OT&E occurs late in the process,

after the satellite is orbiting in space and the ground stations are fielded, well after key

acquisition decisions, investments, and critical launch decisions have already been made. This

article presents the U.S. Air Force Operational Test and Evaluation Center’s Space Test

Initiative. The Space Test Initiative delivers an OT&E model that better fits the National

Security Space system’s acquisition model outlined in NSS 03-01 and delivers better value to

both the acquisition and operational decision makers by moving OT&E well before launch.

Key words: Acquisition strategy; integrated testing; investment; OT&E test anatomy;

space acquisition; system of systems evaluation.

The U.S. Air Force Operational Testand Evaluation Center (AFOTEC) isresponsible for the operational testingand evaluation (OT&E) of all Acqui-sition Category I and II weapon system

programs as well as those on Director of OperationalTest and Evaluation oversight, acquired by the AirForce and often our Joint partners, to determineoperational effectiveness, suitability, and degree ofmission capability in the system’s intended operationalenvironment. Since AFOTEC’s inception in 1974 andthe creation of Air Force Space Command in 1982,OT&E of space systems has occurred after satellites areon orbit and ground stations are fielded. Therefore,AFOTEC could not fully meet its responsibility toprovide independent OT&E data to key decisionmakers in a timely manner with regard to theacquisition and deployment decisions of space systemsbecause the tests occurred after the decisions werealready made.

The need for fully informed decisions regardingthese increasingly expensive, yet indispensible capabil-ities is crucial in today’s environment of constrainedresources. For more than 20 years, AFOTEC and theother service operational test agencies (OTAs) con-

ducted OT&E of space and other high-tech, limited-quantity systems using a model more appropriate formilitary systems with large-scale production decisions.Using an OT&E model that does not match thesystem’s acquisition strategy renders the results ofOT&E largely irrelevant. AFOTEC’s ‘‘Space TestInitiative’’ delivers an OT&E model that better fits theNational Security Space (NSS) system’s acquisitionmodel outlined in NSS 03-01 (DoD 2004) andprovides fact-based decision quality data to decisionmakers in time to support their key space systemacquisition decisions.

Figure 1 further illustrates the issue. In a traditionalacquisition program governed by Department ofDefense Directive (DoDD) 5000.1 (DoD 2003),expenditures are relatively small in the research anddevelopment and investment phases compared to thecost of production and system operation. For thesetraditional acquisitions, operational testing (OT)occurs just before the major investment or productiondecision and provides data to inform those decisionsadequately.

However, most of the investment for space systemsoccurs early in the program, most often without amajor production decision. In the current space OT&E

ITEA Journal 2008; 29: 351–359

Copyright ’ 2008 by the International Test and Evaluation Association

29(4) N December 2008 351

Report Documentation Page Form ApprovedOMB No. 0704-0188

Public reporting burden for the collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering andmaintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information,including suggestions for reducing this burden, to Washington Headquarters Services, Directorate for Information Operations and Reports, 1215 Jefferson Davis Highway, Suite 1204, ArlingtonVA 22202-4302. Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to a penalty for failing to comply with a collection of information if itdoes not display a currently valid OMB control number.

1. REPORT DATE DEC 2008 2. REPORT TYPE

3. DATES COVERED 00-00-2008 to 00-00-2008

4. TITLE AND SUBTITLE AFOTEC’s Space Test Initiative: Transforming Operational Testing andEvaluation of Space System Capabilities

5a. CONTRACT NUMBER

5b. GRANT NUMBER

5c. PROGRAM ELEMENT NUMBER

6. AUTHOR(S) 5d. PROJECT NUMBER

5e. TASK NUMBER

5f. WORK UNIT NUMBER

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) U.S. Air Force Operational Test and Evaluation Center,Kirtland AFB,NM,87117

8. PERFORMING ORGANIZATIONREPORT NUMBER

9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSOR/MONITOR’S ACRONYM(S)

11. SPONSOR/MONITOR’S REPORT NUMBER(S)

12. DISTRIBUTION/AVAILABILITY STATEMENT Approved for public release; distribution unlimited

13. SUPPLEMENTARY NOTES

14. ABSTRACT

15. SUBJECT TERMS

16. SECURITY CLASSIFICATION OF: 17. LIMITATION OF ABSTRACT Same as

Report (SAR)

18. NUMBEROF PAGES

9

19a. NAME OFRESPONSIBLE PERSON

a. REPORT unclassified

b. ABSTRACT unclassified

c. THIS PAGE unclassified

Standard Form 298 (Rev. 8-98) Prescribed by ANSI Std Z39-18

model, OT&E takes place at the same point in theacquisition cycle as with the DoDD 5000.1 (DoD2003) programs. However, by this point in NSS 03-01(DoD 2004) programs, most of the investment hasbeen made, most of the key acquisition decisions havebeen made, and the critical operational decision tolaunch the satellite has been made and executed. Theground station and associated software often lag indeployment, making timely post-launch OT&E diffi-cult, if not impossible. Making these key decisionsbefore the execution of OT&E severely limits the valueof OT&E.

AFOTEC’s Space Test Initiative provides anOT&E model that better fits the space systemsacquisition model, delivering better value to both theacquisition and operational decision makers by movingOT&E activity well before launch. The three keytenets of the Space Test Initiative are:

N early and continuous integrated testing involve-ment throughout the life cycle of the system,

N agile analysis and reporting,N focus on system-of-system evaluations.

Space test anatomyAFOTEC’s OT&E guide provides an ‘‘Anatomy of

an OT&E’’ that describes OT&E activities associatedwith each phase of a typical acquisition program. Theanatomy is built on the DoDD 5000.1 acquisitionmodel, which did not fit well for space system

acquisition. In order to guide the OT activities ofspace systems, a NSS 03-01-focused OT&E anatomyneeded development. In July 2008, AFOTEC hostedan Air Force Space Summit at Kirtland Air Force Base,New Mexico, where space acquisition, operations, andtesting experts from across the Air Force gathered tobuild a new test anatomy. After the summit, eventorganizers socialized the ideas to the broader spaceacquisition and testing community both inside andoutside the Air Force. This action included the otherService OTAs, the Joint Staff, Undersecretary ofDefense (Acquisition, Technology, and Logistics),the national intelligence community, and the Directorof OT&E. Comments received during that socializa-tion resulted in slight modifications to the summit’smodel. In this article, we will walk through theresulting anatomy in a phased approach.

The activities shown in orange at the top of theanatomy (Figure 2) are conducted by the acquisitioncommunity. Those shown in light blue are conductedby the developmental test (DT) community. The greyregion with the activities highlighted in yellow areintegrated test activities, conducted by both the DTand OT communities. Finally, the blue boxes near thebottom of the anatomy are activities led by the OTcommunity.

Beginning at the left of the anatomy, early in theacquisition process, the acquisition community receivesstrategic guidance or a description of the operationalmission need. The acquisition community beginsdevelopment of the initial Functional Solution Anal-ysis or system concepts to address the operationalmission need.

During the pre-Key Decision Point (KDP)-Aperiod, the integrated test (IT) community beginsdevelopment of an early involvement strategy. Theearly involvement strategy tailors this generic model tothe specifics of the program, taking into considerationthe required decisions, development, testing activities,etc. In addition, during this early phase the groupresponsible for building operational requirementsforms the Integrated Concept Team. Members of theDT and OT communities also form the IntegratedTest Team (ITT) and develop the ITT charter.

As the Integrated Concept Team develops theFunctional Solution Analysis and the draft InitialCapabilities Document, the IT community is involvedin the early reviews of the proposed concepts togenerate a Concept Assessment Report. The reportprovides input to the concept decision, focused on thedegree to which the system concept meets the missionneeds stated in the strategic guidance.

While the acquisition community moves into thesolution definition phase, the IT community partici-

Figure 1. Department of Defense Directive 5000.1 versus

National Security Space 03-01 life cycle costs

Sargeant & Beers

352 ITEA Journal

pates in the analysis of alternatives (AoA) and course ofaction (COA) development processes. The participa-tion of the ITT in the AoA provides candidateevaluation criteria, potential measures of effectivenessand suitability, and operational scenarios for eachalternative being considered. As the acquisitioncommunity develops the AoA and COA, the ITTdevelops the first test and evaluation strategy bymelding DT and OT objectives.

The ITT’s participation in the AoA/COA culmi-nates in an operational assessment (OA). The resultingOA report informs the KDP-A, Concept Approval,decision. The OA report provides information on thedegree of potential operational effectiveness andsuitability, highlights any disconnects between thealternatives and the operational mission need, andidentifies any potential testing issues of the AoA’salternatives and the COA’s acquisition strategies. TheOA report does not advocate or recommend analternative.

Post-KDP-A to KDP-B, conceptdevelopment phase

Throughout the KDP-A to KDP-B concept devel-opment phase (Figure 3), the acquisition communityrefines the acquisition concept and matures both thetechnology and functional capabilities of the system.Meanwhile, the ITT continues to refine the test andevaluation strategy and builds the integrated test plan.

During the concept development phase, as theacquisition community translates the operationalrequirements into a set of technical requirements to

serve as the basis of the Request for Proposals, the ITTevaluates the Capability Development Document/Technical Requirements Document traceability (seeFigure 4). The look by the ITT at traceability focuseson the translation of operational requirements into thetechnical requirements that will ultimately serve as thebasis for the system design. Throughout the systemrequirements review and system design review process,the technical maturation and functional developmentprocess generates concepts and prototypes. The ITTconducts OAs on these prototypes to evaluate theirpotential operational effectiveness, suitability, anddegree to which they will meet the operational missionneed, and to highlight any other operational issuesnoted during early testing.

The IT planning process culminates in the publica-tion of the initial version of the Test and EvaluationMaster Plan describing the integrated test approach.Finally, the IT community conducts an OA to assessthe system’s concept just before KDP-B to inform theKDP-B decision with an operationally focused evalu-ation of the system concept (see Figure 5).

Post-KDP-B to KDP-C, preliminarydesign phase

In the KDP-B to KDP-C preliminary design phase(Figure 6), the acquisition community refines thesystem design through a series of design reviews andtechnology demonstrations. The IT community fur-ther refines their IT planning documents, wrapping upthe preliminary design phase with a Test andEvaluation Master Plan update and an initial OT&E

Figure 2. Pre-Key Decision Point-A activities

Testing Space System Capabilities

29(4) N December 2008 353

plan that fleshes out the details of how OT objectiveswill be addressed by traditional dedicated DT testingactivities, such as laboratory and chamber testing.

During the preliminary design phase, developersconduct technical demonstrations to evaluate incre-ments or components of the proposed system. TheITT is involved to provide status reports to the systemprogram office on the potential operational effective-ness, suitability, the degree to which they will meet theoperational mission need, and any other notedoperational issues. In addition, these status reportsbegin to form an assessment of the system-of-system

interfaces required for the system to operate success-fully within its operational architecture.

In conjunction with the preliminary design review,the OTA conducts an OA to aggregate the informationgathered through the preliminary design review stage toinform the KDP-C, Final Design Entry, decision on thepotential operational effectiveness, suitability, anddegree to which they will meet the operational missionneed. Additionally, if the acquisition authority decidesduring this timeframe to allow the contractor to procurelong lead items, part of the OA evaluates the operationalaspects of those system components.

Figure 3. Key Decision Point-A activities

Figure 4. Key Decision Point (KDP)-A to KDP-B activities

Sargeant & Beers

354 ITEA Journal

KDP-C to build approval, final designphase

In the final design phase (Figure 7), the acquisitioncommunity refines the system design and conducts aseries of risk-reduction tests, building up fromcomponent tests to subsystems to operational systemtests. The IT community is involved with all testingactivities. ITT participation is collaborative, and thegenerated status reports foster open communicationbetween testers and developers as the system design isfinalized.

At the conclusion of the critical design review, theOT&E community produces an Operational Assess-

ment Report providing information on the potentialoperational effectiveness, suitability, and degree towhich the proposed design will meet the operationalmission need. The critical design review and DesignAssessment Report inform the Build Approval deci-sion.

System production to OT&E phase IAfter Build Approval, the acquisition community

produces the system and conducts a series of testactivities, building up from the component to subsys-tem to full operational system testing. During thesystem production to OT&E phase I period (Figure 8),

Figure 5. Key Decision Point-B activities

Figure 6. Key Decision Point (KDP)-B to KDP-C activities

Testing Space System Capabilities

29(4) N December 2008 355

the ITT participates in the testing, taking fulladvantage of planned DT events to inject OT testmeasures and scenarios and gather information tofulfill OT&E test objectives. Status reports informingdevelopers on how the system production is progress-ing, from both the adherence of the development tospecification and the operational community’s assess-ment of meeting operational requirements, keep thelines of communication open between the operationaland developmental communities.

The system production period culminates in anOT&E Phase I, with its associated Program ElementOfficer certification and Test Readiness Review

processes. The OT&E Phase I puts the system in asnear an operational environment as can be replicatedon the ground to support OT&E to inform theConsent to Ship decision. The Phase I OT&E takesinto consideration the results of integrated testing, aswell as the status of the system-of-systems required toprovide mission capability to the warfighter. Forexample, this report may highlight that the satellite isready for launch, but the ground segment will notbe completed for another 2 years, enabling a consciousdecision to delay satellite preparation for launchuntil the right time to optimize value to thewarfighter.

Figure 7. Key Decision Point-C to build approval activities

Figure 8. System production to operational test and evaluation Phase 1 activities

Sargeant & Beers

356 ITEA Journal

Launch range compatibility testingAfter deciding to ship the satellite from the

manufacturing facility, the system is moved to thelaunch range, mated with the booster, and finalintegration and communication testing occurs. Again,integrated testing will inject OT test measures andscenarios into the DT-centric checkout events toprovide an operational impact to any technical issuesidentified during compatibility testing. Integratedtesting, documented in a Status Report, informsdecision-making at the launch go/no-go decisionpoint.

Launch and early orbit operations, OT&Ephase II

After launch and during test and checkout, earlyorbit operations, and sensor checkout, the operationaltesting community participates to the greatest extentpossible to inject operationally realistic scenarios,backgrounds, and procedures (Figure 9). At theconclusion of the test and checkout period, theProgram Element Officer certifies the system is readyto enter OT&E Phase II, the final 10 percent checkoutof the operational capability of the system. OT&EPhase II takes a final look at whether the system madeits ride to orbit successfully, if the performancereported throughout early integrated testing bears outin the operational environment of space, and that thesystem-of-system environments represent the trueoperational architecture and operate as expected.

AFOTEC conducts OT&E Phase II in conjunctionwith the users’ operational trial period to facilitatedelivering mission capability to the warfighter. At the

conclusion of the OT&E Phase II and exit from thetrial period, AFOTEC generates a status report toidentify the hard-hitting, show-stopping issues foundduring this final stage of operational testing. The statusreport informs the Operational Acceptance Decision.

Depending on the program, the interim summaryreport, an approximately 20-page document thatbegins to draw conclusions and ratings, informsdecisions such as the USSTRATCOM/J65 certifica-tion decision. Finally, AFOTEC publishes the OT&Ereport to provide full details on the results of theanalyses. This report informs the Director of OT&E’sReport to Congress, Initial Operational Capabilitydecisions, future system upgrade decisions, etc.

WayaheadTo develop the next level of detail and implement

the Space Test Initiative, a number of actions arerequired and in most cases are already in works. Theseactions include:

Understand/include detailed DT activities. Thedevelopmental test activities associated with the designdevelopment and maturation phases and systemproduction cycles need further definition and inclusionin this model.

Define necessary policy. Current DoD, Air Force,Air Force Space Command, and AFOTEC policy doesnot speak to conducting space operational testing inthe manner described in the Space Test Initiative.Therefore, AFOTEC initiated a policy crosswalk todetermine what is in existing policy and what must bewritten to allow and direct the Space T&E Anatomy.

Figure 9. Launch and early orbit operations

Testing Space System Capabilities

29(4) N December 2008 357

AFOTEC, in conjunction with the Air Staff, will draftthe necessary policy documentation for incorporationinto the current regulations.

Identify and define underlying test and eval-uation processes. AFOTEC will define the processesrequired to execute this Space T&E Anatomy, includedetails on organizational roles and responsibilities, andentrance/exit criteria for each phase.

Identify and define test personnel resources. Thenumber of personnel required to execute the SpaceTest Initiative, along with the required skill sets, willbe defined. It is likely that AFOTEC will not have, orbe able to increase, their personnel pool to provide thetechnical expertise necessary to execute the Space TestInitiative, particularly the early engineering-focusedactivities. Therefore, we must build agreements amongthe members of the integrated test and developmentcommunity to share personnel resources.

Define capabilities and gaps in test infrastructure.Execution of the initiative’s OT&E Phase I testinfrastructure requires improvement in order toemulate an operationally realistic test space environ-ment on the ground. For example, OT&E Phase I willhave to use vacuum chambers that provide thecapability to connect operational communication andcommand and control links.

Select a long-term candidate program to define cost/benefit. While AFOTEC Detachment 4 intends toapply this concept to all future space OT&E programs,they will select a pilot program to demonstrate anddefine the cost and benefits of this new approach. Inaddition, AFOTEC will use the pilot program torefine the concept, adding lessons learned as we executethese ideas from beginning to end on a space program.

Identify and define required contract changes. Mostcurrent space acquisition programs, particularly thoseinitiated during the acquisition reform era, providelimited opportunity for government participation orinsight into most development activities, or provide fortest community access to developmental testing data.We require future contracts be written to allow theintegrated test activities, as the ability to implementthe Space Test Initiative depends on access todevelopmental data for analysis.

Space Test Initiative benefitsAFOTEC’s Space Test Initiative provides the basis

for knowledge-based acquisition and operationaldecisions throughout the life cycle of our nationalsecurity space systems. It provides early operationalinvolvement that will deliver a number of benefits,

including: (a) ensuring the warfighter receives neededmission capabilities, (b) providing early clarity andcontinued update of operational requirements, (c)influencing early and continual development andrefinement of the Concept of Operations, (d) ensuringfrequent reviews of threat documents to ensure thesystem design addresses current threats, (e) highlight-ing program shortfalls and benefits throughout thedevelopment process when they can be addressed mostefficiently and inexpensively, (f) enabling the user tounderstand and accept acquisition risks and adjust theirmission requirements and plans accordingly, and (g)addressing and correcting systemic suitability issuesearly in the program development.

Other applicationsAlthough AFOTEC’s initiative focuses on space

systems with its satellite-specific activities of Consent toShip, Launch, and Early Orbit Operations, the modelcan be applied to other high-tech, small-quantityprograms, such as one-of-a-kind command and controland information systems. Information systems can alsobenefit from the model of early testing since theseprograms are similarly front-loaded on investment withrelatively little expense on production, operations, andmaintenance once fielded. Like most space programs, notwo information system programs are the same and fewfollow the DoDD 5000.1 template exactly. Unlike spaceprograms, however, the DoD does not field informationsystems at one time (launch). Instead, DoD fieldsinformation systems in increments of capability. Thefielding difference drives a requirement to test sooner andmore often than space programs. However, the SpaceTest Initiative offers a model for information systemsbecause the fundamental principles apply: (a) early andcontinuous integrated test involvement throughout thesystem’s life cycle, (b) agile analysis and reporting, and (c)focus on system-of-systems evaluations. If a flexible, agiletest approach is not used, the warfighter faces thedilemma of fielding capabilities before testing.

SummaryAFOTEC’s proposed Space T&E Anatomy pro-

vides a model for testing systems governed by NSS 03-01. It identifies early test, evaluation, and reportingactivities to inform acquisition and operational deci-sions, providing a roadmap for early program influence.The anatomy also provides an overarching model foreach individual program’s tailored implementation, asno two NSS programs (or DoDD 5000.1 programs forthat matter) follow the standard NSS 03-01 model.

The benefit of the AFOTEC Space Test Initiativewill be better space warfighting systems acquiredthrough early, continuous integrated testing involve-

Sargeant & Beers

358 ITEA Journal

ment, providing inputs to the requirements processesto ensure the system addresses the mission capabilitygap and informing early program decisions whenchanges are less costly. The initiative focuses themajority of the OT&E effort, conceptually 90 percentof the OT&E community’s time, on pre-launch toinform the key Consent to Ship decision. With earlyand continuous involvement, we will ensure thatleaders make conscious, fact-based decisions to sendsatellites into orbit and field new ground stations whenthe complete system-of-systems required to deliverwarfighting capability is in place. %

MAJOR GENERAL STEPHEN T. SARGEANT is theCommander of Air Force Operational Test and Evalua-tion Center (AFOTEC) at Kirtland Air Force Base, NewMexico. Major General Sargeant reports to the Air ForceChief of Staff regarding the operational test andevaluation of more than 200 major programs beingassessed at 22 different locations. He directs the activities ofmore than 950 civilian and military personnel. Asa member of the testing and evaluation community, heworks directly with the Office of the Secretary of Defenseand Headquarters U.S. Air Force, Washington D.C., toensure realistic, objective, and impartial operationaltesting is conducted on Air Force and Joint use systems.Major General Sargeant has served as the commandant ofthe Air Force Weapons School at Nellis AFB, Nevada,commanded the 8th Fighter Wing at Kunsan Air Base,South Korea, and the 56th Fighter Wing at Luke AFB,Arizona. He has also served in numerous Air Force, Joint,and Coalition staff assignments, including 18 months inBaghdad, Iraq, as the C-5 for CJTF-7 and MNF-I. He isa command pilot with more than 3,000 flying hours in theA-10/A and F-16 A/B/C/D. E-mail: steve.sargeant@afotec.af.mil

COLONEL SUZANNE M. BEERS is the Commander of Air

Force Operational Test and Evaluation Center Detach-ment 4 at Peterson Air Force Base, Colorado. She is

responsible for leading, planning, executing, and reportingoperational tests and evaluations of 21 space, interconti-

nental ballistic missile and national missile defense testprograms valued at more than $140 billion. Colonel Beers

advises and interfaces with Air Force Space Command,U.S. Strategic Command, U.S. Northern Command,

North American Aerospace Defense Command, Air ForceMateriel Command, and the Missile Defense Agency to

ensure test partnerships provide the warfighter with thebest possible systems and the operational tests conducted by

AFOTEC describe the impact the systems will have in theoperational battlespace. Colonel Beers has held assignments

in Air Force research and development, acquisition,testing, and sustainment for cruise missiles, satellites, and

space command and control centers. She has operationalexperience with ground launched cruise missiles and the AirForce Satellite Control Network and leadership experience

at the acquisition division and operational group level.E-mail: suzanne.beers@peterson.af.mil or suzanne.beers

@us.af.mil

ReferencesDoD. 2003. Defense Acquisition Guidebook, DOD

Directive 5000.1. The Defense Acquisition System.May 12, 2003. Washington, DC: Department ofDefense. Available at https://akss.dau.mil/dag/DoD5000.

DoD. 2004. National Security Space (NSS) TeamAcquisition Policy Number 03-01. December 27,2004. Washington, DC: Department of Defense.Available at https://akss.dau.mil/Documents/Policy/Space%20Acquisition%20NSSAcqPol0301_signed_%2027Dec04%20(GN).pdf.

Testing Space System Capabilities

29(4) N December 2008 359