KC-135 Executive Summary2

12

Click here to load reader

Transcript of KC-135 Executive Summary2

Page 1: KC-135 Executive Summary2

KC-135 Simulator Systems Engineering Case Study

Executive Summary ID 8845

KC-135 SIMULATOR SYSTEMS ENGINEERING

CASE STUDY

EXECUTIVE SUMMARY

By: Don Chislaghi, Richard Dyer, & Jay Free, MacAulay-Brown, Inc.

Air Force Center for Systems Engineering 2900 Hobson Way, Wright-Patterson AFB, OH 45433-7765

Page 2: KC-135 Executive Summary2

KC-135 Simulator Systems Engineering Case Study

Executive Summary ID 8845

Page ii

FOREWORD At the direction of the Secretary of the Air Force, Dr. James G. Roche, the Air Force Institute of Technology (AFIT) in 2002 established an Air Force Center for Systems Engineering (AFCSE) at its Wright-Patterson AFB, Ohio, campus. With academic oversight by a subcommittee on Systems Engineering (SE), chaired by Air Force Chief Scientist Dr. Alex Levis, the AFCSE was tasked to develop case studies of SE implementation during concept definition, acquisition, and sustainment. The committee drafted an initial case-study outline and learning objectives, and suggested the use of the Friedman-Sage Framework to guide overall analysis.

The Department of Defense (DoD) is exponentially increasing the acquisition of joint complex systems and Systems of Systems (SoS) that deliver needed capabilities demanded by our warfighter. SE is the technical and technical management process that focuses explicitly on delivering and sustaining robust, high-quality, affordable solutions. The Air Force leadership has collectively stated the need to mature a sound SE process throughout the Air Force. Gaining an understanding of the past and distilling learning principles that are then shared with others through our formal education and practitioner support, are critical to achieving continuous improvement.

The AFCSE has published eight case studies thus far, including the C-5A, F-111, Hubble Telescope, Theater Battle Management Core System, B-2, Joint Air-to-Surface Standoff Missile, A-10, Global Positioning System and Peacekeeper ICBM. All case studies are available on the AFCSE web site [http://www.afit.edu/cse]. These case studies support academic instruction on SE within military service academies, civilian and military graduate schools, industry continuing education programs, and those practicing SE in the field. Each of the case studies comprises elements of success as well as examples of SE decisions that, in hindsight, were not optimal. Both types of examples are useful for learning.

Along with discovering historical facts, we have conducted key interviews with program managers and chief engineers, both within the government and those working for the various prime and subcontractors. From this information, we have concluded that the discipline needed to implement SE and the political and acquisition environment surrounding programs continue to challenge our ability to provide balanced technical solutions. We look forward to your comments on this KC-135 Flight Simulator case study and our other AFCSE published case studies.

GEORGE E. MOONEY, SES Director, AF Center for Systems Engineering Air Force Institute of Technology http://www.afit.edu/cse

Approved for Public Release; Distribution Unlimited

The views expressed in this Case Study are those of the author(s) and do not reflect the official policy or position of the United States Air Force, the Department of Defense, or the United Stated Government.

Page 3: KC-135 Executive Summary2

KC-135 Simulator Systems Engineering Case Study

Executive Summary ID 8845

Page iii

TABLE OF CONTENTS EXECUTIVE SUMMARY .......................................................................................................... 1

1.1 KC-135 SIMULATOR CASE STUDY LEARNING PRINCIPLES .............................................. 3 1.2 LEARNING PRINCIPLE-1: SYSTEMS ENGINEERING MUST TRANSLATE PROGRAM GOALS AND OBJECTIVES INTO CLEARLY DEFINED AND VERIFIABLE SYSTEM REQUIREMENTS, FOCUSING ON THE ENTIRE LIFE CYCLE. ......................................................................................................... 4 1.3 LEARNING PRINCIPLE-2 THE SYSTEMS ENGINEERING PROCESS MUST BE STRUCTURED TO PROPERLY MITIGATE CHALLENGES GENERATED BY THIRD-PARTY MODIFICATION CONTRACTORS. ............................................................................................................................ 5 1.4 LEARNING PRINCIPLE-3: SYSTEMS ENGINEERS MUST BE RESPONSIBLE FOR ENSURING THAT ALL STAKEHOLDERS ARE INVOLVED DURING KEY DECISION TECHNICAL PLANNING AND EXECUTION PROCESS REVIEWS. .................................................................................................... 6 1.5 LEARNING PRINCIPLE-4: INTEGRATED LOGISTICS/MAINTAINABILITY SUPPORT STRUCTURE AVOIDS PARTS OBSOLESCENCE AND DIMINISHING MANUFACTURER SUPPLY ISSUES. . 7 1.6 LEARNING PRINCIPLE-5: SIMULATOR MODELING DATA/MODIFICATION REQUIRES VERIFICATION AND VALIDATION TO ENSURE AIRCRAFT-LIKE FLYING QUALITIES. ........................ 8

LIST OF FIGURES Figure 1. KC-135 OFT WITH SIX DEGREE OF FREEDOM MOTION BASE. ................................................... 1 Figure 2. FRIEDMAN-SAGE FRAMEWORK KC-135 SIMULATOR CASE STUDY LEARNING PRINCIPLES. ..... 4

Page 4: KC-135 Executive Summary2

KC-135 Simulator Systems Engineering Case Study

Executive Summary ID 8845

Page 1

EXECUTIVE SUMMARY From its beginning in the early 1960s as mobile KC-135 simulators housed in railroad cars, the KC-135 Aircrew Training System (ATS) has evolved into an effective ground based aircrew training system while simultaneously achieving a high level of customer acceptance and approval. In order to realize the goal of increased usage of the ATS for crew training, several key elements needed to be in place: a cultural change was made within the Air Force tanker community in that training is achievable by effective use of ground based trainers. In addition, money was allocated to effectively operate, maintain, and upgrade ATS capabilities. Corresponding improvements to the hardware and software were realized in a cost effective manner and good relationships between the Government and Contractors were established and maintained. And finally, common goals associated with the efficient development and effective operation of the ATS, were established. The AF KC-135 Aircrew Training System (ATS) is comprised of 19 KC-135 model Operational Flight Trainers (OFT), 27 Global Air Traffic Management Interactive Hand Controller Part Task Trainers, a number of Cockpit Familiarization Trainers, 258 Computer Based Training (CBT) Workstations, one Cargo Loading Trainer, 40 Air Force Mission Support System Computers, and 102 CBT laptop computers. This equipment is distributed across 13 worldwide bases with the main schoolhouse located at Altus AFB. The KC-135 ATS provides initial qualification, re-qualification, upgrade training, difference training, conversion training, the central flight instructor course, and selected continuation training through academics and simulator applications.

The KC-135 Simulator Case Study reflects the KC-135 ATS team’s SE processes as they matured during a phase of major modifications/upgrades made to the KC-135 ATS during the period 1992 – 2007. This timeframe illustrates a wide range of issues encountered/associated with the delivery of realistic and concurrent modifications required to maintain a modern air vehicle training system. The integration of these upgrades is equally complex as those associated with major aircraft weapon systems. Not only must they replicate the aircraft’s physical configuration and performance they must also replicate the environment in which the aircraft operates. The current KC-135 Operational Flight Trainer, see Figure 1, is a fully replicated and functional cockpit trainer with a visual system capable of meeting FAA level C certification and equipped with a full six-Degree of Freedom motion system.

Figure 1. KC-135 OFT with Six Degree of Freedom Motion Base.

Page 5: KC-135 Executive Summary2

KC-135 Simulator Systems Engineering Case Study

Executive Summary ID 8845

Page 2

During this period Air Mobility Command (AMC) established two key program goals that formed the foundation of the KC-135 ATS upgrade strategy. The first addressed the need for concurrency which is to ensure the OFT is upgraded and ready for training prior to the aircraft with its modifications being fielded. The second addressed the Command’s goal to upgrade operational flight simulator training effectiveness.

The KC-135 ATS program took advantage of advancements in computing and visual technology and the use of stimulated actual aircraft equipment to achieve significant improvements in providing cost-effective training to AMC crews. Stimulated equipment uses the actual aircraft line replacement units with appropriate software changes necessary to have the unit function in the simulator as it would in the aircraft. These improvements have allowed the OFT to achieve FAA AC-120-40B Level C simulator certification and demonstrate 99 percent availability ensuring KC-135 air crew training is delivered when and where needed.

Major modifications to the OFT included the addition of a six-degree motion system, a new visual system, aerodynamic model enhancements, incorporation of the KC-135 Pacer Compass Radar and Global Positioning System (CRAG) avionics modernization upgrade, and incorporation of Communications, Navigation, and Surveillance/Air Traffic Management (CNS/ATM) Block 40 capabilities. Implementation of these improvements to the 19 OFTs to maintain commonality and aircraft concurrency have resulted in a dynamic engineering coordination effort between the Air Logistics Center (ALC) aircraft office, Air Education and Training Command (AETC), AMC, the acquisition community at Hill AFB, support service contractor FlightSafety, and third-party subsystem upgrade contractors.

What drove the requirement for a structured SE process was the need for the contractor to ensure that a timely, consistent level of training is provided to the Air Force. Specifically, ensuring all courseware, documentation, hardware, and software (development and integration) is sufficient to provide training value to the user. Some of the reasons given for the team’s success include the fact that the prime contractor understands the user’s training needs because they are the trainers. The products they develop are used by themselves. Therefore, they have good insight into what the products must do. Furthermore, the ATS team consists of a small, highly skilled group of government and contractor personnel who have established, over a period of 17 years, a working relationship based on mutual trust and respect that facilitates their ability to successfully achieve a high level of system performance and customer acceptance.

One of the key processes employed by the team is a rigorous design review cycle with participation by all training system stakeholders including participating in KC-135 weapon system design reviews to ensure the training community is working in parallel with the aircraft community. One of the characteristics of the team is their ability to be flexible and react quickly to customer needs. Ultimately the team’s biggest triumph can be illustrated by the fact that the simulators are typically completed and ready for training before the fielding of the aircraft with the applicable modifications installed. Today, 3,900 aircrew members receive effective training on the KC-135 ATS every year at bases in the United States, United Kingdom, and Japan.

One of the challenges facing the Government in the future is to foster the advantages associated with long term support contracts (i.e., workforce continuity, knowledgeable support personnel, program stability, sense of ownership, incentives for process improvements, incentive for long-range planning) while meeting the Government’s requirements for increased competition and shorter term contracts. The success of the current systems engineering process is due partly to the

Page 6: KC-135 Executive Summary2

KC-135 Simulator Systems Engineering Case Study

Executive Summary ID 8845

Page 3

strength of the highly knowledgeable government and contractor (prime and subs) team that has been assembled.

This case study examines the continuing maturation of the applied systems engineering processes and the interactions of government organizations and contractors necessary to deliver and maintain first rate training equipment. Numerous interviews were conducted with individuals from the using command, acquisition community and support contractor to generate this story.

1.1 KC-135 Simulator Case Study Learning Principles The Friedman-Sage matrix was used to examine the systems engineering and systems management concepts encountered in the development of the KC-135 case study. This examination resulted in the identification of the learning principles depicted in Figure 2. As noted in the case study, the systems engineering processes and management have a significant place in the successful execution of an evolutionary development and support program like the KC-135 simulator.

Concept Domain Responsibility Domain 1. Contractor

Responsibility 2. Shared

Responsibility 3. Government

Responsibility

A. Requirements Definition and Management

LP1 - Systems Engineering must translate program goals and objectives into clearly defined and verifiable system requirements, focusing on the entire life cycle.

B. Systems Architecting and Conceptual Design

LP4- Integrated logistics/maintainability support structure avoids parts obsolescence and diminishing manufacturer supply issues.

LP3 - Systems engineers must be responsible for ensuring that all stakeholders are involved during key decision technical planning and execution process reviews.

C. System and Subsystem Detailed Design and Implementation

LP3 - Systems engineers must be responsible for ensuring that all stakeholders are involved during key decision technical planning and execution process reviews.

LP1 - Systems engineering must translate program goals and objectives into clearly defined and verifiable system requirements, focusing on the entire life cycle.

D. Systems and Interface Integration

E. Validation and Verification LP5 – Simulator modeling data/modification requires verification/validation to ensure aircraft like flying qualities.

F. Deployment and Post Deployment

Page 7: KC-135 Executive Summary2

KC-135 Simulator Systems Engineering Case Study

Executive Summary ID 8845

Page 4

Concept Domain Responsibility Domain G. Life Cycle Support LP4- Integrated

logistics/maintainability support structure avoids parts obsolescence and diminishing manufacturer supply issues.

H. Risk Assessment and Management

I. System and Program Management LP2 - Systems engineering process must be structured to properly mitigate challenges generated by third party Developers.

Figure 2. Friedman-Sage Framework KC-135 Simulator Case Study Learning Principles.

1.2 Learning Principle-1: Systems Engineering must translate program goals and objectives into clearly defined and verifiable system requirements, focusing on the entire life cycle.

Systems engineering needs to be cognizant at all times of the over-arching program goals (i.e., concurrency and Level C certification) to ensure that the subsequent modification requirements lead to and achieve the top-level program goals.

A key element of this process is the upfront reviews that focus on ensuring that the KC-135 ATS team has established a level of mutual expectations and understanding of the system requirements and that the proposed preliminary designs will satisfy those requirements. From these reviews, the contractor can then draw a box around what specific capabilities are to be delivered. The entire ATS is reviewed at quarterly System Review Boards (SRBs) while student critiques are reviewed during monthly Training System Configuration Working Group (TSCWG) meetings at Altus. The entire KC-135 training program is reviewed annually in group forum via the Realistic Training Review Board.

Requirements analysis is a critical tenet of systems engineering and must remain focused not only on the development phase but also on the system’s entire life cycle. As a part of the Pacer CRAG Block 40 program, a FlightSafety systems engineer recognized a potential issue associated with the future operation and maintenance of Block 40 configured OFTs at multiple KC-135 bases and as a result initiated a program requirement for system reconfiguration. Since modifying aircraft from a Block 30 configuration to a Block 40 configuration involved major changes to the aircraft and ATS, systems engineering’s assessment was that a scheduling nightmare might occur given the uncertainty of which bases and which batch of aircraft would get the Block 40 modification. Their derived requirement for system convertibility or reconfiguration, which was flowed down to their subcontractors, was to be able to convert a Block 30 simulator to a Block 40 configuration in eight hours by two people in order to maintain the required training schedule and student throughput. The actual conversion takes less than four hours today and is a main contributor to the OFT’s availability for training.

The systems engineer must be cognizant of all elements that make up the system and ensure the proper identification and allocation of requirements flow down to those elements. As an example, for a ground-based simulator one element is the facility the device is housed in. One

Page 8: KC-135 Executive Summary2

KC-135 Simulator Systems Engineering Case Study

Executive Summary ID 8845

Page 5

such consequence of bringing motion to the KC-135 OFTs related to the “excursion envelope” the motion system required for the trainer within the facility. In some cases, the facility was a few inches too small. In other cases, a new facility was required. Early interaction with local units and the detailed engineering studies of the new motion system requirements allowed local MILCON budgets to be modified and/or initiated so that the facilities would be ready for the motion system. The requirements allocation steps must also address maturity of the technology. During the development of the Visual Upgrade Program there were several technology risks that played a larger role in the modification than the government realized at the time. The technology for cross-window coherent, wide field-of-view visual systems existed, but had never been scaled to the dimensions required to support air refueling and engine out requirements. The contractor was confident this could be done quite easily and convinced the government team. The SE requirements allocation steps failed to identify this as a risk and assumptions were made about the capacity of the motion platforms for both the KC-135 as well as the KC-10 simulators.

Other key elements of requirements allocation include early involvement by the support contractor with the aircraft systems prime contractor(s) to ensure simulator specific requirements are addressed. Because aircraft upgrades are identified by the KC-135 Program Office at Tinker AFB there are roadmap meetings held at Tinker where upcoming modifications to the KC-135 aircraft are discussed. The KC-135 ATS O&M contractor and the Training System Program Office at Ogden now are present to assess those modifications and ensure the ATS requirements are included in the early planning process. During the KC-135 Block 40 upgrade that was developed by Rockwell-Collins, FlightSafety sent personnel to Rockwell-Collins to obtain specific Block 40 Type 1 Training and to attend all of their design reviews to ensure they (FlightSafety personnel) understood fully how the system was intended to function on the aircraft. By working closely with Rockwell-Collins, this early involvement also provided the ATS community with an opportunity to begin the planning and coordinating process for incorporating training system requirements into the aircraft program as needed thereby reducing cost and schedule risk to the ATS upgrade.

For example, Rockwell-Collins modified the aircraft software to incorporate software “hooks” that were needed to facilitate training system development which helped to ensure the fielding of a Block 40 configured training device before aircraft arriving on the ramp. Thus, the Rockwell-Collins’s design solution met the user requirements, but also in this case, facilitated increased trainer efficiency and availability.

Since the Government and Contractor personnel have worked together for over a period of 17 years, an open working relationship has been developed that readily enables FlightSafety to transform Government (KC-135 ATS Program Office and the KC-135 Program Office) program objectives and goals into training systems engineering modifications that meets Government needs. These open communication lines help to ensure that Government expectations are achieved by the FlightSafety proposed modification.

1.3 Learning Principle-2 The Systems Engineering process must be structured to properly mitigate challenges generated by third-party Modification Contractors.

The KC-135 O&M contract states the prime contractor, FlightSafety, is responsible for meeting the overall performance requirements of the training system including trained students that meet Government standards. Competitive contracting for early OFT upgrades did not formally involve

Page 9: KC-135 Executive Summary2

KC-135 Simulator Systems Engineering Case Study

Executive Summary ID 8845

Page 6

buy-in from the O&M contractor who retains ultimate responsibility for providing a “guaranteed” student to the Government. This was recognized and remedied in future contracts by involving the support contractor in a more disciplined manner to ensure early involvement in the development effort. For example, the requirement for a Performance Work Statement (PWS) has evolved that identifies specific tasks to be accomplished by the KC-135 ATS prime in order to ensure training system requirements are identified at the ATS level and properly allocated to the various subsystems under development. Initially, in the contract phase, Associate Contractor Agreements (ACAs) were used to define the relationship between the aircraft subsystem prime contractor and the KC-135 ATS prime contractor. These ACAs typically varied in scope, and as a result, problems occurred such as modifications to the ATS not being thoroughly tested, documentation not in a usable format, interfaces not well-defined, ATPs needing to be properly structured in order to validate performance of the system under development, and in some cases, insufficient spares being bought. In addition, maintaining configuration control of the ATS was an issue since the ATS prime contractor did not have much of a hammer to ensure documentation/drawings provided by third-party contractors were correct and of good quality. Infusing the tenets of a structured systems engineering process early in the development phase mitigate these unacceptable third-party consequences.

1.4 Learning Principle-3: Systems engineers must be responsible for ensuring that all stakeholders are involved during key decision technical planning and execution process reviews.

The KC-135 program requires a tailored set of formal reviews be held during the development phase that is based on the size and complexity of the modification program. These reviews ensure that the entire KC-135 ATS team is working to the same requirements, designing and developing the correct modifications, adequately testing the modifications and generating the appropriate courseware changes. The reviews employed throughout the modification development and verification efforts may include a Systems Requirements Review (SRR), Preliminary Design Review (PDR), Critical Design Review (CDR), Test Readiness Review (TRR), Required Assets Available Review (RAAR) and In Process Reviews (IPR). KC-135 simulator reviews are structured to ensure that the KC-135 Simulator team has mutual expectations and understanding of requirements and that the contractor’s proposed preliminary designs and program plan satisfies the development specification. In addition to internal reviews pertaining specifically to ATS development efforts, it is critical for the ATS systems engineer(s) to remain focused and cognizant of planned KC-135 aircraft modifications to ensure appropriate changes affecting training system concurrency and training effectiveness are addressed early in the development process.

For example, to ensure the proper emphasis is placed on concurrency, KC-135 ATS systems engineers both within the Government and the ATS support contractor review every modification to the aircraft to determine if the modification will affect the OFT and aircrew training programs. For the Pacer CRAG Block 40 upgrade, during the early phase of development the ATS team ensured simulator specific needs were addressed in the aircraft design by working closely with Rockwell-Collins, the aircraft prime contractor. The result of this close coordination was that Rockwell-Collins modified the aircraft software to incorporate software “hooks” that were needed to facilitate training system development ensuring the fielding of a Block 40 configured training device before aircraft arriving on the ramp. In the flight simulator arena, the application of hooks into the aircraft’s operational flight software

Page 10: KC-135 Executive Summary2

KC-135 Simulator Systems Engineering Case Study

Executive Summary ID 8845

Page 7

allows the training system to easily incorporate such simulator unique functions as freeze frame or halting the simulation and return to a previous state of simulation such as repositioning the aircraft on final to repeat a landing sequence.

1.5 Learning Principle-4: Integrated logistics/maintainability support structure avoids parts obsolescence and diminishing manufacturer supply issues.

As with aircraft systems, training systems must also have detailed technical plans that integrate a logistics/maintainability support structure to ensure continued future operation. During each technical planning activity, systems engineering must be concerned with the total support of the system to assure its economic and effective operation throughout its life cycle. Logistics objectives for the program need to be included in these technical plans to ensure achieving stated readiness objectives such as system availability, programmed flying training throughput, establishment of Reliability and Maintainability performance requirements needed to support readiness objectives, and emphasizing logistics support considerations in all design trade studies. During the development phase the systems engineer must incorporate requirements identified by support organizations in order to properly reflect those in the applicable system specification/specification change notices, ensure all design trade studies address these sustainment requirements, and that key design reviews are structured to ensure these issues are adequately and timely addressed. In addition, it is important for the systems engineer to maintain awareness of potential life cycle issues, such as diminishing manufacturing sources (DMS) or parts obsolescence (PO), to ensure there is adequate planning early in the design phase that address the impacts such issues may cause once the design is fielded.

As an example, the KC-135 ATS program has a requirement for Critical Single Point Failure Items to be spared at site level unless Mean Time Between Failure (MTBF) data indicates cost is sufficiently high and failure rate low or item failure will not result in loss of training mission. Critical items will then be stocked at depot level and supplied to the applicable site with a delay of less than 48 hours. These requirements are the responsibility of and managed by the Training System Support Center (TSSC) which has direct responsibility for the continued operation and maintenance of the ATS and is charged with addressing repair, parts obsolescence (PO), and Diminishing Manufacturing Sources (DMS). Systems engineering is charged with ensuring issues and risk mitigation planning associated in these areas are briefed at the quarterly SRBs. This also facilitates the development and maintenance of a list of priority modifications for AMC that can be implemented as fall-out money becomes available. For example, FlightSafety personnel advised AMC of the need for new power supplies. As a result of identifying a potential problem early, power supplies were procured from a third party thereby ensuring critical student throughput requirements continued to be met.

A further example of systems engineering proactively addressing potential DMS/PO issues was the requirement to include a state-of-the-art chip design for many of the subsystems comprising the VUE. The design approach specified required the system to be designed around a family of chips that were essentially backward and forward compatible. This commonality of design aided the support system contractor in maintaining system concurrency and minimizing future sustainment issues.

Page 11: KC-135 Executive Summary2

KC-135 Simulator Systems Engineering Case Study

Executive Summary ID 8845

Page 8

1.6 Learning Principle-5: Simulator modeling data/modification requires verification and validation to ensure aircraft-like flying qualities.

The Systems Engineering process requires both quantitative and qualitative (training value) verification and validation of the modifications to the Operational Flight Trainer (OFT). Quantitative verification ensures that component changes to the OFT are validated via testing to ensure that the modification works in the ATS environment as intended. Low-level, detailed testing by subcontractors and/or third-party contractors is performed to verify the subsystem performance. Functional mission tests are Government-conducted tests of the prototype modification that use Government-defined scenarios to evaluate the operational characteristics of the systems within the context of conducting the mission. Revisions to courseware products, such as, classroom lecture, computer based training (CBT), training device, and aircraft lessons are verified and validated via the formative evaluation processes that include subject matter expert (SME) review, and individual tryouts (ITOs) and small group tryouts (SGTOs) where applicable.

Ultimately, for an OFT to be effective as a training device, the aero models, visual system models, aircraft/cockpit sounds, etc., must provide the student with sufficient cues that are realistic enough to provide for realistic training. For ground-based training devices, this is really a qualitative assessment about the realism of the simulator—meaning it’s a judgment call by the test crews and Air Force instructors about the systems “training value.” To the ATS systems engineer, this issue of subjective testing has been an ongoing dilemma. It has proven extremely difficult for systems engineers to quantitatively specify this training value. No matter how much experience a team has quantifying and measuring simulator performance, in reality it remains a qualitative assessment about the realism of the simulator. The challenge for the systems engineer in a training program is to not only develop performance requirements that can be measured and verified but also develop the process by which “training value” can be qualitatively assessed and validated while protecting against personality-driven assessments that can change with Government personnel turnover.

Therefore, the verification/validation process employed by the KC-135 ATS team relies on a combination of qualitative and quantitative test procedures including the development of Acceptance Test Procedures (ATPs) that are conducted to verify compliance of the modification with the requirements as specified in the Prime Item Development Specifications. The KC-135 simulator program incorporates user validation of system performance, through the simulator certification (SIMCERT) process, to ensure that the OFT “flies” like the aircraft and the graduate meets AMC training standards. Formal SIMCERTs, conducted approximately every six months by the Air Force, certifies that the ATS continues to meet system specification requirements for the hardware and software. Additionally feedback from students is received and analyzed to ensure that the training is provided in an effective manner.

AMC and FlightSafety personnel have also developed a test and evaluation process that promotes confidence that KC-135 simulator modifications “fly” like the KC-135 aircraft. Even though the systems integrating contractor and the Government quantitatively specify many requirements, the final evaluation of the training realism is still subjectively validated. In the end, the trainer model must be correct enough to allow training, which means it’s a judgment call by the test crews and Air Force instructors about the system’s “training value.” The KC-135 ATS team has implemented an approach, which utilizes no more than one or two contractor instructor (FlightSafety) pilots and one Air Force instructor pilot to minimize extended test periods and

Page 12: KC-135 Executive Summary2

KC-135 Simulator Systems Engineering Case Study

Executive Summary ID 8845

Page 9

facilitate reaching consensus on a modification’s training value. Ultimately, both Air Force and FlightSafety pilots validate via testing that the modification fulfills its intended use when placed in its ATS environment.

Finally, the KC-135 ATS team relies on course ending surveys/comments prepared by students, which includes a rating of the training value received (scale of 1 to 5), consistent monitoring of student’s performance and progression, and, as a final proof, a Government-conducted check ride to ensure this requirement is met.