SOFIA Program SE&I Lessons Learned SOFIA

1

Interstellar Medium Formation of Stars

and Planets Planetary Science Galaxies and the Galactic Center

SOFIA Stratospheric Observatory for Infrared Astronomy

SOFIA Program SE&I Lessons Learned NASA PM Challenge 9 February 2011 Presented by: Ron Ray - SOFIA Program SE&I Lead Laura Fobel - SOFIA Program CM & IT Lead Mike Brignola - Platform Project SE&I Lead

The purpose of this presentation is to describe the Systems Engineering solutions applied in the middle of the “troubled” SOFIA Program that helped it become successful

9 February 2011 PM Challenge 2

Table of Contents

9 February 2011 SOFIA PMR 3

Background 4

SOFIA SE&I Approach 10

Requirements Management 29

Configuration Management 41

SOFIA SE&I Summary 55

Background

Ron Ray


Why SOFIA?

• SOFIA is the largest portable telescope in the world –  2.5-meter (100-inch) telescope in a Boeing 747SP

• SOFIA is reconfigurable and highly flexible –  Every SOFIA flight series is comparable to a Hubble

reservicing mission o Science instruments can be routinely changed and upgraded

–  Able to quickly respond to all astronomical events

• SOFIA will exceed the performance of ground-based Infrared Telescopes

–  Flies above 99.9% of the water vapor

• SOFIA is designed to be productive –  140 eight-hour research flights per year; 20 year lifetime

• SOFIA cost much less than space-based observatories 11 February 2011 PM Challenge 5

Observatory Science and Mission Operations Center

Major Components of SOFIA


Telescope Assembly

Science Instruments

Aircraft Operations Center

Background

• SOFIA was established as a 80/20 partnership between the U.S. (NASA) and Germany (DLR)

– Original NASA/DLR MOU signed 1996 – Germany supplied telescope assembly and other significant

contributions – NASA supplied modified aircraft and Science Operations

Center – NASA receives 80% of available science time, DLR 20%

•  Initial program model was contractor led with NASA oversight (privatized)

• Overtime, a series of schedule slips, cost increases, contract issues and mishaps occurred


Background

• NASA withheld funding and the Program was slated for cancellation in the spring of 2006

– Members of Congress, Germans and the Science Community “pressured” NASA to continue Program

– NASA commissioned an independent review team to consider options

• The Agency approved the Program for continued funding in the fall 2006

–  The Program was restructured: o Government led, contractor supported o Program management moved to Dryden o  Two projects; Science and Platform


SOFIA Program Organization During Development


DLR Program Manager

Astrophysics Division Director

Airframe Development and Test Engineering

Airframe Development and Test Operations

Program Executive Program Analyst Program Scientist

Associate Administrator Science Mission Directorate

SOFIA Observatory IPT

Chaired by Program Chief Engineer

Project Scientist

Project Manager for Platform Project

Project Manager for Science

Early Science Mission Operations

SOFIA Program Manager Deputy Program Manager Program Office includes

Chief Engineer Office, SE&I, SMA, Program Control, and E/PO

Telescope Assembly and SI Integration

Integration IPT

Software IPT

Simulation IPT

Science & Mission Ops Development

Science Instrument Development

SOFIA SE&I Approach

Ron Ray


SOFIA Systems Engineering Transition

• The new Program Office initiated an independent review of SOFIA Systems Engineering (Summer 2006)

• The SE&I Lead position was transferred to Dryden (Sept. 2006)

– Reviewed SOFIA SE&I history & existing processes – Reviewed the SE&I independent assessment and

recommendations – Completed additional assessments

• Several significant issues with SE&I were identified (See following pages)


SOFIA SE&I Assessment

• System Requirements were lacking and fragmented – Needed government ownership and greater priority – Only a small percentage of Specifications had been

baselined –  The Interface Control Documents (ICDs) were not centrally

managed (not clear who owned what)

• Program CM process was dysfunctional (over 100 documents were tied up in the old process)

–  A small group made all of the decisions creating a bottleneck – Hardware was being built to unapproved documents


SOFIA SE&I Assessment (Cont.)

• Program had an immediate need for a formal Risk Management Process

– New PM was working this informally because the previous system was unmanageable

• The amount of information already assembled for SOFIA was vast and users had difficultly finding things on the central Data Management System

– Over 100,000 data records existed in hard and soft copy – Over 50,000 Telescope Assembly (TA) documents existed in

the Data center only in hard copy and filed chronologically – Many documents were owned and managed by the

Contractors using various document control processes


SOFIA SE Lessons Learned

It is never too late to fix Systems Engineering (SE) deficiencies


SOFIA Program Systems Engineering Dilemma

• The new Program Management faced a major dilemma with Systems Engineering:

–  Either stop and “fix” the Systems Engineering and Integration (SE&I) problems identified at the time of transition

or – Continue at risk and try to “rebuild” SE&I along the way

• Some consideration factors –  Priority was to get the aircraft from Waco to Dryden and

demonstrate progress after the threat of cancellation –  The near-term challenges were not considered as difficult as

the long-term challenges –  The new Dryden team members were still coming up to

speed on the SOFIA systems


New Implementation Strategy for SE&I

• The Program made the decision to continue at risk and “rebuild” SE&I as we go

• Risk mitigation decisions/activities –  Phase the remaining development into increments which

would give key SE activities a chance to catch-up o Add an “Early Science” Milestone to recapture schedule o Conduct both near-term and long-term SE activities

simultaneously – Work more collaboratively between the stakeholders and

developers to compensate for requirement gaps o Conduct a series of “delta” System Requirements Reviews

focusing on near-term needs and requirements o  Implement cross-Project Integrated Product Teams (IPTs)

–  Establish a new set of SE&I priorities and provide a dedicated staff to facilitate the rebuilding process


The Use of Risk Management on SOFIA

•  SOFIA distributed Program and Project level risks and made Risk Management an integrated but delegated management tool

•  The SOFIA Program focused on the “top priority” risks and tracked a larger set of “threats” (potential risks)

Risk List from March 28, 2008

17 9 February 2011 PM Challenge

SOFIA identified the lack of Requirements Definition as a risk


Breaking complex development activities into increments can improve the overall chance of success


“Sometimes the questions are complicated and the answers are simple.” Dr. Seuss

New SOFIA Life Cycle: Incremental Development

19

SRR

PDR

CDR

ΔORD

ΔPDR

ΔCDR ΔCDR ΔCDR

Flt Test

Data Review

ΔSAR ΔSAR ΔSAR

CDS/CDDS/CECS MCCS Build #1

User Need

A/C Modifica4on TA/CECS

ΔSAR

Final Upgrades

Ground Test

Original Requirements Baseline

Requirements Re-‐Baseline

ΔCDR

Status -‐ Sept 2007

Instrumenta4on CECS Improvements

Flt Test

Data Review

Ground Test

Flt Test

Data Review

Ground Test

Flt Test

Data Review

Ground Test

Decom

posi@on & Defini@on In

tegra@on

& Verifica@o

n

Fab. & Assemble

Par3ally Complete

Cavity Environmental Control System (CECS)

Decom

posi@on & Defini@on

Integra@on

& Verifica@o

n

Fab. & Assemble

A/C Modifica4on Telescope Assembly

Cavity Door Drive System (CDDS)

Science Instruments (SI)

Complete

Remaining

Decom


tegra@on

& Verifica@o

n

Fab. & Assemble

Decom


tegra@on

& Verifica@o

n

Fab. & Assemble

Cavity Door System (CDS)

Decom


tegra@on

& Verifica@o

n

Fab. & Assemble

Mission Control & Communica4ons System (MCCS)

Decom

posi@on & Defini@on

Integra@on

& Verifica@o

n

Fab. & Assemble

Status of 9 Science Instruments Varies Dependent on the Instrument

ORD

Flt Test

Data Review

SAR

Ground Test

FRR

Flt Test

Data Review ΔSAR

Ground Test

FRR

Final MCCS/CECS-‐LN2 Science Instruments

MCCS for Early Science Early Science Instruments

ΔSRR ΔSRR ΔSRR FRR FRR FRR ORR ΔSRR

Segment 0 Segment 1 Segment 2 Segment 3 Segment 4

Rework Required

ΔPDR

Advantages of the Incremental Development Life Cycle for SOFIA

20

SRR

PDR

CDR

ΔORD

ΔPDR

ΔCDR ΔCDR ΔCDR

Flt Test

Data Review

ΔSAR ΔSAR ΔSAR

CDS/CDDS/CECS MCCS Build #1

User Need

A/C Modifica4on TA/CECS

ΔSAR

Final Upgrades

Ground Test

Original Requirements Baseline

Requirements Re-‐Baseline

ΔCDR

Instrumenta4on CECS Improvements

Flt Test

Data Review

Ground Test

Flt Test

Data Review

Ground Test

Flt Test

Data Review

Ground Test

ORD

Flt Test

Data Review

SAR

Ground Test

FRR

Flt Test

Data Review ΔSAR

Ground Test

FRR

Final MCCS/CECS-‐LN2 Science Instruments

MCCS for Early Science Early Science Instruments

ΔSRR ΔSRR ΔSRR FRR FRR FRR ORR ΔSRR

Segment 0 Segment 1 Segment 2 Segment 3 Segment 4

ΔPDR

•  Allowed science data to be obtained significantly sooner helping retain science community support

•  Allowed requirements time to catch-up over the long term •  Allowed integration issues to be identified and better isolated as system

complexity grew •  Allowed for Observatory performance to be assessed earlier

–  Early 1st Light gave initial indication we have no major performance deficiencies

Original Program Lifecycle

1st Science Flight 1st Light

SOFIA SE Technical Priorities

• Organize and establish Systems Engineering leads and support teams for key SE tasks

–  Established a dedicated Requirements Manager (High Priority)

• Revise Program SE&I documents and processes –  Risk Management – IT Management –  Configuration Management – Data Management

• Develop a new Systems Engineering Management Plan (SEMP) to define technical process and requirements

–  Complies with NPR7123.1

•  Establish Program Management Control Boards –  PMB: Programmatic Control – OCCB: Observatory Control

•  Establish a SOFIA Observatory-Level IPT (SOLIPT) –  Addresses Observatory and “cross project” technical issues

•  Establish a process to manage and track the status critical Program and technical documents


SOFIA Program SE&I Organization

22

SOFIA Program Manager

System Requirements Management

Configuration Management

Data Management

IT Systems Management

Technical Planning &

Support

•  Specifications •  ICDs •  ORDs •  Database Mgt •  Req. Traceability •  V&V Tracking

•  Program PMB •  Observatory OCCB •  Platform PCB •  Science PCB •  Contractor CCBs •  CM Records

•  Program DM •  Platform Proj. DM •  Science Proj. DM •  Science Data •  SOFIA Data Center •  Export Control •  Records Retention •  Data Mgt. System

•  Program IT •  Platform Proj. IT •  Science Proj. IT •  IT Security •  System Admin. •  Account Mgt. •  Sys. Development

•  SE&I Plans •  SE&I Processes •  Technical Reviews

Observatory-Level: •  System Integration •  V&V •  Discrepancy Resolution •  Risk/Threat Mitigation

Project Chief Engineer

System-Level: •  Design Solution •  Implementation •  Integration •  Verification

SE&I Manager

Observatory-Level: •  Requirements •  Validation •  Performance

Program Chief

Engineer

Project Scientists

9 February 2011 PM Challenge


Management needs clear insight on the status of SE products


24

SOFIA Program SE&I Documentation Tracking Tool


•  Illustrates a summary chart presented to SOFIA Management to track documentation progress

See conclusion chart for more recent status

Status on 12/01/2007


SE must account for and tailor to various Center and cultural differences


“Scientists investigate that which already is; engineers create that which has never been.” Albert Einstein

The Relationship Between Engineers and Scientists

•  Engineers and Scientists must have clear and distinct roles and responsibilities

• On SOFIA (during the development phase) the Scientist is the “customer” and the Engineer is the “implementer”

–  SE&I is often the interface

26

Scientists: • Specify needs and requirements • Develop the Concept of Operations • Participate in technical reviews • Accept verification • Provide validation

Engineers: •  Interpret and decompose requirements • Conduct trade studies • Develop design & implementation strategies • Provide verification • Participate in validation

Systems Engineer: • Manages requirements •  Implements supporting processes • Establishes entrance/exit criteria for technical reviews • Maintains the V&V Matrix


“Better is the enemy of good enough”


•  Engineers want to know what are the minimum requirements so they can meet them

•  Scientists want the best they can get with no constraints: “Good enough is the enemy of the great”

Systems Engineering is an Optimization Process

• Too little or too much SE causes problems –  SE must be “value added”

• When addressing SE&I in the middle of a Program, there is never enough time, resources, and budget to complete all processes

–  SE priorities must be developed and documented but also must fit within the overall Program/Project priorities

• SOFIA used the Risk Management process to understand and accept the risks of “deliberately” leaving some things out due to schedule and budget realities


Requirements Management

Mike Brignola



Making the “Lack of Requirements Definition” a Program risk, is an effective way to highlight and address the problem


•  This allowed the Program Management to establish a long-term mitigation strategy to drive down the risk

•  SOFIA Management made a long-term commitment to correcting requirements deficiencies

Requirements Deficiency Risk Mitigating Actions

1.  Develop SE plan and process to audit, develop, and manage requirements. Implement early with adequate staff

•  Utilize Product/Specification Tree to facilitate communication •  Utilize RM Database tool to manage 4000 requirements, trace and allocate

2.  Establishing a NASA Requirements Manager with broad systems knowledge to bridge stovepipes

•  NASA is now managing and controlling the requirements •  Keep management informed, elevate issues, status reporting

3.  Establish frequent technical interchange meetings to ensure requirements definition and coordination

4.  Prioritize and baseline near-term requirements for “Early Science” 5.  Establish an Observatory Integration IPT to coordinate V&V planning and

execution between the two projects, the science instrument teams, and the international partner

6.  Complete Early Science Observatory V&V Plan 7.  Complete long-term requirements for final SOFIA configuration (including

ICDs, Specs, and Verification/Validation plans) (On-going) 9 February 2011 PM Challenge 31

Requirements Deficiency Risk Mitigation Waterfall

9 February 2011 32 PM Challenge Jan-09 Jul-09 Jan-10 Jul-10 Jan-11 Jul-11 Jan-12 0

5

10

15

20

25

• Over time, the SOFIA Requirements Deficiency Risk has been significantly reduced do to several mitigating actions


Phasing system development has bought time to establish a significantly improved set of “final” requirements


•  Valuable experience was gained accomplishing the “Early Science” Phase that will greatly benefit the final SOFIA system design

Benefits of Phasing Development on SOFIA Requirements

• The “final” SOFIA system requirements will benefit from the knowledge gained during Early Science

–  The “near-term” requirements for meeting the “Early Science” goals were less stringent but still challenging

–  Several issues with requirements definitions had to be resolved for Early Science

o  Identified gaps and misunderstandings in requirements –  SOFIA employed an “Agile Development” process (frequent

iterations with collaborative feedback) to deal with these issues o Some degree of product rework was tolerated or procedural “work

arounds” were employed to meet Customer expectations –  The development team gained valuable experience

• Phasing allowed valuable time to refine the Product Tree and systematically review “final” requirements



It takes time to become knowledgeable enough of complex systems to effectively develop “good” requirements


•  It took the new Program team a significant period of time to become proficient with the complex SOFIA systems

–  This knowledge is critical to being effective at requirement decomposing and establishing good traceability


Having a comprehensive specification/product tree (and ICD list) is critical to system integration


•  At transition, SOFIA had to deal with new “observatory-level” requirements that were inserted to address missing overall system performance values

SOFIA Spec Tree Rev B 2008 Baseline

Compare to May 2007 Original Spec Tree Rev A

Summer 2007 goal -‐ top 4 specs approval SE01-‐003 (SOFIA), SE01-‐013 (Observatory), SE01-‐004 (AircraT), SE01-‐005 (MCCS)

 Top 4 specs have been program reviewed through delta SRR, MCCS Redesign, SOLIPT, SE&I and Program assessments

37 Pg 1

38

Current Rev F Spec Tree 2010

Compare to 2008 Baselined Spec Tree

Between Jul 2007 & Dec 2010: Program + SE&I reviewed 30 Specifica4on Documents, containing over 3000 requirements

Forums: Delta SRRs, Design Reviews, IPTs, SE&I, and system assessments

Pg 1

39

SOFIA Interface Control Document Status History

Summer 2007 goal: •  Identify and list all SOFIA ICDs •  Establish initial status and

ownership

33

16

49 Total ICD’s

ICDs July 2007 ICDs Dec 2010

68 Total ICD’s

Between Jul 2007 & Dec 2010: •  Several new ICDs identified •  Several key ICDs completed

or updated

44

10

14

Approved

Not Approved

Approved

Need Update

Not Written


Requirements Management Challenges

• Availability of key personnel and conflicting priorities –  Planners = owners = implementers = testers (all the same

person)

• Requirements creep due to lack of complete/baselined requirements or well defined interfaces

• Traceability of design requirements completion status to V&V test plans/results

–  Lack of overarching program guidance and integrated test plans (No program integration office)

• Although SOFIA has made a significant amount of progress, a lot remains to get done

– Delta system level SRRs are on-going –  Striving for more formality in Segment 3 (final build)


Configuration Management, Data Management and Related Topics

Laura Fobel



To improve CM process efficiency, delegate CM responsibilities to the lowest level possible


The Delegation of Configuration Management (CM) Authority

• SOFIA developed a hierarchy of CM Boards to drive CM authority down to the lowest appropriate level

–  Improves efficiency by distributing the work load –  CM hierarchy parallels the product hierarchy

• Over time SOFIA’s CM needs changed –  Initially a single CM Board may have made sense on SOFIA –  As development work expanded and system complexity grew, a

more distributed CM process was needed

• SOFIA established a separate Control Board to manage the “Observatory” configuration

–  Includes Program, Platform and Science Project members –  Focuses on configuration management of the “integrated system”

and related discrepancies


The Delegation of CM Authority on SOFIA

44

Observatory Change Control Board (OCCB)

COTR

Platform Project Control Board (PCB)

USRA

Program Level

Project Level

Element Level

Contract Level

Science Project Control Board (PCB)

DSI MPC

COTR

L3 Com

COTR

Platform Project Office

Engineering

Science Project Office

SOFIA Program Office

SOFIA Observatory Integrated Product Team (SOLIPT)

Program Management Board (PMB)

Headquarters Science Mission Directorate

Aircraft Operations

Telescope Assembly

Science Instruments

Science Mission Ops

DLR

DLR



Informal collaboration with contractors improves the probability of success of formal deliverables

•  The distributed CM process facilitated more collaboration with contractors

Previous Organization

Informal Reviews

Formal Deliverables

Restructured Organization

Formal Deliverables

Shift From Contractor Run / Government Oversight To Government Lead / Subcontractor Relationship

MPC

USRA

L3 Com

Project Office


DLR

DFRC Platform Project Office

ARC Science Project Office


Engineering Operations Telescope Assembly

Science Instruments

Science Mission Ops

DLR

COTR

USRA MPC

COTR

L3 Com DSI

COTR


Informal Collaboration Improves Probability of Success

DLR

The Value of Informal Collaboration

• Too much back-and-forth over the fence is inefficient –  A significant amount of rework occurred when not enough

informal collaboration occurred with the contractors

•  It’s important to establish a cooperative environment with contractors

–  SOFIA applied an “agile development” process by allowing informal software builds to be delivered early in the development process to flush-out problems prior to formal deliveries

o Collaboration occurred at the lower levels and included stakeholders

o Deliverables still went through the formal acceptance process to be baselined



On SOFIA it was beneficial to have a problem reporting process that spanned informal development activities and formal acceptance testing


The Value of Problem Reporting and Discrepancy Resolution

• By establishing a Problem Reporting system early (during informal software testing) issues were identified and resolved sooner (prior to formal delivery)

–  Allowed customers to capture issues and collaborate with developers to understand and refine formal requirements

–  Supported the “Agile Development” process

• The Observatory-level control board allowed cross-Project issues to be identified and resolved jointly

– Chaired by the Program Chief Engineer o  Provided independent authority

–  Established priorities and assignments to Projects for resolving integration issues

–  Facilitated communication of issues and their resolution



The lack of a carefully designed Data Management systems hinders effective communication and collaboration


•  SOFIA team members had a difficult time finding the information they needed

– Old and obsolete data mixed with relevant data contributed to the problem

SOFIA Data Management Improvements

• Established a central repository to improve control and management of the data originating from various sources

– Reorganized the data and archived obsolete documents

• Defined data attributes for each document –  Product ID – Document number – Data retention – Export control – Owner – CM authority – Descriptive search keywords

• Considered Configuration Management, Data Management, Export Control, Records Retention, and Data Access as part of one integrated process


SOFIA Sample Records Retention Schedule


Sample SOFIA Document Attributes


File attributes facilitate Data Management, Export Control, Access Control and Records Retention processes.

• Data Management (Document Name): SOF-DA-ICD-SE03-002

• Export Control/Access Control: “Not Reviewed for Export Control”

• Records Retention: Date (2003-03-28) and Records Retention Schedule reference (03-S8-103.3)

SOFIA CM and DM Remaining Challenges

• Establishing ownership and control of all SOFIA documents and drawings

– Contractors still own important information like “models” • Shortcomings of the Data Management System

–  Search engine and user interface complexity – User familiarity – Data access by Foreign Nationals

• Catching up with Export Control and Records Retention attribute labeling


SOFIA SE&I Summary

Ron Ray



Management must set the cultural tone for the importance of SE on a Program/Project


•  The commitment the SOFIA Management Team has made to SE has helped turn around the once “troubled” Program

SOFIA Program Documentation Status as of: 12/03/2010

57

SOFIA First Light Image May 16th 2010

• SOFIA is beginning to produce outstanding science data at a fraction of the cost of comparable space based observatories

http://www.nasa.gov/mission_pages/SOFIA/

http://www.dlr.de/DesktopDefault.aspx/tabid-1/117_read-28014/

•  Image of the Orion star-formation region obtained by SOFIA compared to images obtained from ground-based telescopes

SOFIA First Science Image Dec 1st 2010

SOFIA (FORCAST) Near infrared (ESO VLT) Visible light

(ground-based)

• After almost being cancelled and a major Program structure change, the SOFIA Program operated at Risk with known Systems Engineering deficiencies

• Several important strategies were employed to mitigate the risk

–  Established a new incremental life-cycle to complete system development

–  Worked more collaboratively –  Systematically rebuilt SE&I along the way

o  Provided adequate staffing and priority –  Made correcting requirements deficiencies a high-priority –  Distributed CM authority –  Tracked and status SE Progress

• SOFIA has used Risk Management effectively to compensate for Systems Engineering deficiencies


Concluding Message

Summary of SE Lessons Learned

•  It is never too late to fix Systems Engineering (SE) deficiencies

• Breaking complex development activities into increments can improve the overall chance of success

• Management needs clear insight on the status of SE products

• SE must account for and tailor to various Center and cultural differences

• “Better is the enemy of good enough”



• Making the “Lack of Requirements Definition” a Program risk, is an effective way to highlight and address the problem

• Phasing system development has bought time to establish a significantly improved set of “final” requirements

•  It takes time to become knowledgeable enough of complex systems to effectively develop “good” requirements

• Having a comprehensive specification/product tree (and ICD list) is critical to system integration



• To improve CM process efficiency, delegate CM responsibilities to the lowest level possible

•  Informal collaboration with contractors improves the probability of success of formal deliverables

• On SOFIA it was beneficial to have a problem reporting process that spanned informal development activities and formal acceptance testing

• The lack of a carefully designed Data Management systems hinders effective communication and collaboration

• Management must set the cultural tone for the importance of SE on a Program/Project


SOFIA Program SE&I Lessons Learned SOFIA

Documents

Transcript of SOFIA Program SE&I Lessons Learned SOFIA