7/23/2019 Risk Poster Spring2014 48x42

http://slidepdf.com/reader/full/risk-poster-spring2014-48x42 1/1

Trade Study Results

Trade study results indicate that

changing the Life Cycle combination

factor and the launch indicators

were worse fits than the baseline

assumptions.

The overall conclusion of these

trade studies was that the baseline

u-curve function and role value

scheme should be replaced by:  

With this research and its associated software tool, CubeSat mission designers will be able to input mission characteristics, such as their CubeSat form factor, mass, development cycle, and launch information, in order to determine the mission risk root causes which are of

the highest concern for their mission. Historical data has been gathered from the CubeSat community and analyzed in order to provide a statistical background to characterize these Risk Estimating Relationships (RERs). The RER development uses general error regression

models to obtain the best fit relationship between root cause consequence and likelihood values and the input factors of interest. These root causes are combined into seven overall CubeSat mission risks which are then graphed on the industry-standard 5x5 Likelihood-

Consequence (L-C) chart to help mission designers quickly identify areas of concern within their mission. Decision analysis methods will be available for spacecraft designers to choose risk mitigation strategies which optimize their preferences.

CubeSat Mission Design Software Tool for Risk Estimating RelationshipsKatharine Brumbaugh Gamble

Advisor: Dr. Glenn Lightsey

References:“Agency Risk Management ProceduralRequirements,”  (2008): NASA Procedural Requirements, NPR 8000.4A.

Brumbaugh, K.M., Lightsey, E.G., "Systematic Approach to Risk Management for Small Satellites," Journal of Small Satellites. JoSS, Vol.

2, No. 1, p. 147-160, July 2013.

Snee, R. (1977). "Validation of Regression Models: Methods and Examples." Technometrics Vol. 19 (No. 4).

Stone, M. (1974). Cross-validating choice and assessment of statistical predictions (with discussion). J. Roy. Statist. Soc. B, 36, 111-147.

What is Risk?

“A future event with a negative consequence that has some probability

of occurring. A risk can also be defined as the potential for performance

shortfalls . A risk poses a threat to the spacecraft safety, program cost,

schedule, or major mission objective. An item whose resolution is

unlikely without focused management attention.” (NASA, 2008) 

What is Risk Management?

•Risk Management has been applied on larger s/c

o Much too detailed, too time intensive for low-cost missions

o Often hard to get access to risk database

•Basic Risk Management Plan follows these steps (Brumbaugh, 2013):

o  Risk Identification

o  Determine Mitigation Techniques

o  Track Progress

•Risk is typically shown on a 5x5 L-C chart

o The Good: Quick and visual explanation of spacecraft status,

especially with tracking

o The Bad: Likelihood and Consequence values are subjectively

determined based on the systems engineer’s perspective of

things.Image credit: Wikipedia

Mathematical Background – Regression Analysis

•Given a set of data, regression analysis finds the

line of best fit to describe the data

•Regression techniques include:

o Ordinary Least Squares

Traditionally used for linear models and

additive models

Minimizes square standard erroro General Error Regression Techniques:

Can use additive for or multiplicative

error models.

Minimum Percentage Error (MPE)

Iterated Least Squares / Minimum

Unbiased Percentage Error (IRLS / MUPE)

Minimum Percentage Error – Zero

Percentage Bias (MPE-ZPE)

Background Research Plan

Choosing Risk Functions

Once the data has been processed, it can be run through the MinimumPercentage Error – Zero Percentage Bias (MPE-ZPB) General Error

Regression (GER) technique to obtain the 68 risk estimating relationships.

The risk function forms were chosen based on engineering judgment and

experience with cost estimating relationships. The primary factors of

interest are form factor (ff), whether the mission has launched (launch),

and the time in each life cycle phase – development (dev), integration (int),

spacecraft functional testing (scfunc), environmental testing (environ),

waiting for launch (wait), and operations (ops).

Verification & Validation

Trade studies were completed to show that assumptions made in thedevelopment of the Risk Estimating Relationships (RERs) were reasonable.

Four trade studies were completed by changing the:

Risk Analysis Software Tool

This inputs page is considered a necessity for the development of the riskmanagement tool in an effort to make the tool as user-friendly as possible.

The mission designer simply has to input their form factor, mass, select a

launch option from four choices, input their launch date, and input the

months in development, integration, functional testing, environmental

testing, awaiting launch, and operations. The internal calculations of the

tool will then output the root cause likelihood and consequence values on

the outputs page. The user also has the ability to track the spacecraft risks

at multiple milestones by indicating on the Options bar to which milestone

the current inputs correspond.

Data Gathering & Processing

The data then needed to be processed before being run through regression

techniques. This processing included:

1. Combining expert opinions through the use of a u-curve function and

the evaluation of the respondent’s role on the mission. 

2. Combining life cycle duration estimates based on predicated and actual

values

Acknowledgements: •Funded by the National Defense Science and Engineering Graduate Fellowship (NDSEG) from the Department of Defense.

•Texas Spacecraft Lab and all the team members for their hard work and dedication.

•The Aerospace Engineering Department for their continued support

•Dr. Glenn Lightsey for his continued support and attitude as a great advisor.

Future Work: The next, and final, phase of this research is to generate the risk acceptance and decision analysis methodology. That is,

what should the user do once the risk has been identified and given a likelihood and consequence value? Based on a

historical database of lessons learned and mitigation techniques, the user will be able to select the method by which they

hope to increase their likelihood of mission success.

CubeSat Mission

Risk Survey

Response 1

Years

experience

Role on

team

U-value

Response n

Years

experience

Role on

team

U-value

Normalize U-values to become weights

Weighted average of root cause response

values and life cycle estimates become

combined response

Remove duplicate mission entries in

matrices/vectors, add combined response

Mass

Form

factor

Launch

choice

Launch

date

Processedlife cycle

estimates

Parameter Input  Actual or

Predicted? Description

Form factor 3

Entera numericvaluecorresponding to thenumberof U's yourspacecraft design uses (e.g. 3U

would beentered as "3")

Mass 4 Entera numericvalueof themass limit (in kg)

Launched?

No, but we have alaunch promised

(ELaNaorsimilar)

Select an answerusing thedrop-down menu: Yes, thes/chas launched; No, but we'vebeen

manifested; No, but wehavea launch promised (ELaNa orsimilar); No, we havenot been

manifested orgiven a promiseof a launch

Launch Date 2014

Givethedateof thelaunch; If thes/c has yet to belaunched, givetheprojected date. (Can bein

MM/DD/YYYYorMM/YYYYformat)

Months in

Development 24Actual

Entera numericvaluecorresponding to thenumberof months in s/cdesign and development,

including everything up untilflight integration; Indicatewhetherthis valueis actualorpredicted 

Months in Integration 6Actual

Entera numericvaluecorresponding to thenumberof months taken fors/cintegration; Indicate

whetherthis valueis actualorpredicted 

Months in S/C

Functional Testing 4Actual

Entera numericvaluecorresponding to thenumberof months spent on integrated s/ctesting at

theorganization level, including functionaltesting; Indicatewhetherthis valueis actualor 

 predicted 

Months in S/C

Environmental Testing 6Actual

Entera numericvaluecorresponding to thenumberof months spent on necessary testing to

satisfy launch providerrequirements (usually includes thermalvac, vib tables, and mass

 properties testing); Indicatewhetherthis valueis actualorpredicted 

Months S/Cis awaiting

launch 1Actual

Entera numericvaluecorresponding to thenumberof months thespacecraft was "on theshelf"

waiting forlaunch afteralltesting had been completed; Indicatewhetherthis valueis actualor 

 predicted 

Months S/Cis in

operations 12Predicted

Entera numericvaluecorresponding to thenumberof months thespacecraft was operationalin

orbit;Indicatewhetherthis valueis actualorpredicted 

Milestone LVINT Enterthenameof themilestoneforwhich thesenumbers reflect thestatus

Options:

CalculateL-Cvalues

for Milestone1

CalculateL-Cvalues

for Milestone2

CalculateL-Cvalues

for Milestone3

Current factors of interest

in regression analysis

Macro buttons will calculate the L-Cvalues for multiple milestones

Life cycle values may not

be based on experience, but

could be predicted

Missio nRisk Ro o tCause

Consequence

value

Likelihood

value

Consequence

value

Likelihood

value

Consequence

value

Likelihood

valueSchedule   3. 31804782 4. 487230123 3. 290284849 4. 471377851 3. 17733186 4.535742296

1. Inability tofind desired spacecraft

components   2. 212231943 4. 352609117   2. 212231943 4. 392936827 2. 212231943 4. 557244851

2. Mechanical design delays (such as

issues with the CADordrawings)   2. 576567724 4. 409706951   2. 498982973 4. 362327041 2. 220156905 4. 534473471

3. Software design delays (such as basic

component functionality orembedded

codingissues)   3. 941404598 4.774450701   3. 869117987 4. 699644896 3. 431920442 4. 693574931

4. Delay due toissuse with payload

provider(may be related todelivery of

EDUorflight unit, documentation, or

interface issues)   3. 564926097 4. 242310987   3. 564926097 4. 293131741 3. 564926097 4. 405439749

5. Delay due toinadequate

documentation   2. 202138245 3. 960906613   2. 375918631 4. 055460527 3. 130304667 4. 242332588

Payload   3. 347989002 4. 478869868 3. 3175176 4. 538279758 3. 435401292 4. 709801103

1. Software interface issues between

payload and spacecraft bus   3. 319286913 4. 716094717   3. 021747664 4. 713521567 2. 931091293 4. 853501468

2. Hardware/electrical interface issues

between payload and spacecraft bus   2. 9451276 4. 457863799   2. 924689458 4. 58150592 3. 148496586 4. 757762118

3. Payload malfunction due tomechanical

issues   3. 100044056 4. 267186994   3. 349764832 4. 422110122 3. 684628181 4. 678684548

4. Payload malfunction due tosoftware

issues   3. 545703713 4. 436069919   3. 498190998 4. 527144099 3. 380372317 4. 66821002

M il es to ne 1 M il es to ne 2 M il es to ne 3

Root causes calculatedvia VBA-programmed

functions

Mission risk L-C values calculatedvia rank reciprocal weighting

scheme (see JoSSpaper)

Currently up to 3 milestones can betracked at one time

Milestone

1 C L

SCH 3 . 32 4 . 49

PER 2 . 65 4 . 46

PAY 3 . 35 4 . 48

SC-1 2 . 6 4 . 56

SC-2 2 .9 4 .4

SC-3 2 . 58 4 . 46

COST 3 . 16 4 . 85

C L C L

SCH 3 . 29 4 . 47 1   SCH 3.177 4.536

      1

      2

      3

      4

      L      I      K      E      L      I      H      O      O      D

      5

M il es to ne 2 M il es to ne 3

4 5

CONSEQUENCE

1 2 3

Select missionriskto

bringtofront onplot:

Whichmilestones

wouldyoulike toplot?

P E R P A YSC1

COST

Delete all riskshapes

Plotall risks

SCH

SC1

Milestone 1

PER

SC2

Milestone 2

PAY

SC3

Milestone 3

COST All Miles tones

PLOT!

 Ability to choose which risks andmilestones to plot

Rank reciprocal weighted missionrisks (same as on outputs page)

 Ability to bring desired risk to front

 Ability to clear plot

 Ability to plot multiplemilestones (all risks)

Once the option to

calculate the L-C

values is selected, a

VBA program takes

the user inputs and

calculates the

consequence and

likelihood values for

each root cause

based on the

formulas

established from

the regression

techniques.

The plots page

provides a quick

and easy way to

view the mission

risks on a

traditional L-C

chart. The mission

risk consequence is

plotted on the x-

axis while the

likelihood is plotted

on the y-axis.

Excel macro calculatescombined rank and outputsbest function

Minimize SSD whilekeeping Bias = 0

General R^2 value allowscomparison between any model

(not just linear)

The MPE-ZPB technique calls for minimizing the

Sum of Squared Deviations (SSD) while keeping

the bias (BM) equal to zero. Excel Solver was used

to accomplish this task while also comparing the

generalized R2 value for each function form under

consideration.

   

  

   

  n

ii

ii M 

a x f 

a x f  ySSD

1

2

2

),(

),(

   

  

   

  n

ii

ii M 

a x f 

 ya x f 

n B

1 ),(

),(1

})(}{)({

}))(({2222

22

iiii

iiii

 y yn x xn

 y x y xn R

Risk Estimating Relationship function forms:

(1) L1 = a + b*ff

(2) L2 = a + b*ff + cc*dev + d*int + e*scfunc + f*environ + g*wait + h*ops

(3) L3 = a + b*ff + cc*launch

(4) L4 = a + b*launch

(5) L5 = a + b*dev + cc*int + d*scfunc + e*environ + f*wait + g*ops

(6) E1 = a + b*cc^ff

(7) E2 = a + b*cc^launch

(8) E3 = a + b*cc^(dev+int+scfunc+environ+wait+ops)

(9) T1 = a + b*ff^cc

(10) T2 = a + b*dev^cc + d*int^e + f*scfunc^g + h*environ^I + j*wait^k + l*ops^m

(11) T3 = a + b*launch^cc

(12) T4 = a + b*ff^cc + d*launch^e

Model Numberof RCs <0 % RCs <0 TOTAvg<0 Avgof Avg

uTrial_1 5 1 7 5. 00 0% 1 4 - 1 .4 45 22 98 64

uTrial_2 4 3 6 3. 23 5% 1 2 - 0 .3 67 80 49 57

uTrial_3 4 4 6 4. 70 6% 1 0 - 0. 53 98 74 02

uTrial_4 4 7 6 9. 11 8% 1 2 - 0 .8 35 21 90 03

uTrial_5 5 1 7 5. 00 0% 1 1 - 1. 02 68 41 92

rTrial_1 43 63. 235% 9 - 0. 29767468

rTrial_2 4 2 6 1. 76 5% 1 0 - 0 .6 96 29 93 16

rTrial_3 4 6 6 7. 64 7% 1 1 - 0 .5 62 52 90 72

lcTrial_1 26 38.235% 7 0.3652918

lcTrial_2 1 7 2 5. 00 0% 2 1 .7 28 000 38 4

lcTrial_3 1 6 2 3. 52 9% 2 2 .2 07 289 73 8

lcTrial_4 1 9 2 7. 94 1% 4 1 .9 11 504 42 1

laTrial_1 1 1.471% 1 0.424133482

laTrial_2 15 22. 059% 7 - 0. 44827084

laTrial_3 4 5.882% 2 0.376422029

urTrial_1 5 4 7 9. 41 2% 1 3 - 1 .4 64 24 36 42

urTrial_2 5 1 7 5. 00 0% 1 2 - 1 .1 17 28 01 53

1. U-curve function (uTrial)

2. Role numerical values (rTrial)

3. Life cycle combination factor

(lcTrial)

4. Launch indicator numerical

mapping values (laTrial)

Role Weight

Principal

Investigator

0.5

P ro gr am Ma na ge r 1

S ys te ms E n gi ne er 0 .7 5

C hi ef E ng in ee r 0 .7 5

SubsystemLead 0.25

T ea m me mb er 0 .0 5 (f or e ac h

subsystem)

6/14/1),(   role yrsrole yrsu  

 

Leave-Out-One Model

Validation (Stone, 1974) was

used to validate the functions

chosen in the GER MPE-ZPB

algorithm. This model validation

involves:

The method shows that a

majority of the cases were

accurate within +/- 50% for both

the Consequence and Likelihood

calculations. It is concluded that

the model is validated.

The model equations were also

validated by moving outside the

data range (Snee, 1977) and the

results indicate a high correlation

of a decreased input valueleading to large deviations from

expected values. Increasing

values, though, rarely cause large

variations in output values. It is

concluded that this is not an

issue and the user will be made

aware of the data ranges and

the danger of moving outside of

the limits. 

Range C L

(+ /- ) 5 0 % 6 4 .0 4 6% 7 0 .6 4 8%

(+ /- ) 1 0 0 % 8 0 .4 9 2% 7 6 .5 3 1%

(+ /- ) 1 5 0 % 9 1 .1 1 6% 8 1 .8 7 3%

To have the statistical analysis based onhistorical data, it was necessary to

collect data from the C ubeSat

community through the use of an

online survey form. These results were

collected over the course of 8 months,

with a total of 66 responses.

Demographic analysis of the

results showed that mostly

university missions completed

the survey, with CubeSats of

the 3U form factor.

Additionally, the respondents

tended to have 0-5 years

experience in building

spacecraft.

CubeSat Mission Risk Survey

April 2013 – November 2013:

Gathered data

December 2013:

High-level demographic

analysis of data

Risk Estimating Relationships

September 2013:

General Error Regression using

MPE-ZPB technique to

determine risk function form

December 2013:

Trade studies and model

validation of chosen risk

function form

Data processing

techniques

September 2013:

Initial algorithms

completed with

artificial data

December 2013:

Algorithms run on

actual data

CubeSat Risk Analysis Tool

September 2013:

Macro-enabled Excel interface

with inputs, outputs, and plots

pages

April 2014:

Release to CubeSat Community

for feedback

Spring 2014:

Establish risk acceptance

techniques

Summer 2014:

Incorporate feedback and

develop risk mitigation database

      1

      2

      3

      4

      L      I      K      E      L      I      H      O      O      D

      5

4 5

CONSEQUENCE

1 2 3

i i l st s

u l u l i t l t

P E R P A Y

S C1

COST

lotall risks

SCH

E

Y

CST

Inputs page

Outputs page

Plots page

0

5

10

15

20

25

30

35

40

0 2.5 5

Modified response values

Modified Conseq Responsefor Cases

Case 14 Conseq

Case 13 Conseq

Case 12 Conseq

Case 11 Conseq

Case 10 Conseq

Case 9 Conseq

Case 8 Conseq

Case 7 Conseq

Case 6 Conseq

Case 5 Conseq

Case 4 Conseq

Case 3 Conseq

Case 2 Conseq

Case 1 Conseq

0

5

10

15

20

25

30

35

40

45

0 2.5 5

Modified response values

Modified Likel Responsefor Cases

Case 14 Conseq

Case 13 Conseq

Case 12 Conseq

Case 11 Conseq

Case 10 Conseq

Case 9 Conseq

Case 8 Conseq

Case 7 Conseq

Case 6 Conseq

Case 5 Conseq

Case 4 Conseq

Case 3 Conseq

Case 2 Conseq

Case 1 Conseq

What is a CubeSat?

•California Polytechnic State University (Cal Poly)

established a standard launch mechanism called

the Poly-Picosatellite Orbital Deployer (P-POD)

•The P-POD holds 10 cm cubed satellites – called

1-Unit (1U) CubeSats

•Common configurations are in 1U, 2U, 3U, and

now 6U and even 12U•The first six CubeSats were deployed from a P-

POD in June 2003

•CubeSats fly as secondary cargo aboard any

launch available, depending on the mission

requirements for orbit parameters

Bevo-2 (left) and RACE (right)

3U CubeSats at UT-Austin

Typical 5x5 L-C chart

Example regression line

High-level demographic

analysis of survey results

Combining

experts

diagram

Example GER spreadsheet

This plot shows the

RERs have a strong

preference for

function form 10

Moving Outside Data Range Results