Risk Poster Spring2014 48x42
Transcript of Risk Poster Spring2014 48x42
![Page 1: Risk Poster Spring2014 48x42](https://reader036.fdocuments.us/reader036/viewer/2022081810/5695d0f61a28ab9b02949863/html5/thumbnails/1.jpg)
7/23/2019 Risk Poster Spring2014 48x42
http://slidepdf.com/reader/full/risk-poster-spring2014-48x42 1/1
Trade Study Results
Trade study results indicate that
changing the Life Cycle combination
factor and the launch indicators
were worse fits than the baseline
assumptions.
The overall conclusion of these
trade studies was that the baseline
u-curve function and role value
scheme should be replaced by:
With this research and its associated software tool, CubeSat mission designers will be able to input mission characteristics, such as their CubeSat form factor, mass, development cycle, and launch information, in order to determine the mission risk root causes which are of
the highest concern for their mission. Historical data has been gathered from the CubeSat community and analyzed in order to provide a statistical background to characterize these Risk Estimating Relationships (RERs). The RER development uses general error regression
models to obtain the best fit relationship between root cause consequence and likelihood values and the input factors of interest. These root causes are combined into seven overall CubeSat mission risks which are then graphed on the industry-standard 5x5 Likelihood-
Consequence (L-C) chart to help mission designers quickly identify areas of concern within their mission. Decision analysis methods will be available for spacecraft designers to choose risk mitigation strategies which optimize their preferences.
CubeSat Mission Design Software Tool for Risk Estimating RelationshipsKatharine Brumbaugh Gamble
Advisor: Dr. Glenn Lightsey
References:“Agency Risk Management ProceduralRequirements,” (2008): NASA Procedural Requirements, NPR 8000.4A.
Brumbaugh, K.M., Lightsey, E.G., "Systematic Approach to Risk Management for Small Satellites," Journal of Small Satellites. JoSS, Vol.
2, No. 1, p. 147-160, July 2013.
Snee, R. (1977). "Validation of Regression Models: Methods and Examples." Technometrics Vol. 19 (No. 4).
Stone, M. (1974). Cross-validating choice and assessment of statistical predictions (with discussion). J. Roy. Statist. Soc. B, 36, 111-147.
What is Risk?
“A future event with a negative consequence that has some probability
of occurring. A risk can also be defined as the potential for performance
shortfalls . A risk poses a threat to the spacecraft safety, program cost,
schedule, or major mission objective. An item whose resolution is
unlikely without focused management attention.” (NASA, 2008)
What is Risk Management?
•Risk Management has been applied on larger s/c
o Much too detailed, too time intensive for low-cost missions
o Often hard to get access to risk database
•Basic Risk Management Plan follows these steps (Brumbaugh, 2013):
o Risk Identification
o Determine Mitigation Techniques
o Track Progress
•Risk is typically shown on a 5x5 L-C chart
o The Good: Quick and visual explanation of spacecraft status,
especially with tracking
o The Bad: Likelihood and Consequence values are subjectively
determined based on the systems engineer’s perspective of
things.Image credit: Wikipedia
Mathematical Background – Regression Analysis
•Given a set of data, regression analysis finds the
line of best fit to describe the data
•Regression techniques include:
o Ordinary Least Squares
Traditionally used for linear models and
additive models
Minimizes square standard erroro General Error Regression Techniques:
Can use additive for or multiplicative
error models.
Minimum Percentage Error (MPE)
Iterated Least Squares / Minimum
Unbiased Percentage Error (IRLS / MUPE)
Minimum Percentage Error – Zero
Percentage Bias (MPE-ZPE)
Background Research Plan
Choosing Risk Functions
Once the data has been processed, it can be run through the MinimumPercentage Error – Zero Percentage Bias (MPE-ZPB) General Error
Regression (GER) technique to obtain the 68 risk estimating relationships.
The risk function forms were chosen based on engineering judgment and
experience with cost estimating relationships. The primary factors of
interest are form factor (ff), whether the mission has launched (launch),
and the time in each life cycle phase – development (dev), integration (int),
spacecraft functional testing (scfunc), environmental testing (environ),
waiting for launch (wait), and operations (ops).
Verification & Validation
Trade studies were completed to show that assumptions made in thedevelopment of the Risk Estimating Relationships (RERs) were reasonable.
Four trade studies were completed by changing the:
Risk Analysis Software Tool
This inputs page is considered a necessity for the development of the riskmanagement tool in an effort to make the tool as user-friendly as possible.
The mission designer simply has to input their form factor, mass, select a
launch option from four choices, input their launch date, and input the
months in development, integration, functional testing, environmental
testing, awaiting launch, and operations. The internal calculations of the
tool will then output the root cause likelihood and consequence values on
the outputs page. The user also has the ability to track the spacecraft risks
at multiple milestones by indicating on the Options bar to which milestone
the current inputs correspond.
Data Gathering & Processing
The data then needed to be processed before being run through regression
techniques. This processing included:
1. Combining expert opinions through the use of a u-curve function and
the evaluation of the respondent’s role on the mission.
2. Combining life cycle duration estimates based on predicated and actual
values
Acknowledgements: •Funded by the National Defense Science and Engineering Graduate Fellowship (NDSEG) from the Department of Defense.
•Texas Spacecraft Lab and all the team members for their hard work and dedication.
•The Aerospace Engineering Department for their continued support
•Dr. Glenn Lightsey for his continued support and attitude as a great advisor.
Future Work: The next, and final, phase of this research is to generate the risk acceptance and decision analysis methodology. That is,
what should the user do once the risk has been identified and given a likelihood and consequence value? Based on a
historical database of lessons learned and mitigation techniques, the user will be able to select the method by which they
hope to increase their likelihood of mission success.
CubeSat Mission
Risk Survey
Response 1
Years
experience
Role on
team
U-value
Response n
Years
experience
Role on
team
U-value
Normalize U-values to become weights
Weighted average of root cause response
values and life cycle estimates become
combined response
Remove duplicate mission entries in
matrices/vectors, add combined response
Mass
Form
factor
Launch
choice
Launch
date
Processedlife cycle
estimates
Parameter Input Actual or
Predicted? Description
Form factor 3
Entera numericvaluecorresponding to thenumberof U's yourspacecraft design uses (e.g. 3U
would beentered as "3")
Mass 4 Entera numericvalueof themass limit (in kg)
Launched?
No, but we have alaunch promised
(ELaNaorsimilar)
Select an answerusing thedrop-down menu: Yes, thes/chas launched; No, but we'vebeen
manifested; No, but wehavea launch promised (ELaNa orsimilar); No, we havenot been
manifested orgiven a promiseof a launch
Launch Date 2014
Givethedateof thelaunch; If thes/c has yet to belaunched, givetheprojected date. (Can bein
MM/DD/YYYYorMM/YYYYformat)
Months in
Development 24Actual
Entera numericvaluecorresponding to thenumberof months in s/cdesign and development,
including everything up untilflight integration; Indicatewhetherthis valueis actualorpredicted
Months in Integration 6Actual
Entera numericvaluecorresponding to thenumberof months taken fors/cintegration; Indicate
whetherthis valueis actualorpredicted
Months in S/C
Functional Testing 4Actual
Entera numericvaluecorresponding to thenumberof months spent on integrated s/ctesting at
theorganization level, including functionaltesting; Indicatewhetherthis valueis actualor
predicted
Months in S/C
Environmental Testing 6Actual
Entera numericvaluecorresponding to thenumberof months spent on necessary testing to
satisfy launch providerrequirements (usually includes thermalvac, vib tables, and mass
properties testing); Indicatewhetherthis valueis actualorpredicted
Months S/Cis awaiting
launch 1Actual
Entera numericvaluecorresponding to thenumberof months thespacecraft was "on theshelf"
waiting forlaunch afteralltesting had been completed; Indicatewhetherthis valueis actualor
predicted
Months S/Cis in
operations 12Predicted
Entera numericvaluecorresponding to thenumberof months thespacecraft was operationalin
orbit;Indicatewhetherthis valueis actualorpredicted
Milestone LVINT Enterthenameof themilestoneforwhich thesenumbers reflect thestatus
Options:
CalculateL-Cvalues
for Milestone1
CalculateL-Cvalues
for Milestone2
CalculateL-Cvalues
for Milestone3
Current factors of interest
in regression analysis
Macro buttons will calculate the L-Cvalues for multiple milestones
Life cycle values may not
be based on experience, but
could be predicted
Missio nRisk Ro o tCause
Consequence
value
Likelihood
value
Consequence
value
Likelihood
value
Consequence
value
Likelihood
valueSchedule 3. 31804782 4. 487230123 3. 290284849 4. 471377851 3. 17733186 4.535742296
1. Inability tofind desired spacecraft
components 2. 212231943 4. 352609117 2. 212231943 4. 392936827 2. 212231943 4. 557244851
2. Mechanical design delays (such as
issues with the CADordrawings) 2. 576567724 4. 409706951 2. 498982973 4. 362327041 2. 220156905 4. 534473471
3. Software design delays (such as basic
component functionality orembedded
codingissues) 3. 941404598 4.774450701 3. 869117987 4. 699644896 3. 431920442 4. 693574931
4. Delay due toissuse with payload
provider(may be related todelivery of
EDUorflight unit, documentation, or
interface issues) 3. 564926097 4. 242310987 3. 564926097 4. 293131741 3. 564926097 4. 405439749
5. Delay due toinadequate
documentation 2. 202138245 3. 960906613 2. 375918631 4. 055460527 3. 130304667 4. 242332588
Payload 3. 347989002 4. 478869868 3. 3175176 4. 538279758 3. 435401292 4. 709801103
1. Software interface issues between
payload and spacecraft bus 3. 319286913 4. 716094717 3. 021747664 4. 713521567 2. 931091293 4. 853501468
2. Hardware/electrical interface issues
between payload and spacecraft bus 2. 9451276 4. 457863799 2. 924689458 4. 58150592 3. 148496586 4. 757762118
3. Payload malfunction due tomechanical
issues 3. 100044056 4. 267186994 3. 349764832 4. 422110122 3. 684628181 4. 678684548
4. Payload malfunction due tosoftware
issues 3. 545703713 4. 436069919 3. 498190998 4. 527144099 3. 380372317 4. 66821002
M il es to ne 1 M il es to ne 2 M il es to ne 3
Root causes calculatedvia VBA-programmed
functions
Mission risk L-C values calculatedvia rank reciprocal weighting
scheme (see JoSSpaper)
Currently up to 3 milestones can betracked at one time
Milestone
1 C L
SCH 3 . 32 4 . 49
PER 2 . 65 4 . 46
PAY 3 . 35 4 . 48
SC-1 2 . 6 4 . 56
SC-2 2 .9 4 .4
SC-3 2 . 58 4 . 46
COST 3 . 16 4 . 85
C L C L
SCH 3 . 29 4 . 47 1 SCH 3.177 4.536
1
2
3
4
L I K E L I H O O D
5
M il es to ne 2 M il es to ne 3
4 5
CONSEQUENCE
1 2 3
Select missionriskto
bringtofront onplot:
Whichmilestones
wouldyoulike toplot?
P E R P A YSC1
COST
Delete all riskshapes
Plotall risks
SCH
SC1
Milestone 1
PER
SC2
Milestone 2
PAY
SC3
Milestone 3
COST All Miles tones
PLOT!
Ability to choose which risks andmilestones to plot
Rank reciprocal weighted missionrisks (same as on outputs page)
Ability to bring desired risk to front
Ability to clear plot
Ability to plot multiplemilestones (all risks)
Once the option to
calculate the L-C
values is selected, a
VBA program takes
the user inputs and
calculates the
consequence and
likelihood values for
each root cause
based on the
formulas
established from
the regression
techniques.
The plots page
provides a quick
and easy way to
view the mission
risks on a
traditional L-C
chart. The mission
risk consequence is
plotted on the x-
axis while the
likelihood is plotted
on the y-axis.
Excel macro calculatescombined rank and outputsbest function
Minimize SSD whilekeeping Bias = 0
General R^2 value allowscomparison between any model
(not just linear)
The MPE-ZPB technique calls for minimizing the
Sum of Squared Deviations (SSD) while keeping
the bias (BM) equal to zero. Excel Solver was used
to accomplish this task while also comparing the
generalized R2 value for each function form under
consideration.
n
ii
ii M
a x f
a x f ySSD
1
2
2
),(
),(
n
ii
ii M
a x f
ya x f
n B
1 ),(
),(1
})(}{)({
}))(({2222
22
iiii
iiii
y yn x xn
y x y xn R
Risk Estimating Relationship function forms:
(1) L1 = a + b*ff
(2) L2 = a + b*ff + cc*dev + d*int + e*scfunc + f*environ + g*wait + h*ops
(3) L3 = a + b*ff + cc*launch
(4) L4 = a + b*launch
(5) L5 = a + b*dev + cc*int + d*scfunc + e*environ + f*wait + g*ops
(6) E1 = a + b*cc^ff
(7) E2 = a + b*cc^launch
(8) E3 = a + b*cc^(dev+int+scfunc+environ+wait+ops)
(9) T1 = a + b*ff^cc
(10) T2 = a + b*dev^cc + d*int^e + f*scfunc^g + h*environ^I + j*wait^k + l*ops^m
(11) T3 = a + b*launch^cc
(12) T4 = a + b*ff^cc + d*launch^e
Model Numberof RCs <0 % RCs <0 TOTAvg<0 Avgof Avg
uTrial_1 5 1 7 5. 00 0% 1 4 - 1 .4 45 22 98 64
uTrial_2 4 3 6 3. 23 5% 1 2 - 0 .3 67 80 49 57
uTrial_3 4 4 6 4. 70 6% 1 0 - 0. 53 98 74 02
uTrial_4 4 7 6 9. 11 8% 1 2 - 0 .8 35 21 90 03
uTrial_5 5 1 7 5. 00 0% 1 1 - 1. 02 68 41 92
rTrial_1 43 63. 235% 9 - 0. 29767468
rTrial_2 4 2 6 1. 76 5% 1 0 - 0 .6 96 29 93 16
rTrial_3 4 6 6 7. 64 7% 1 1 - 0 .5 62 52 90 72
lcTrial_1 26 38.235% 7 0.3652918
lcTrial_2 1 7 2 5. 00 0% 2 1 .7 28 000 38 4
lcTrial_3 1 6 2 3. 52 9% 2 2 .2 07 289 73 8
lcTrial_4 1 9 2 7. 94 1% 4 1 .9 11 504 42 1
laTrial_1 1 1.471% 1 0.424133482
laTrial_2 15 22. 059% 7 - 0. 44827084
laTrial_3 4 5.882% 2 0.376422029
urTrial_1 5 4 7 9. 41 2% 1 3 - 1 .4 64 24 36 42
urTrial_2 5 1 7 5. 00 0% 1 2 - 1 .1 17 28 01 53
1. U-curve function (uTrial)
2. Role numerical values (rTrial)
3. Life cycle combination factor
(lcTrial)
4. Launch indicator numerical
mapping values (laTrial)
Role Weight
Principal
Investigator
0.5
P ro gr am Ma na ge r 1
S ys te ms E n gi ne er 0 .7 5
C hi ef E ng in ee r 0 .7 5
SubsystemLead 0.25
T ea m me mb er 0 .0 5 (f or e ac h
subsystem)
6/14/1),( role yrsrole yrsu
Leave-Out-One Model
Validation (Stone, 1974) was
used to validate the functions
chosen in the GER MPE-ZPB
algorithm. This model validation
involves:
The method shows that a
majority of the cases were
accurate within +/- 50% for both
the Consequence and Likelihood
calculations. It is concluded that
the model is validated.
The model equations were also
validated by moving outside the
data range (Snee, 1977) and the
results indicate a high correlation
of a decreased input valueleading to large deviations from
expected values. Increasing
values, though, rarely cause large
variations in output values. It is
concluded that this is not an
issue and the user will be made
aware of the data ranges and
the danger of moving outside of
the limits.
Range C L
(+ /- ) 5 0 % 6 4 .0 4 6% 7 0 .6 4 8%
(+ /- ) 1 0 0 % 8 0 .4 9 2% 7 6 .5 3 1%
(+ /- ) 1 5 0 % 9 1 .1 1 6% 8 1 .8 7 3%
To have the statistical analysis based onhistorical data, it was necessary to
collect data from the C ubeSat
community through the use of an
online survey form. These results were
collected over the course of 8 months,
with a total of 66 responses.
Demographic analysis of the
results showed that mostly
university missions completed
the survey, with CubeSats of
the 3U form factor.
Additionally, the respondents
tended to have 0-5 years
experience in building
spacecraft.
CubeSat Mission Risk Survey
April 2013 – November 2013:
Gathered data
December 2013:
High-level demographic
analysis of data
Risk Estimating Relationships
September 2013:
General Error Regression using
MPE-ZPB technique to
determine risk function form
December 2013:
Trade studies and model
validation of chosen risk
function form
Data processing
techniques
September 2013:
Initial algorithms
completed with
artificial data
December 2013:
Algorithms run on
actual data
CubeSat Risk Analysis Tool
September 2013:
Macro-enabled Excel interface
with inputs, outputs, and plots
pages
April 2014:
Release to CubeSat Community
for feedback
Spring 2014:
Establish risk acceptance
techniques
Summer 2014:
Incorporate feedback and
develop risk mitigation database
1
2
3
4
L I K E L I H O O D
5
4 5
CONSEQUENCE
1 2 3
i i l st s
u l u l i t l t
P E R P A Y
S C1
COST
lotall risks
SCH
E
Y
CST
Inputs page
Outputs page
Plots page
0
5
10
15
20
25
30
35
40
0 2.5 5
Modified response values
Modified Conseq Responsefor Cases
Case 14 Conseq
Case 13 Conseq
Case 12 Conseq
Case 11 Conseq
Case 10 Conseq
Case 9 Conseq
Case 8 Conseq
Case 7 Conseq
Case 6 Conseq
Case 5 Conseq
Case 4 Conseq
Case 3 Conseq
Case 2 Conseq
Case 1 Conseq
0
5
10
15
20
25
30
35
40
45
0 2.5 5
Modified response values
Modified Likel Responsefor Cases
Case 14 Conseq
Case 13 Conseq
Case 12 Conseq
Case 11 Conseq
Case 10 Conseq
Case 9 Conseq
Case 8 Conseq
Case 7 Conseq
Case 6 Conseq
Case 5 Conseq
Case 4 Conseq
Case 3 Conseq
Case 2 Conseq
Case 1 Conseq
What is a CubeSat?
•California Polytechnic State University (Cal Poly)
established a standard launch mechanism called
the Poly-Picosatellite Orbital Deployer (P-POD)
•The P-POD holds 10 cm cubed satellites – called
1-Unit (1U) CubeSats
•Common configurations are in 1U, 2U, 3U, and
now 6U and even 12U•The first six CubeSats were deployed from a P-
POD in June 2003
•CubeSats fly as secondary cargo aboard any
launch available, depending on the mission
requirements for orbit parameters
Bevo-2 (left) and RACE (right)
3U CubeSats at UT-Austin
Typical 5x5 L-C chart
Example regression line
High-level demographic
analysis of survey results
Combining
experts
diagram
Example GER spreadsheet
This plot shows the
RERs have a strong
preference for
function form 10
Moving Outside Data Range Results