Revisiting the Tradespace Exploration Paradigm...
Transcript of Revisiting the Tradespace Exploration Paradigm...
Revisiting the Tradespace Exploration Paradigm: Structuring the Exploration Process
Adam M. Ross, Hugh L. McManus, Donna H. Rhodes, and Daniel E. Hastingsg
August 31, 2010
Track 40-MIL-2: Technology TransitionTrack 40-MIL-2: Technology TransitionAIAA Space 2010
Outline
• Introduction• Advances in Tradespace Exploration• Question-guided TSE• Question-guided TSE• Discussion• Conclusion
seari.mit.edu © 2010 Massachusetts Institute of Technology 2
Introduction
• Early design process is high leverage, with consequences for achievable benefits and cost goals
• Early design is rife with uncertainty and possibilities as well, making “good” decisions more difficult
Tradespace database to be explored
g– Many possible designs– Many possible (changing?) needs
and stakeholders– New (changing?) technology
be explored
Util
ityU
tility
( g g ) gy– New contexts (and missions)
• Tradespace exploration compares many designs on a common, quantitative basis
Maps structure of design space onto stakeholder value (attributes)
CostCost
– Maps structure of design space onto stakeholder value (attributes)– Uses computer-based models to assess thousands of designs, avoiding limits of
local point solutions– Simulation can be used to account for design uncertainties (e.g., cost, schedule,
performance uncertainty)
seari.mit.edu © 2010 Massachusetts Institute of Technology 3
Value-based assessments allow for comparison of many different alternatives
performance uncertainty)
Tradespace Data Representation
Rich data sets canRich data sets can be explored to reveal complex relationships
bbetween design-space and
value-space for generatingfor generating intuition into
problem—a multi-dimensional l t hianalogy to graphing
y=f(x)
seari.mit.edu © 2010 Massachusetts Institute of Technology 4
“Explore” tradespace data to develop intuition into complex design-value relationships
Tradespace Exploration Paradigm: Avoiding Point Designs (2005)Avoiding Point Designs (2005)
Ross, A.M. and Hastings, D.E., “The Tradespace Exploration Paradigm,” INCOSE International Symposium 2005, Rochester, NY, July 2005.
UtilityDiffering types of trades
1. Local point solution trades
Cost
Designi = {X1, X2, X3,…,Xj}
seari.mit.edu © 2010 Massachusetts Institute of Technology 5
Tradespace Exploration Paradigm: Avoiding Point Designs (2005)Avoiding Point Designs (2005)
Ross, A.M. and Hastings, D.E., “The Tradespace Exploration Paradigm,” INCOSE International Symposium 2005, Rochester, NY, July 2005.
UtilityDiffering types of trades
1. Local point solution trades
2. Frontier subset solutions
Cost
Designi = {X1, X2, X3,…,Xj}
seari.mit.edu © 2010 Massachusetts Institute of Technology 6
Tradespace Exploration Paradigm: Avoiding Point Designs (2005)Avoiding Point Designs (2005)
Ross, A.M. and Hastings, D.E., “The Tradespace Exploration Paradigm,” INCOSE International Symposium 2005, Rochester, NY, July 2005.
UtilityDiffering types of trades
1. Local point solution trades
2. Frontier subset solutions
3. Frontier solution set
Cost
Designi = {X1, X2, X3,…,Xj}
seari.mit.edu © 2010 Massachusetts Institute of Technology 7
Tradespace Exploration Paradigm: Avoiding Point Designs (2005)Avoiding Point Designs (2005)
Ross, A.M. and Hastings, D.E., “The Tradespace Exploration Paradigm,” INCOSE International Symposium 2005, Rochester, NY, July 2005.
UtilityDiffering types of trades
1. Local point solution trades
2. Frontier subset solutions
3. Frontier solution set
4. Full tradespace exploration
Cost
Designi = {X1, X2, X3,…,Xj}
seari.mit.edu © 2010 Massachusetts Institute of Technology 8
Tradespace exploration enables big picture understanding
Tradespace Exploration Paradigm: Avoiding Point Designs (2010)Avoiding Point Designs (2010)
Differing types of “trades”Utility
g yp
1. Local point solution trades
0. Choose a solution
p
2. Multiple points with trades
3. Frontier solution set3 o t e so ut o set
4. Full tradespace exploration
Cost
Designi = {X1, X2, X3,…,Xj}
seari.mit.edu © 2010 Massachusetts Institute of Technology 9
Tradespace exploration enables big picture understanding
Tradespace Exploration Paradigm: Avoiding Point Designs (2010)Avoiding Point Designs (2010)
Differing types of “trades”Utility
g yp
1. Local point solution trades
0. Choose a solution
p
2. Multiple points with trades
3. Frontier solution set3 o t e so ut o set
4. Full tradespace exploration
Cost
Designi = {X1, X2, X3,…,Xj}
seari.mit.edu © 2010 Massachusetts Institute of Technology 10
Tradespace exploration enables big picture understanding
Tradespace Exploration Paradigm: Avoiding Point Designs (2010)Avoiding Point Designs (2010)
Differing types of “trades”Utility
g yp
1. Local point solution trades
0. Choose a solution
p
2. Multiple points with trades
3. Frontier solution set3 o t e so ut o set
4. Full tradespace exploration
Cost
Designi = {X1, X2, X3,…,Xj}
seari.mit.edu © 2010 Massachusetts Institute of Technology 11
Tradespace exploration enables big picture understanding
Tradespace Exploration Paradigm: Avoiding Point Designs (2010)Avoiding Point Designs (2010)
Differing types of “trades”Utility
g yp
1. Local point solution trades
0. Choose a solution
p
2. Multiple points with trades
3. Frontier solution set3 o t e so ut o set
4. Full tradespace exploration
Cost
Designi = {X1, X2, X3,…,Xj}
seari.mit.edu © 2010 Massachusetts Institute of Technology 12
Tradespace exploration enables big picture understanding
Tradespace Exploration Paradigm: Avoiding Point Designs (2010)Avoiding Point Designs (2010)
Differing types of “trades”Utility
g yp
1. Local point solution trades
0. Choose a solution
p
2. Multiple points with trades
3. Frontier solution set3 o t e so ut o set
4. Full tradespace exploration
5. Many tradespace explorationsCost
Designi = {X1, X2, X3,…,Xj}
5. Many tradespace explorations
seari.mit.edu © 2010 Massachusetts Institute of Technology 13
Tradespace exploration enables big picture understanding
Example Tradespace InsightsSPACETUG
• General purpose orbit transfer vehicles • Trades propulsion systems and grappling/observation capabilities
Technical Cost and Utility UncertaintyTechnical Cost and Utility Uncertainty
Understanding limiting physical or mission constraints
Understanding differential uncertainty
11
Comparing alternatives on common basis
nsio
nles
s)
0.6
0.8
1Technical, Cost, and Utility Uncertainty
nsio
nles
s)
0.6
0.8
1Technical, Cost, and Utility Uncertainty
dim
ensi
onle
ss)
1
0.6
0.8
dim
ensi
onle
ss)
1
0.6
0.8
dim
ensi
onle
ss)
1
0.6
0.8
0.6
0.8
1
ensi
onle
ss)
0.6
0.8
1
ensi
onle
ss)
Util
ity (d
imen
0.2
0.4
Util
ity (d
imen
0.2
0.4Util
ity (d
0.2
0.4Util
ity (d
0.2
0.4Util
ity (d
0.2
0.4 Lines show increasing fuel mass fraction
Biprop CryoElectricNuclear0.2
0.4
Util
ity (d
ime
Designs from traditional process
Biprop CryoElectricNuclear0.2
0.4
Util
ity (d
ime
Designs from traditional processDesigns from traditional process
Hits “wall” of either physics (can’t change) or
Cost ($M)0 100 200 300 400 500 600 700 800
0
Cost ($M)0 100 200 300 400 500 600 700 800
0
Different designs subject
Cost (M$)0 500 1000 1500 2000 2500 3000 3500 40000
Cost (M$)0 500 1000 1500 2000 2500 3000 3500 40000
Cost (M$)0 500 1000 1500 2000 2500 3000 3500 40000
Cost (M$)
00 500 1000 1500 2000 2500 3000 3500 4000
Cost (M$)
00 500 1000 1500 2000 2500 3000 3500 4000
Common “value” definition can compare old and new
seari.mit.edu © 2010 Massachusetts Institute of Technology 14
physics (can t change) or utility (can)
Different designs subject to different risks
can compare old and new heterogeneous systems
Multi-Concept Operationally Responsive Disaster SurveillanceResponsive Disaster Surveillance
Chattopadhyay, D., Ross, A.M., and Rhodes, D.H., "Demonstration of System of Systems Multi-Attribute Tradespace Exploration on a Multi-Concept Surveillance Architecture," 7th Conference on Systems Engineering Research, Loughborough University, UK, April 2009.
StakeholdersDesign ConceptsAi ft Firefighter
ORS Owner
• Aircraft• Satellite • Sensor Swarm• SoS designs consisting of
t f bany two of above
Diverse concepts can be compared on same basis; satellites only make
seari.mit.edu © 2010 Massachusetts Institute of Technology 15
p p ysense if costs can be amortized over many disasters…
Transportation Planning
BLS(n=20,000)
Nickel, J., "Using Multi-Attribute Tradespace Exploration for the Architecting and Design of Transportation Systems," Master of Science Thesis, Engineering Systems Division, MIT, Cambridge, MA, February 2010.
BLS
• Freedom to make changes
BRT
changes• Competition
agreements• QOS:
Fare levelFrequency
R t 2
FrequencyTravel timeAmenitiesSpan of service
Route 2
• Operating costs• Concession payment
Prematurely focusing on point solutions is not unique to aerospace!
seari.mit.edu © 2010 Massachusetts Institute of Technology 16
Route 2 (dominated) is the only concept planners had initially considered
p y solutions is not unique to aerospace!
Tradespaces Over Time
Koo, C.K.K., “Investigating Army Systems and Systems of Systems for Value Robustness ” Master of Science in Engineering Management System Design
Today Possible futures (epochs)
Robustness, Master of Science in Engineering Management, System Design and Management Program, MIT, Cambridge, MA, February 2010.
Util
ity
EpochCost
Util
ity
EpochCost
Util
ity
EpochCost
D iDemonstrations• Diverse set of “point designs”
compared on common basis
• Many tradespaces evaluated• Many tradespaces evaluated over changing contexts (e.g., technology levels) and needs (e.g., missions and utilities)
• Allows for identification of
seari.mit.edu © 2010 Massachusetts Institute of Technology 17
Allows for identification of alternatives robust against uncertainties revealed over time
Structuring Exploration: Question-Guided ApproachGuided Approach
• Decade of tradespace research encapsulated in papers and theses, but “art” of exploration resided in expertsNeed for codif ing e ploration in order to mat re the method and aid in• Need for codifying exploration in order to mature the method and aid in deployment
• Proposed series of case study explorations guided by inquiry (i.e., “questions”) in order to codify expertise (through hypothetical “user” sessions)
Activity uses VisLab* software and pre populated database– Activity uses VisLab* software and pre-populated database– During a TSE session, users will seek to answer the following questions:
1. Can we find “good value” designs? 2. What are strengths and weakness of selected designs?3. Are lower cost designs feasible?4. What about time and change?5. What about uncertainty6. How can detailed design development be initiated to have
increased chance of program success?
Questions guide the exploration of the datasets, helping to select the proper tools and representations for knowledge-generation
increased chance of program success?
seari.mit.edu © 2010 Massachusetts Institute of Technology 18
*VisLab software is MATLAB®-based, in-house analysis and visualization program for interactive TSE, but any suite of applications that can perform the necessary functions can be used
TSE Input and Calculation ToolsIn order to conduct the question-guided exploration, the following input and calculation tools are needed:
Name Function
Pareto Calculator Find the Pareto front on any plot. Multi-dimensional Pareto capability also useful.
Preference Input Ability to accept changes in the Worst and Best values, and the Weights, for Attributes. Also ability to change the utility curves.
tools are needed:
Preference Calculator
Ability to recalculate the single- and multi-attribute utilities using the new preferences, and use them as the basis for all of the above displays.
seari.mit.edu © 2010 Massachusetts Institute of Technology 19
TSE Display ToolsName Function
Tradespace Plot Plot single- or multi-attribute utilities versus cost. Use color to represent a third dimension (e.g., design vector values).
Strength/Weakness Plots
Multiple plots showing physical attributes and their associated utilities, against cost or other factors. Use color for a third dimension.
Sensitivity Plots Multiple plots showing sensitivities of one factor to another (e.g., attributes to design vector values).
Design Definition Ability to pick a point on any of the above plots and find out what design it is associated with.
Favorites List List of favored designs, with key information and a symbol or icon. Display these designs on all plots using their special symbol.
Di l d th h i l h t i ti (d i t l ) d f ( tt ib tComparison Table Display and compare the physical characteristics (design vector values) and performance (attributes and utilities) of selected designs.
Era Viewer Multiple plots showing a tradespace under a variety of conditions (epochs) that together represent a scenario for changes over time.
Era Animator Animation of the era; a single plot that shifts as conditions change across epochs.
seari.mit.edu © 2010 Massachusetts Institute of Technology 20
These display tools allow for rapid TS interpretation
Q1: Can we find good value designs?
• Define “good”• Identify good value designs y g g
– Find high utility, low cost designs, with associated physical design parameters
• Understand utility vs. cost t d fftradeoffs
– Find Pareto front, investigate design relationships
• Look at detailsU d t d d i d– Understand design and performance variations along Pareto front
– Develop first set of interpretations• CautionsCautions
– develop trust in model, – avoid anchoring, – do not assume Pareto front has all
good designs
seari.mit.edu © 2010 Massachusetts Institute of Technology 21
g g
Q2: What are the strengths and weaknesses of selected designs?weaknesses of selected designs?
• Quick look– Tabulate performance of
ffavored designs,– Compare to acceptance
ranges,– Compare to achievable values
of other designsof other designs• Tradespace look
– Observe tradespace achievable ranges for each attribute
– Compare location of ParetoCompare location of Pareto front designs to other designs
• Cautions– Insights may be relative to
specifically evaluated tradespace
– Attribute performance may be coupled
seari.mit.edu © 2010 Massachusetts Institute of Technology 22
Q3: Are lower cost designs feasible?
• Find “low cost” designs in Pareto front• Determine if attribute acceptance ranges exclude lower cost designs
Experimentally relax attribute ranges and replot– Experimentally relax attribute ranges and replot• Expand design space to include more low cost designs
– Enumerate new levels of design parameters– Propose new design parameters that may lead to lower cost designs
R l t t d if d d ( d i t d l• Re-evaluate tradespace if needed (e.g., new designs types or models needed)
• Cautions– Tradeoff of cost for utility can only be interpreted by decision maker
seari.mit.edu © 2010 Massachusetts Institute of Technology 23
– “Small” differences in utility is difficult to determine in “utility-space”– Relaxation of attribute ranges must be accepted by affected decision maker
Q4: What about time and change?Needs change
• Define and evaluate epochs– Needs, contexts,– Rerun model as needed
• Use multi-epoch metrics to identify interesting designs and epochs– Pareto Trace, Fuzzy Pareto Trace, tradespace yield
• Define an era (time ordered sequence of epochs)Define an era (time ordered sequence of epochs)• Identify differences across epochs
– Colors and animations can help to illustrate • Cautions
Balance completeness with practicality when enumerating epochs and eras
seari.mit.edu © 2010 Massachusetts Institute of Technology 24
– Balance completeness with practicality when enumerating epochs and eras
Q4: What about time and change?Needs change
Pareto Trace Number# Pareto Sets containing design
(measure of passive robustness)
Pareto Trace Number# Pareto Sets containing design
(measure of passive robustness)(measure of passive robustness)
m o
f des
igns
Util
ity
(measure of passive robustness)
m o
f des
igns
m o
f des
igns
Util
ityU
tility
• Define and evaluate epochs– Needs, contexts
Num
Pareto Trace NumberEpochCost
Num
Pareto Trace Number
Num
Pareto Trace NumberEpochCost
EpochCost
Across many epochs, track number of times solution appears in Pareto Set,
– Rerun model as needed• Use multi-epoch metrics to identify interesting designs and epochs
– Pareto Trace, Fuzzy Pareto Trace, tradespace yield• Define an era (time ordered sequence of epochs)
times solution appears in Pareto Set
Define an era (time ordered sequence of epochs)• Identify differences across epochs
– Colors and animations can help to illustrate • Cautions
Balance completeness with practicality when enumerating epochs and eras
seari.mit.edu © 2010 Massachusetts Institute of Technology 25
– Balance completeness with practicality when enumerating epochs and eras
Q5: What about uncertainty?
• Investigate sensitivities– Uncertainty affecting sensitive– Uncertainty affecting sensitive
factors may indicate risk• Use Epoch-Era Analysis to
understand uncertainties due to discrete changes in contexts and needs– Memory intensive, so limit scope
Sensitivities
• Use Monte Carlo analysis to understand propagation of uncertainty in performance of s bset of designssubset of designs– Computationally intensive, so limit
scope
seari.mit.edu © 2010 Massachusetts Institute of Technology 26
Monte Carlo-derived uncertainties
Q6: How can detailed design be initiated to maximize program success?to maximize program success?
Tradespace exploration appears to have promise in addressing:addressing:– Picking good projects
• TSE shows what is possible in terms of tradeoffs• TSE can be used to identify “favored” designs• TSE can be used to identify favored designs
– Specifying good requirements• TSE shows impact of acceptability ranges on possible utility and
cost, including “good” and “bad” constraints on designscost, including good and bad constraints on designs– Understanding risk areas
• TSE allows for consistent comparison of alternative’s sensitivities and uncertainty propagation tendencies
– Understanding alternatives• TSE allows for gaining insights into multiple alternatives
simultaneously, creating essential contingency knowledge and design choice justification
seari.mit.edu © 2010 Massachusetts Institute of Technology 27
design choice justification
Discussion• New demonstrated TSE capabilities/insights
– Multiple cost types– Different concepts on same tradespace– Systems of systems can be represented, but require more sophisticated
modeling/fusion (ongoing research topic)– Dynamic issues difficult to analyze/visualize, metrics can be used to y y
screen/filter large datasets (ongoing research topic)• Structured question-guided tradespace exploration
– Guided process, with enabling tools, much faster and complete than ad hoc explorationp
– Rapid feedback results in ability to perform additional analysis as well as more “advanced” synthesis
– Deeper TS exploration into the data and pursuit of more nuanced analysis may require access to data-generators (i.e., “modelers”) as well asmay require access to data generators (i.e., modelers ) as well as preference constructors (i.e., “stakeholders”)
• Need to understand actual vs. perceived limitations and assumptions– TS data demonstrates correlation; SMEs needed to determine causation
seari.mit.edu © 2010 Massachusetts Institute of Technology 28
Ultimate goal is knowledge-generation, so trust in the data and tradespace exploration methods is essential
Tradespace Exploration Benefits
• Forces alignment of solutions to needs
The following strengths of TSE were identified by a user of the method
Forces alignment of solutions to needs• Reveals structure of design-value spaces not apparent with few point
designs– Akin to graphing calculator showing function shapes, tradespaces give g p g g p , p g
insight/intuition into complex design-value space relationships• Facilitates cross-domain socio-technical conversation• Ability to discover compromise solutions
– Beyond “optimized” per stakeholder solutions– Experts often unable to find “suboptimal” solution that may be better
compromise across stakeholdersStructured means for considering large array of possible futures for• Structured means for considering large array of possible futures for discovering robust systems and strategies
TSE methods highlight and help to focus attention on important trades, possibly o erlooked b traditional methods
seari.mit.edu © 2010 Massachusetts Institute of Technology 29
overlooked by traditional methodsOn-going research is further developing TS visualizations, metrics, and analyses