Portfolio Decision Analysis Methods and...
Transcript of Portfolio Decision Analysis Methods and...
Helsinki University of TechnologySystems Analysis Laboratory
11
Portfolio Decision Analysis Portfolio Decision Analysis Methods and ApplicationsMethods and Applications
Ahti Salo (with contributions by
Juuso Liesiö, Ville Brummer, Juuso Nissinen)Systems Analysis Laboratory
Helsinki University of [email protected]
Helsinki University of TechnologySystems Analysis Laboratory
2
Helsinki University of TechnologySystems Analysis Laboratory
3
Characteristics of PDA problemsCharacteristics of PDA problems
Large number of proposals – Typically dozens or even hundreds of proposals
Only a fraction can be selected with available resources – Even non-monetary resources are important (e.g., critical competences)
“Value” may be measured with regard to several criteria – International collaboration, innovativeness, feasibility of plans
Reliable information about value is hard to obtain– Different experts may give different ratings
Helsinki University of TechnologySystems Analysis Laboratory
4
Major investments decisions, e.g., industrial plants
Important resource allocation decisions:• R&D projects• Product development projects• Equipment purchases• Maintenance programs
Selection of actions for implementing strategies and development plans
One-of-a-kind major strategic decisions
Day-to-day management of activities
Financial portfolio management
Project Portfolio Management (PPM)
Small-scale resource adjustments and adaptations to plans
Enterprise resource planning, supply chain mgmt etc.
Portfolio Decision Analysis
Where is Portfolio Decision Analysis positioned? Where is Portfolio Decision Analysis positioned?
Strategy development
Helsinki University of TechnologySystems Analysis Laboratory
5
Benefits along several dimensions Benefits along several dimensions
A shared analytical framework can be highly valuable– Clarifies the ‘units of analysis’ and corresponding their evaluation criteria– Supports communication by establishing a shared terminology
DA offers several desirable process characteristics – Ability to engage many stakeholder groups – Transparency, Consistency, Comprehensiveness, Modularity, Scalability,
Repeatability, …
‘Optimality’ is difficult if not impossible to prove ex post
Helsinki University of TechnologySystems Analysis Laboratory
6
Perspectives in project evaluation and selection Perspectives in project evaluation and selection
Methods for selecting projects within a given process – How to deal with uncertainties about criterion-specific evaluations?– How to implement requirements at the portfolio level?
» Balance: “At least 25% shall be focused on the market segment A”» Logical relationships: “Projects X and Y are mutually exclusive”
Design of the evaluation and selection process – Process properties can be analyzed through simulation– Key parameters
» Approval rate (i.e., the share of those projects that can be actually funded) » The number, quality and cost of evaluation statements» Sequencing of steps in the evaluation process and decision making » Amount of effort required by proposal preparation
Helsinki University of TechnologySystems Analysis Laboratory
7
Applications of Portfolio Decision Analysis at TKKApplications of Portfolio Decision Analysis at TKK
Maintenance of built infrastructures (Finnish Road Administration)– Bridge repair programs– Allocation of resources to ro road maintenance products
Priorities for joint research activities in view of 10-15 years– Strategic-Research Agenda for Forest-Based Industries– Strategies for industrial federations (packaging, wood products, food and drink,… )
Evaluation of the cost-efficiency of weapons systems (MATINE)
Allocation of resources to standardization activities (Nokia)
Helsinki University of TechnologySystems Analysis Laboratory
8
OptimizerOptimizer’’s Curse in Portfolio Decision Analysiss Curse in Portfolio Decision Analysis
Helsinki University of TechnologySystems Analysis Laboratory
9
Projects offer different amounts of value (eg NPV)
Estimates about projects’ values are uncertain
Decisions are based on these uncertain value estimates
Projects whose values have been overestimated have a higher chance of getting selected
Thus the DM should expect to be disappointed with the performance of the selected portfolio
Rationale for optimizerRationale for optimizer’’s curses curse
Helsinki University of TechnologySystems Analysis Laboratory
10
Example on choosing 6 out of 12 projectsExample on choosing 6 out of 12 projects
Helsinki University of TechnologySystems Analysis Laboratory
11
Value of information and optimality in DA Value of information and optimality in DA The optimizer’s curse: skepticism and postdecision surprise in decision analysis (Smith and Winkler, 2006) – Positively correlated errors aggravate the curse
Value of information in project portfolio selection (Keisler, 2004)– Different selection rules have an impact on the quality of the selected portfolio
How ‘bad’ is the optimizer’s curse in project portfolio selection?
What selection rules are better than others?
Helsinki University of TechnologySystems Analysis Laboratory
12
Approach and research questionsApproach and research questions
Key questions– How does (i) the number and (ii) quality of evaluation statements impact the
optimal project portfolio? – What kinds of evaluation and selection procedures outperform others?
Concepts– True value: Value (e.g., quality, research output) which would be produced, if
the project were to be funded– Estimated value: Value that the expert reports in his/her evaluation statement– Optimal portfolio: The portfolio that maximizes the aggregate sum of true values
(typically not known, can be determined only if true values are known)– Selected portfolio: The portfolio that maximizes the sum of estimated values
Results based on simulation and optimization models
Helsinki University of TechnologySystems Analysis Laboratory
13
100 project proposals – Only 20 will be selected ( approval rate 20 %)
“True” underlying value distributed on the range 1-5
At least one evaluation statement per each proposal – Evaluation statements convey information about the true value – All statements have the same cost (e.g., about 0.5% of project costs)
Decisions are based on evaluation statements
Illustration of project evaluation and selection Illustration of project evaluation and selection
Helsinki University of TechnologySystems Analysis Laboratory
14
Examples of selection mechanisms Examples of selection mechanisms
One-phase (“batch-mode”) – Equally many evaluations (1 or several) on each proposal – Projects selected on the basis of the average of reported
ratings on the evaluation statements
Two-phase 1. Discard 50 % of proposals based on a single evaluation statement2. Acquire additional statements on the remaining 50 %3. Select projects on the basis of the average of ratings on the reported
statements Additional
statements on the remaining 50%
Discard 50% based on 1 statement
Choose 20%
Proposals
Choose 20%
Statements
Proposals
Helsinki University of TechnologySystems Analysis Laboratory
15
Distributions of underlying value and statementsDistributions of underlying value and statements
Distribution of “true” value is modelled through a probability distribution
Evaluation statements depend on the true value
– “Good” proposals are likely to have a higher rating on the 1-5 scale
v =
1~ ( , )
x
i x
i x
ee
x N
µ
α σ
←+
1~ (0, )
x
i xeV
eN
ε
ε
εε σ
+
+←+
Helsinki University of TechnologySystems Analysis Laboratory
16
OptimizerOptimizer’’s curse is mitigated by more informations curse is mitigated by more information
(based on the distributions on the preceding slide)
Evaluation cost (% of project cost)
Ave
rage
val
ue o
f sel
ecte
d pr
ojec
ts 2-phase (real)
2-phase (estimated)1-phase (real)1-phase (estimated)
Helsinki University of TechnologySystems Analysis Laboratory
17
Information brings closer to the optimumInformation brings closer to the optimum
Small uncertaintiesLarge uncertainties
Evaluation cost (% of project cost)
2-phase 1-phase
Val
ue o
f the
sel
ecte
d po
rtfol
io a
s %
of t
he
optim
um p
ortfo
lio
Helsinki University of TechnologySystems Analysis Laboratory
18
But justice to the individual is difficult to guaranteeBut justice to the individual is difficult to guarantee
Small uncertaintiesLarge uncertainties
Sha
re o
f sel
ecte
d pr
ojec
ts (%
) tha
t are
als
o in
the
optim
al p
ortfo
lio
Evaluation cost (% of project cost)
2-phase 1-phase
Helsinki University of TechnologySystems Analysis Laboratory
19
Impact of competitive tendering on productivity 1(3)Impact of competitive tendering on productivity 1(3)
Include the effort of proposal preparation– Approval rate 20 % (select 20 projects out of 100 proposals)
When do the benefits of further statements exceed the cost of obtaining them? – Evaluation costs estimated here at 0.5% of project costs – A statement on a 100 000€ project costs 500 €
Account for the efforts required by proposal preparation, too– Preparation efforts estimated at 5% of project costs (100 000€ *0.05 = 5000€)– If one statement is obtained on all projects, the total cost will be
20*100 000€ + 100*5500€ = 2,55 M€
Helsinki University of TechnologySystems Analysis Laboratory
20
Impact of competitive tendering on productivity 2(3)Impact of competitive tendering on productivity 2(3)
(based on larger uncertainties)
Aggregate preparation and evaluation cost (% of project cost)
10% preparation cost
5% preparation cost
0% preparationcost
2-phase selection1-phase selection random selection with no tendering
Ave
rage
val
ue (r
esea
rch
prod
uctio
n) p
er u
nit
of to
tal e
xpen
ditu
re
Helsinki University of TechnologySystems Analysis Laboratory
21
Impact of competitive tendering on productivity 3(3)Impact of competitive tendering on productivity 3(3)Competitive tendering enhances productivity when – There is high variability in the quality of proposals – Approval rate is high enough– Proposal preparation does not require excessive efforts – Evaluation statements are reasonably good (i.e., correlated with actual quality)
Observations– Preceding results merely exemplify what kinds of questions can be answered – Parameters can be estimated from data (databases, expert judgements)– Lends support for improving evaluation and selection processes
Helsinki University of TechnologySystems Analysis Laboratory
22
Application: Application:
Development of National Research PrioritiesDevelopment of National Research Priorities
Helsinki University of TechnologySystems Analysis Laboratory
23
ForestForest--Based Sector Technology Platform (FTP)Based Sector Technology Platform (FTP)
One of the over 30 European Technology Platforms– Coordination of industry-lead European R&D activities– Establishment of the European Research Area (ERA)
This particular Technology Platform initiated by – European Confederation of Woodworking Industries– Confederation of European Forest Owners– Confederation of European Paper Industries
Over 30 countries involved– Long-term perspective (2030)– Development of the Strategic Research Agenda (SRA) in Member States and
at the European level
Helsinki University of TechnologySystems Analysis Laboratory
24
1 2 3 4 5 6 7 8 9 10 11 12
Organisation establishedValue-chain leaders elected.Setting up the guidelinesStep 1: Collection of inputsStep 2: European priorizationStep 3: Strategic objectives and research themesStep 4: Open discussion and finalisingDeveloping SRA documentFinal SRA 1. Dec.
Step 1: Each country was requested to identify 10 -15 most relevant research themes in view of national priorities
Strategic Research Agenda (SRA) for the FTPStrategic Research Agenda (SRA) for the FTP
Helsinki University of TechnologySystems Analysis Laboratory
25
ChallengesChallenges
New policy instrument No established approaches
A very broad range of issues to be covered– Many stakeholder groups (e.g., pulp and paper industry, bioenergy, forestry)– Long time scale considerable uncertainties
Tight timetable– Only 7 weeks Need for a structured decision support process
Multiple interfaces to other policy processes – E.g., preparation of Framework Program (FP7) in Europe
Helsinki University of TechnologySystems Analysis Laboratory
26
Finnish Case: National SRA process for the FTPFinnish Case: National SRA process for the FTP
Systematic process to engage Finnish key stakeholders – Development of the national SRA– Linked explicitly to the Vision 2030 document at the European level
Five value chains– Forestry, Wood Products, Pulp and Paper, Bio Energy, Specialties/ New
Businesses– Independent but interrelated process for each value chain
Identification and assessment of research themes– Internet questionnaires – MCDA analysis - interactive decision workshops
Synthesis of national results at the end of the process
Helsinki University of TechnologySystems Analysis Laboratory
27
Participants and rolesParticipants and rolesSteering Group– Coordinators and selected key persons (~ 10 people)
Coordinators – Chairs of national value chain Working Groups (5 people)
TKK Group– Research team of Prof. Ahti Salo at the Systems Analysis Laboratory / TKK
Respondents – 20-30 participants from each value chain
Referees – 6-10 participants from each value chain
Helsinki University of TechnologySystems Analysis Laboratory
28
Process designProcess designProcess steps Weeks Key participants
I Step: Internet-based solicitation of research themes
1-2 Respondents
II Step: Internet-based assessment of research themes
3-4 Referees
III Step: Multi-criteria analysis of research themes
4-5 TKK group
IV Step: Value chain workshops for the formulation of relevant research areas
5-6 Value Chain Coordinators and invited Respondents, Referees and other experts
VI Step: Steering Group workshop for the formulation of Finnish SRA priorities
7 Steering Group
Helsinki University of TechnologySystems Analysis Laboratory
29
Task 1: Solicitation of research themesTask 1: Solicitation of research themes
Timetable: April 27 – May 8
Participants: 20-30 Respondents from each value chain
Task: In each value chain, respondents proposed research themes with the Opinions-Online© decision support tool
Result: Total 146 research themes
Task 1 Example
Helsinki University of TechnologySystems Analysis Laboratory
30
Task 2: Assessment of research themesTask 2: Assessment of research themes
Timetable: Mid-May
Participants: 6-10 Referees from each value chain
Task: In each value chain, referees assessed research themes with Opinions-Online©
Result: Numerical assessment of research themes with regard to feasibility, industrial relevance, novelty
Task 2
Helsinki University of TechnologySystems Analysis Laboratory
31
Task 3: Results of Portfolio Decision AnalysisTask 3: Results of Portfolio Decision Analysis
imetable: Mid-May
articipants: Research group at TKK
ask: TKK analysed the results using RPM-methodology– ”Research themes” treated as ”projects”
– Scores defined as averages of criterion-specific evaluations
– Seven most interesting themes from the whole set highlighted )
esult: Shortlists of interesting themes in each value chain
Helsinki University of TechnologySystems Analysis Laboratory
32
Task 4: Value chain workshopsTask 4: Value chain workshops
Timetable: May 23 – 31
Participants: Value chain working groups – Selected respondents, referees and other experts
Task: Value chain Working Groups discussed on the results and identified most relevant research themes
Result: 3-7 the most relevant themes from each value chain
Helsinki University of TechnologySystems Analysis Laboratory
33
Task 5: SRA workshopsTask 5: SRA workshops
Timetable: June 8
Participants: SRA Steering Group (inclu. value chain coordinators)
Task: Based on the results from previous tasks and especially from value chain workshops, SRA steering group identified the most relevant 15 research themes
Result: Most relevant research themes– Taken forward to European level
Helsinki University of TechnologySystems Analysis Laboratory
34
ObservationsObservations
Systematic priority-setting processes– Permits extensive stakeholder participation even with tight schedules– Is transparent in terms of methodology – Supports workshop discussions through MCDA analyses
Considerations– Formal MCDA results need to be complemented discussions– Multiple perspectives can be highlighted through different criteria weights
Applicable in several other contexts, too
Helsinki University of TechnologySystems Analysis Laboratory
35