TIA Consulting, Inc.
Portfolio Evaluation: Toward a Common Evaluation Methodology for
Funding Decisions Across Multiple
Interdependent R&D Portfolios of a U.S. Agency
New Frontiers In EvaluationPlattform FTEvalApril 24-25, 2006Vienna, Austria
Rosalie T. RueggManaging DirectorTIA Consulting, [email protected]
2TIA Consulting, Inc.
Background: Retrospective Evaluation of DOE Applied Energy
R&D Programs• $Billions allocated to federal applied energy
R&D programs.• Evaluations used different models, data, and
assumptions across programs.• Congressional appropriators expressed
frustration with evaluations offered in support of their budget decisions.
• 2000, Congress ordered comprehensive retrospective study of these programs (2001 NRC Retrospective Evaluation Report).
3TIA Consulting, Inc.
Retrospective Evaluation Matrix
Realized Benefits and Costs
Options Benefits and Costs
Knowledge Benefits and Costs
Economic Benefits and Costs
Environmental Benefits and Costs
Security Benefits and Costs
4TIA Consulting, Inc.
Accompanying Tool for “Loosely” Characterizing Uncertainty in
Retrospective Study Technology
DevelopmentEconomic/Policy Conditions
TechnologyDeveloped
Technology Development in Progress
Technology Development Failed
Will be favorable for commercial-ization
Realized benefits Knowledge benefits
Knowledge benefits
Might become Favorable for commercialization
Options benefits Knowledge benefits
Knowledge benefits
Will not become favorable for commercialization
Knowledge benefits
Knowledge benefits
Knowledge benefits
Source: NRC Report, 2001.
5TIA Consulting, Inc.
More Recent Portfolio Assessment
• Congress continued to express interest in benefits assessment for these programs.
• 2003, Congress requested the NRC to - develop a comprehensive prospective evaluation methodology
- apply it on a continuing basis to review the programs, - to inform Congressional decision making.
• 2005, NRC Prospective Evaluation Methodology Report Published. (Phase 1 of the study)
• 2006, NRC-implementation of new method to provide Congress a better understanding of portfolio benefits and costs is underway. (Phase 2 of the Study)
6TIA Consulting, Inc.
NRC Committee & Staff on Prospective Benefits of DOE’s Energy Efficiency and Fossil
Energy R&D ProgramsCommittee Members-Phase 1:Robert Fri, Chair, Resources for the FutureLinda Cohen, University of CaliforniaJames Corman, Energy Alternatives Studies, Inc.Paul DeCotis, New York State Energy R&D AuthorityWesley Harris, MITGeorge Norton, Virginia TechRosalie Ruegg, TIA Consulting, Inc.Maxine Savitz, Honeywell, Inc. (retired)Jack Siegel, Energy Resources International, Inc.James Smith, Duke UniversityTerry Surles, Electricity Innovation InsituteJames Sweeney, Stanford UniversityJohn Wise, Mobil R&D Company (retired)
NRC Staff:Martin Offutt, Study Director
7TIA Consulting, Inc.
Priorities in Developing the Prospective Benefits Methodology
• Adoption to the extent feasible the work of the retrospective study and subsequent work done by DOE and OMB.
• Development of a methodology rigorous in its calculation of benefits and assessment of risks, but at the same time practical and consistent for applying across a variety of DOE programs.
• Development of a methodology that is transparent, easy to use, feasible to implement from the standpoint of resource requirements, and that produces results understandable by key stakeholders.
8TIA Consulting, Inc.
Prospective Benefits Methodology: 6 Key Elements
1. Consistent definition of benefits2. Common scenarios for analysis3. Explicit accounting of risks & gov’t role4. Consistent estimation of probabilities5. Simplified models for calculating EV benefits6. A results matrix for uniform summary
9TIA Consulting, Inc.
Results Matrix for Summarizing Prospective Benefits and Costs
PANEL NAME:
Program Name:Program Goals:Year Goals Expected to be Achieved:Program Costs:
Funding to Date: $Current Funding: $Proposed Year Funding: $Expected Cost to Completion: $Industry and Foreign Government Funding: $
Key Complementary/Interdependent DOE Programs:
(First Part of Matrix)
10TIA Consulting, Inc.
Results Matrix for Summarizing Prospective
Benefits and Costs
.
Source: NRC Report, 2005.
Reference Case
High Oil & Gas Prices
Carbon Constrained
ProgramRisks
Technical (cross ref to decision tree)
Market (cross ref to decision tree) Expected ProgramBenefits
Economic
Environmental
Security
Global Scenarios
(Second Part of Matrix)
Note: All benefits are cumulative through 2050 and are reported in 20XX year dollars.
11TIA Consulting, Inc.
Characterization of the Approach
• Harnesses the speed of expert panels• Guides the panels to perform comprehensive
decision tree analysis • Specifies consistent scenarios and calculations• Provides clear, consistent procedures for each
step • Yields cross-program comparable results
12TIA Consulting, Inc.
Selected for Application of Methodology
From DOE’s Office of Energy Efficiency & Renewable Energy:
• distributed energy program, • vehicle technologies program, • industrial technologies program;
From DOE’s Office of Fossil Energy:• integrated gasification combined cycle, • sequestration, • natural gas technologies (exploration and
production).
13TIA Consulting, Inc.
Implications Overview
The developments described have implications for:
• State-of-the-art R&D evaluation• Those who wish to organize and conduct
portfolio-scale evaluation studies in support of real-time policy and budgetary decisions
• User communities who make policy and budgetary decisions based on the results of evaluation studies
• Suppliers of traditional federal agency R&D evaluation
14TIA Consulting, Inc.
For further information…Information on the project is posted at the
National Academies Website. Go to National Research Council and search
“current projects” for “Prospective Benefits of DOE’s Energy Efficiency and Fossil Energy R&D Programs—Phase 2”
[http://www.nationalacademies.org/nrc/]
Top Related