TIA Consulting, Inc. Portfolio Evaluation: Toward a Common Evaluation Methodology for Funding...

TIA Consulting, Inc.

Portfolio Evaluation: Toward a Common Evaluation Methodology for

Funding Decisions Across Multiple

Interdependent R&D Portfolios of a U.S. Agency

New Frontiers In EvaluationPlattform FTEvalApril 24-25, 2006Vienna, Austria

Rosalie T. RueggManaging DirectorTIA Consulting, [email protected]

2TIA Consulting, Inc.

Background: Retrospective Evaluation of DOE Applied Energy

R&D Programs• $Billions allocated to federal applied energy

R&D programs.• Evaluations used different models, data, and

assumptions across programs.• Congressional appropriators expressed

frustration with evaluations offered in support of their budget decisions.

• 2000, Congress ordered comprehensive retrospective study of these programs (2001 NRC Retrospective Evaluation Report).


Retrospective Evaluation Matrix

Realized Benefits and Costs

Options Benefits and Costs

Knowledge Benefits and Costs

Economic Benefits and Costs

Environmental Benefits and Costs

Security Benefits and Costs


Accompanying Tool for “Loosely” Characterizing Uncertainty in

Retrospective Study Technology

DevelopmentEconomic/Policy Conditions

TechnologyDeveloped

Technology Development in Progress

Technology Development Failed

Will be favorable for commercial-ization

Realized benefits Knowledge benefits

Knowledge benefits

Might become Favorable for commercialization

Options benefits Knowledge benefits

Knowledge benefits

Will not become favorable for commercialization

Knowledge benefits

Knowledge benefits

Knowledge benefits

Source: NRC Report, 2001.


More Recent Portfolio Assessment

• Congress continued to express interest in benefits assessment for these programs.

• 2003, Congress requested the NRC to - develop a comprehensive prospective evaluation methodology

- apply it on a continuing basis to review the programs, - to inform Congressional decision making.

• 2005, NRC Prospective Evaluation Methodology Report Published. (Phase 1 of the study)

• 2006, NRC-implementation of new method to provide Congress a better understanding of portfolio benefits and costs is underway. (Phase 2 of the Study)


NRC Committee & Staff on Prospective Benefits of DOE’s Energy Efficiency and Fossil

Energy R&D ProgramsCommittee Members-Phase 1:Robert Fri, Chair, Resources for the FutureLinda Cohen, University of CaliforniaJames Corman, Energy Alternatives Studies, Inc.Paul DeCotis, New York State Energy R&D AuthorityWesley Harris, MITGeorge Norton, Virginia TechRosalie Ruegg, TIA Consulting, Inc.Maxine Savitz, Honeywell, Inc. (retired)Jack Siegel, Energy Resources International, Inc.James Smith, Duke UniversityTerry Surles, Electricity Innovation InsituteJames Sweeney, Stanford UniversityJohn Wise, Mobil R&D Company (retired)

NRC Staff:Martin Offutt, Study Director


Priorities in Developing the Prospective Benefits Methodology

• Adoption to the extent feasible the work of the retrospective study and subsequent work done by DOE and OMB.

• Development of a methodology rigorous in its calculation of benefits and assessment of risks, but at the same time practical and consistent for applying across a variety of DOE programs.

• Development of a methodology that is transparent, easy to use, feasible to implement from the standpoint of resource requirements, and that produces results understandable by key stakeholders.


Prospective Benefits Methodology: 6 Key Elements

1. Consistent definition of benefits2. Common scenarios for analysis3. Explicit accounting of risks & gov’t role4. Consistent estimation of probabilities5. Simplified models for calculating EV benefits6. A results matrix for uniform summary


Results Matrix for Summarizing Prospective Benefits and Costs

PANEL NAME:

Program Name:Program Goals:Year Goals Expected to be Achieved:Program Costs:

Funding to Date: $Current Funding: $Proposed Year Funding: $Expected Cost to Completion: $Industry and Foreign Government Funding: $

Key Complementary/Interdependent DOE Programs:

(First Part of Matrix)


Results Matrix for Summarizing Prospective

Benefits and Costs

.

Source: NRC Report, 2005.

Reference Case

High Oil & Gas Prices

Carbon Constrained

ProgramRisks

Technical (cross ref to decision tree)

Market (cross ref to decision tree) Expected ProgramBenefits

Economic

Environmental

Security

Global Scenarios

(Second Part of Matrix)

Note: All benefits are cumulative through 2050 and are reported in 20XX year dollars.


Characterization of the Approach

• Harnesses the speed of expert panels• Guides the panels to perform comprehensive

decision tree analysis • Specifies consistent scenarios and calculations• Provides clear, consistent procedures for each

step • Yields cross-program comparable results


Selected for Application of Methodology

From DOE’s Office of Energy Efficiency & Renewable Energy:

• distributed energy program, • vehicle technologies program, • industrial technologies program;

From DOE’s Office of Fossil Energy:• integrated gasification combined cycle, • sequestration, • natural gas technologies (exploration and

production).


Implications Overview

The developments described have implications for:

• State-of-the-art R&D evaluation• Those who wish to organize and conduct

portfolio-scale evaluation studies in support of real-time policy and budgetary decisions

• User communities who make policy and budgetary decisions based on the results of evaluation studies

• Suppliers of traditional federal agency R&D evaluation


For further information…Information on the project is posted at the

National Academies Website. Go to National Research Council and search

“current projects” for “Prospective Benefits of DOE’s Energy Efficiency and Fossil Energy R&D Programs—Phase 2”

[http://www.nationalacademies.org/nrc/]

TIA Consulting, Inc. Portfolio Evaluation: Toward a Common Evaluation Methodology for Funding...

Documents

Transcript of TIA Consulting, Inc. Portfolio Evaluation: Toward a Common Evaluation Methodology for Funding...