Cooperative State Research, Education, and Extension Service

Planning and Accountability

Experiences with Quantifiable Research Evaluation:

Portfolio Review Expert Panel

AEA, Toronto, Ontario, CanadaOctober 28, 2005

Cheryl J. Oros, DirectorHenry M. Doan, Djimé D. Adoum,

and Sharon Stout

CSREES Mission and Functions

Knowledge production and dissemination to solve national problems/ meet needs/ seize opportunities identified in the Strategic Plan

Agency works through state and university partners via grants

Planning/Decision Making

• Identify needs, problems, solutions, refinements

•Conceptualization of Program

•Formulation of Evaluation Questions and Design

Implementation

• Actualize the Program Plan

•Collection of Evaluation Data

• Analysis of Data

Evaluation Feedback

•Feedback of Evaluation Findings to Managers

•Refinement of Program

•Continued Funding

Evaluation/Management Cycle

USDA-CSREES Strategic Goals Fully Aligned (5 Goals)

1. Enhance economic opportunities for agricultural producers

2. Support increased economic opportunities and improved quality of life in rural America

3. Enhance protection and safety of the nation’s agriculture and food supply

4. Improve the nation’s nutrition and health5. Protect and enhance the nation’s natural

resource base and environment

Cascading Alignment.

Goal

Strategic Objective

Portfolio

Knowledge Area Code

Program/project

Mission

Alignment ExampleGoal 3: Enhance Protection & Safety of Food Supply

Strategic Objective 3.2 b: Animal Protection

Animal Protection Portfolio

Knowledge Areas• (KA 311) Improve the management of animal diseases• (KA 312) Increase knowledge & improve management of pests & external parasites• (KA 313) Increase knowledge to control internal parasites• (KA 314) Reduce losses in livestock, poultry, and fish• (KA 315) Develop and implement effective animal care

OMB PART OMB’s Program Assessment Rating

Tool Program Purpose & Design Strategic Planning Program Management Program Results

CSREES Goal 1 Portfolio reviewed 2004; Goals 3 & 5 in 2005; Goals 2 & 4 in 2006

Evaluating Research & Education Efforts: Portfolio Approach

OMB PART/BPI led to development of new portfolio assessment tool and measures

Portfolio analysis (meta-analysis) used to assess progress toward goals; guide announcements for grants

Uses OMB R&D criteria (relevance, quality, performance)

Portfolio Review Expert Panel (PREP) Process

Focus on outcomes rather than processes

Level of analysis is a portfolio identified via Knowledge Area codes in databases

A PREP scores portfolio progress & provides recommendations for Agency

Evidentiary materials Tracks papers, citations, patents,

products, educational efforts, adoption of products/ practices

Entire evaluation studies and special analyses

Budget tables to show portfolio priorities and emphases

Unique Features of PREP Upper level panelists are asked to

systematically assess distinct dimensions of the 3 OMB R&D criteria

Scoring process is standardized across portfolios, transparent, & scientifically based

Therefore, PREP can provide quantitative performance assessment of portfolios of research work

“Portfolio” as a new concept

Portfolio as unit of analysis is new concept Funding lines, programs, and organization

of CSREES work units does not include portfolios

Use of Knowledge Areas codes for all work classification is new

Allows complex interrelated programs and funding lines to be described as they address objectives and goals of USDA

Portfolio Review Expert Panel (PREP) Process

1. Selection of high-level panelists with broad experience in topic area, reviewed for absence of conflict of interest:

University Vice-Presidents Deans and Associate Deans Industry Experts (Vice Presidents) Evaluation Experts Experts from other federal agencies

Portfolio Review Expert Panel (PREP) Process (cont.)

2. Self assessment document sent to panelists prior to meeting:

Section I: Agency and PREP Overview Section II: Portfolio Description Section III: Knowledge Area Descriptions Section IV: Discussion of how portfolio

meets R&D Criteria and their Dimensions Compilation of evidentiary materials Self assessment scoring prior to panel

Portfolio Review Expert Panel (PREP) Process (cont.)

3. Panelists meet for 2 ½ days: Day 1 for orientation, short briefings

by managers, along with Q&As Day 2 for further review of

documentation, discussion, voting, and recommendations

Day 3 to complete draft report containing score for PART & BPI + recommendations for portfolio improvement; debriefing by panel

Interim Annual Internal Review

Update self-review document Consider recommendations from

Review Panel and describe Agency responses

Used as interim preparation for next external review at the fifth year

5 Dimensions of Relevance

Scope Focus on critical needs Identification of emerging

issues Integration of CSREES

programs Interdisciplinary integration

4 Dimensions of Quality

Significance of findings & outputs

Stakeholder assessment Alignment of portfolio with

current science Methodological rigor

5 Dimensions of Performance Portfolio productivity Portfolio completeness Portfolio timeliness Agency guidance relevant

to portfolio Portfolio accountability

Panel Scoring Sheet Panel scores each dimension of

each of three R & D criteria using customized anchors on a 3-point scale: 3=Exceeds expectations

2=Meets expectations 1=Needs improvement

Panel Scoring Sheet Example: Relevance

Section 1: Relevance

Dimensions

Purpose of Dimension

Rating: 3

Rating: 2

Rating: 1

40%of total

Total relevance score

1.1 Scope – coverage of the work of the full portfolio

Define & summarize needed & existing portfolio topics

Fully demonstrates exceptional depth

Portfolio coverage is static in depth

Portfolio is falling behind

40%

1.2 Portfolio’s ability to remain focused

Clarify & examine if portfolio focus on critical needs

Fully focused Adequately focused

Needs improvement

20%

1.3 Identification of emerging issues

Identify important new issues consistent with the portfolio mission

Contemporary & emerging issues identified

Missing some emerging issues

Needs coverage of important issues

20%

1.4 Integration of agency programs for portfolio

Demonstrate functional integration

REE fully integrated

Partially integrated

Insufficiently integrated

10%

1.5 Multi-disciplinary balance

Demonstrate disciplinary and scientific balance…

Extensive balance among relevant disciplines

Partial balance

Little balance 10%

Logic models Use strategic planning and

evaluation design features of logic models extensively in the self-review phase of the PREP process

Inputs Outputs

Activities- What we do

Outcomes-Impact

Short term Medium Term Long Term

Assumptions External Factors

EvaluationFocus - Collect Data – Analyze and Interpret - Report

What we invest

•Investments $

•Collaborative Ventures

•Researchers

•Labs & Facilities

Research:

•Conduct studies

•Design new techniques/ technologies

•Develop models

Dissemination:

•Publications

•Citations

•Patents

•Networking

What the short term results are

New Knowledge, technologies, techniques, products developed and commercialized

What the medium term results are

Adoption/ use of new knowledge, technologies, techniques, products

What the ultimate impact(s) is

•Needs met

•Problems solved

•Opportunitiesseized

Situation

Needs, Problems, and Opportunities

Priorities:

Consider:

Mission

Vision

Values

Mandates

Resources

Local Dynamics

Collaborators

Competitors

Intended Outcomes

Generic Research Logic Model Worksheet

• Conduct workshops, meetings• Deliver services• Develop products, curriculum, resources• Train• Provide counseling• Assess• Facilitate• Partner• Work with Media

Inputs Outputs

Activities Participation

Outcomes-Impact

Short term Medium Term Long Term

Assumptions External Factors

EvaluationFocus - Collect Data – Analyze and Interpret - Report

What we invest

• Staff

• Volunteers

• Time

• Money

• Research base

• Materials

• Equipment

• Technology

• Partners

What we do Who we reach

• Participants

• Clients

• Agencies

• Decision- makers

• CustomersSatisfaction

What the short term results are

Learning

Awareness

Knowledge

Attitudes

Skills

Opinions

Aspirations

Motivations

What the medium term results are

Action

Behavior

Practice

Decision-making

Policies

Social Action

What the ultimate impact(s) is

Conditions

Social

Economic

Civic

Environmental

Situation

Needs and Assets

Symptoms versus problems

Stakeholder engagement

Priorities:

Consider:

Mission

Vision

Values

Mandates

Resources

Local Dynamics

Collaborators

Competitors

Intended Outcomes

Generic Extension Logic Model Worksheet

Usefulness of Logic Models

Strategic Planning tool Programmatic tool that facilitates

management of programs Guide to develop indicators for

management feedback Delineates plan for causality and

attribution Guide for monitoring and evaluation

Use of Logic Models in PREP

LM used for portfolio, KA, and program levels to describe plans

Panelists found logic models useful in quickly comprehending portfolios

Allowed easy description of connection of program activities to outcomes

Use of Logic Models in CSREES

Currently not fully integrated into programmatic activities

Will be required for state formula-funded strategic planning in FY2006 submission

Training in use of logic models has been provided to leaders

CSREES Marine Shrimp Virus Research Logic Model

Situation Inputs Activities Outputs Outcomes

•Legislation: Marine Shrimp Farming Program goal is to reduce American multi-billion dollar trade deficit by developing U.S. shrimp farming industry

•Problem: New and virulent viral diseases threaten the US & world shrimp farming industry

•Research needed to mitigate viral threats

•Research needed on high-priority opportunities to develop a sustainable U.S. commercial shrimp farming industry.

US Marine Shrimp Farming Program (USMSP) funds$4 million per year$20 million over 5 year-period

• Human Capital:FacultyResearchersIndustry• Labs

Basic Research

Genetic research for:

•Identification of shrimp diseases

•Disease prevention

•Disease resistance

Applied research

Production:• New pedigree lines for disease resistance• New products

Marketing•Developed Marketing Strategies for shrimp

ExpandedKnowledge basere Marine Shrimp Viruses

Developed biosecurity protocol methods & diagnostic tests

•Marker-assistedSelection for genetic shrimp stock

•Developdiagnostic field kits for viruses

•Disseminate informationvia website

•Breeding strategies for SPF stocks and TSV resistance

•Use biosecurityand bestmanagementpractices

•Reducewater usageratesshrimpproduced

•Reduce Levels of contaminants (TSS, NH3 and CBDO) to lbs of shrimpsproduced

•New lines Created due toresearch

•Reduction of disease transfer

•Produce andDistribute Selected stocks

•Improved economic opportunityfor producers& communities

Short Medium Long

Assumptions- Joint efforts are likely to find better solutions;without the federal lead, research in this areas will not move forward; basic research in genetics is needed and will pay off in applications for combating disease; marketing research needed to complete commercialization

External Factors – Shrimp prices, global trade in shrimp broodstock, new shrimp virus outbreak, intensive vs. Extensive management/production facilities, environmental activists’ reactions, transportation issues

Dissemination:•Publications, citations, patents, Networking; industry briefs

Extension

Train producers in best management practices

Management of Forest and Rangelands Logic Model

Situation Inputs Activities Outputs Outcomes

In the 21st Century, the nation’s forest and grasslands face four threats:

•Fire and fuels•Invasive Species•Loss of open space•Unmanaged recreation

The purpose of this portfolio is to rehabilitate and restore treatment priorities where risks are the greatest; prevent and contain the spread of invasive species; slow the loss of open space due to development; and manage the use of national forest for outdoor activities to increase the sustainability of our national forests and benefits to the American people

FederalUSDAOther FederalCSREESState Appropriations

• Human Capital:FacultyResearchersIndustry• Labs

Basic Research

•Determine Influence of fungal associations on fitness of sibling beetles

•Determine if mycangial fungal associates change in nutrient quality of tissue

•Identify genes for Dutch Elm disease resistance in American Elm.

•Identify inhibitors that will reduce growth of branches and apical shoots.

Applied research

Invasive SpeciesNorth Dakota has developed a novel pesticide which usesherbivore and preferential grazing habits of sheep and cattle to offset the damage of leafy spurge

Develop semiochemical based monitoring and protective techniques to enhance integrated control strategy for pine tip moth

Calculate changes in regional biogeochemistry due to urban expansion during the past several decades

ExtensionOregon State University Cooperative Extension Service assisted ranchers in solving problems related to rangeland ownership and management

Four agencies assisted in the revitalization of coyote CreekExtension programs in cooperation with utility companies have encouraged the planting of trees around homes and buildings

Trained producers in the use of best management practices

Arizona Firewise Communities workshops conducted including homeowners, community leaders, fire professionals, insurance company representatives, real estate professionals and others

Short Medium Long-term

•Increased knowledge of rangelands for land owners and students•Invasive speciesChanged beliefs that pesticides must consist of invasive or abrasive agents•Identified 2 genes for Dutch Elm disease resistance•Discovered gibberellin synthesis inhibitors paclobutrazol and flurprimidol significantly reduced elongation growth•In Arizona, new community partnerships were developed to increase fire awareness

•Implemented new way to extend rangeland and environmental issues beyond a political frameworkInvasive SpeciesUtilized two species to ward off pest.

•Changes in technologies for controlling pests and diseases of urban trees, development of “designer” trees that can adapt to a wide range of urban environment

•Increased overall individual and community fire awareness and action

•Habitat is being restored at a low cost while biodiversity is reboundingRecreation & wilderness

Lawsuits have been reduced which in turn saves money

•A healthy, livable and sustainable urban forest ecosystem•Increased surveillance to guard against invasive species

•Identified promising technologies in fire protection

•Increased economic opportunities for natural resources

External Factors – Bio terrorism, invasive species, environmental disasters/concerns, high energy consumption by competitorsand global trade of wood products

Assumptions –; A well balanced ecosystem is an imperative backbone for the survival of our nation’s natural resources and environmental base.Natural resources play an important socio-economic role for current and future generations and need to be preserved. Source: Planning and Accountability, 2004

Use of Honeycomb Graphics Displays key players; roles and

coordination Address questions of overlap and

duplication across agencies/depts. Show topic area (Portfolio and KAs)

in context of world, U.S. partnerships, federal entities, USDA, and within CSREES

Acc

om

plis

hm

ents

Nee

ds

KA 212 Pathogens and Nematodes Affecting Plants

Detection and

Diagnosis

Prevention

Ecology

Biology

Management

Economics and Safety

Epidemiology

Prevention

Biology and Ecology

Detection & Diagnosis

Epidemiology

Management

Economic Sustainability and Safety

•Disease free seed and stock for some diseases

•Breeding resistant plants

•Novel types of resistance genes

•Marker assisted selection

•Ability to identify some unculturable organisms

•Characterizing take-all suppressive soils

•Ecology of vectors

•Genomic sequencing

•Molecular communication between plants and microbes

•Programmed cell death

•Pathogen’s genetic fingerprint used for rapid diagnosis

•Pathogenicity sequences identified

•Digital diagnosis / NPDN

•Seed propagation certification programs

•Quarantine programs

•Traditional detection methods

•Understanding dynamics of spread

•Understanding mechanisms of spread

•Forecasting of disease based on knowledge of environmental parameters

•Biological control –e.g., A. radiobacter, Trichoderma, nematodes

•Chemical control (pathogen and/or vector)

•Cultural practices, IPM

•Disease loss estimates

•Trade embargoes – international and interstate

•Safer pesticides, reduced use through IPM

Prevention

Biology and Ecology

Detection & Diagnosis

Epidemiology

Management

Economic Sustainability and Safety

•Isolate resistance genes, create resistance genes

•Interfere with mechanisms of signaling, pathogen’s virulence systems

•Durable resistance (understanding)

•Functional genomics, proteomics, bioinformatics

•Non-host resistance

•Publicly accessible databases for genome-enabled biology

•Ability to detect individuals within a microbial population

•Rapid / high-throughput methods of detection (user friendly/economical)

•Genomic reclassification of microbial taxonomy

•Culture collection and characterization, and specific DNA probes for identification

•Other pathogens genetic fingerprint for rapid diagnosis

•Other pathogenicity sequences

•Influence of global change on pathogen spread and disease establishment

•Accurate determination of disease origin

•BC agents – establishment, mechanisms, compatibility

•Alternatives to methyl bromide

•Post-harvest disease

•Chemical induction of resistance

•Chemical resistance management

•Better understanding of impacts of diseases

•Development of sustainable production practices

•Development of trace-back

KA 215

KA 213

KA 212

PA 214

KA 211KA 211

KA 216

Plant Protection Portfolio

Areas of Science

Acc

om

pli

shm

ents

Nee

ds

Marine Shrimp Viruses Honeycomb: a detailed program example from KAs 311-314

KA 315KA 315 NAHLN

KA 313

PA 314

KA 311

PA 312

•Specific pathogen-free and genetically improved shrimp stocks

•Biosecure systems including enclosed, reduced water exchange/increased water-reuse culture systems

•Biosecure management practices

•Co-operative industry-wide disease control strategies.

•Improved breeding strategies for SPF stocks and TSV resistance

•Marker assisted selection with identification of 10 ESTs for the linkage map with 160 markers

•Developed diagnostic field kits for disease prevention and treatment

•Reduced water usage rate from 4500 gal of each lb of shrimp to 250 gal per lb of shrimp produced

•Established nursery runs of 2-gram shrimp yielding up to 6.79 kg/m2

•USMSFP produced and distributed 227,626 selected stocks and potential broodstock and 1,211,000 Kona-line broodstock

•Distributed information through publications and USMSFP website

Marine Shrimp Virus

•Genetically improved stocks

•Quantitative genetics

•Gene markers and microsatellites

•Advanced disease diagnostic and treatment methods

•Biosecurity protocols

•Effects of shrimp farming on the environment

•Impact of effluent on receiving waters

Research Science Areas:

Logic model benefits

1. Improves management2. Facilitates evaluation3. Facilitates accountability

reporting and funding decisions

Our Experience Working With Agency Managers

Producing self review paper requires systematic collection and analysis of program data

Requires close collaboration between P&A staff and NPLs

Extremely demanding process given lack of readily available data

Our Experience Working With Expert Panels

Panelists were highly motivated, but new to the process

Panels require methodical review of process and expectations

Provide needed scores and recommendations Panelists grateful to have support staff to

assist and write panel report drafts P&A presence during panel deliberation

sessions extremely useful to clarify systems and issues

Summary

Self-assessment report referenced throughout the deliberation process

Inclusion of credible support data extremely important

Use of Logic Models and Portfolio graphics provide clear and concise maps of work

Summary (cont.) Meaningful outcome indicators for

OMB PART and BPI provided via scores from panel

Panelists provided useful recommendations

Managers have used PREP recommendations in planning, writing RFAs, budget, etc.

Summary (Cont.) PREP process provided panelists

with new insights in planning and accountability to spread around US

PREP improved panelists’ understanding and willingness to integrate their activities into the overall federal goals identified

P & A Future Directions Network Analysis & other

techniques to understand and eventually better support research system

Evaluation capacity building with agency staff and partners

Data availability: IT & Databases POW/ “One Solution”

P&A Contact Information Telephone: 202/ 720-5623 Fax: 202/ 720-7714 CSREES Web site:

www.csrees.usda.gov Email: coros@csrees.usda.gov