Office of Research and Development November 7th, 2008

Randy WentselNational Program Director

Moving Toward Research Excellence

Presentation to ORD Managers

Office of Research and Development 2

Overview of Briefing

•Role of NPDs•Research excellence•Decision analysis tools•Evaluation of our research•Communication of our research•Accountability•Q & A

04/20/23 3

Role of National Program Directors

• Lead planning and development of research programs

• Cross laboratory view for shifts in program• Primary interface with program office and

regions• Scientific leader - responsible for scientific

direction for specific areas of the research program

• Report to DAA for Science

04/20/23 4

Planning and Development of Research Programs

• Lead Research Coordination Team (RCT) or similar group

• Lead annual research planning• Lead drafting of Multi-year Plans

(MYPs)• Lead Board of Scientific Counselors

reviews• Lead PART review• Lead science discussions with SAB

04/20/23 5

How ORD Evolves its Research Program

NPDs

Decide What Research Area-Specific Work We Do and When We Do It

Planning the Program

Decision Inputs

Programs and Regions (RCTs)

EPA Strategic Plan

Administration’s priorities

Congressional mandates

BOSC Reviews

SAB, NAS, other external advice

Stakeholders

NPDs, SC, MC, EC

ORD Executive Council

Corporate Decisions on

What We Do . . . and . . . How We Do It

Evaluation

Program and Regional Office Feedback

BOSC Program Evaluations

NAS, NAPA, and other advisory bodies

PART Reviews

Implementing the Program

L/C Directors

Decide How ORD Produces its Research

Products

NPDs

Responsible for Communicating Products

to Clients

L/C Directors

Responsible for Developing ORD’s Research Products

04/20/23 6

ORD Programs and NPDswww.epa.gov/ord

Goal 1 - Air (Dan Costa)• NAAQS• Air Toxics

Goal 2 - Water• Drinking Water (Audrey Levine)• Water Quality (Chuck Noss)

Goal 5 - Stewardship Sustainability & Biofuels (Alan Hecht)

Goal 3 - Land (Randy Wentsel)

• Land Protection & Restoration

Goal 4 – Safe Communities and Healthy Ecosystems• Homeland Security (Greg Sayles)• Human Health (Sally Darney) & Ecosystems (Rick Linthurst)• Human Health Risk Assessment (John Vandenberg)• Computational Toxicology (Robert Kavlock)• Endocrine Disrupting Chemicals (Elaine Francis)• Global Change (Joel Scheraga)• Pesticides & Toxics (Elaine Francis)• Nanotechnology (Jeff Morris)

04/20/23 7

The Formula for Research Quality

Quality Scientists/Engineers •Creative, Innovative

•Broadly Trained

•Motivated

Quality Facilities/Equipment •Conductive to Quality Output

•Attracts High Quality S&E’s

Quality Leadership/Management•Environment for Innovation

•Stress Team-Work

•Provides Relevance and Timelines

Quality Research

Sustained Quality Leads to a Brand or Go To Status

•Potential to Make a Real Difference

04/20/23 8

The Formula for Relevant Research

Meets Needs of Decision Makers •Multiple levels of communication

•Provides high quality science

•Informs decisions

Timely Knowledge Transfer •Meets client schedule

•Responsive to requests for assistance

•Enables Tech Transfer

Strong Leadership/Management •Environment for collaboration with customers

•Focus on outcomes

Relevant Research

Sustained Relevance Leads to a Brand or Go To Status

•Makes a Real Difference

04/20/23 9

Research Excellence = f(Technical Quality x Relevance)

Incr

easin

g ex

celle

nce

Job shopping

Ivory tower

Rel

evan

ce

Technical Quality

ResearchExcellence

Research Excellence – How To Measure It?

04/20/23 10

Implementing an ORD Evaluation Framework

Research

Activities

Research

Outputs

Short-Term

Outcomes

Intermediate

Outcomes

Long-Term

Outcomes

Resources

Strategic Goals

Strategic Objectives

Research Program Environmental Outcomes and Results Clients

Independent Expert Evaluation:

Focus: Outcome-oriented progress; R&D investment criteria

Evidence: MYPs; synthesis products; performance data; client feedback

Attribution: Sphere of influence that includes ORD, EPA clients & partners

Policy

Development

04/20/23 11

BOSC Program Evaluations

Help answer the questions: • Are we doing the right science?• Are we doing the science right?

Provide guidance for evolving the research program

Provide evidence for OMB evaluations using the Performance Assessment Rating Tool

04/20/23 12

Program Assessment Rating Tool(PART)

– Required program “audit” by OMB is likely to change, but not go away

– Emphasis has been on 4 areas: Purpose/Design, Strategic Planning, Program Management, and Program Results

– Ratings/scores based largely on meeting annual and long-term performance goals with emphasis on outcomes, and feedback from external program evaluations like BOSC

04/20/23 13

Decision Analysis Tools

• Value of Information is a form of decision analysis that evaluates the benefits of information for reducing uncertainty in a decision making context

• Multi-Criteria Decision Analysis reduces complex problems to a singular basis in order to select a preferred alternative

• These tools may be applied to research planning and/or communicating the value of our research to BOSC, OMB, Congress

4

04/20/23 14

Communication of our research accomplishments

• Performance Measurement and Evaluation– Suite of metrics for quality (bibliometrics), relevance

(client citation), and performance (client use)– Outputs and outcomes, annual and long-term– Partner surveys = feedback and areas for

improvement– Independent peer review = evaluation and

improvement strategies– Informs decisions = impact on human health and the

environment

04/20/23 15

Communication Challenges

• Issues– Complete database of ORD publications that partners

can use– Demonstrating use/impact of research – faster, cheaper,

better science– How do we document impacts?– Value of ORD research

• To who?– Scientific community– Partners (POs, regions, user community)– Congress, OMB, policymakers

Performance Management Challenges

– How can we improve our effectiveness as a team (line mgt & NPDs)?

– What are methods to improve our communication to: the scientific community, partners (POs, regions, user community), Congress, OMB, and policymakers?

– Death by Accountability--GPRA/BOSC/PART/Division-Reviews--How can we measure and evaluate performance, and ensure that ALL individuals are working toward the mission?

– If ORD is to place emphasis on integrated, multidisciplinary programs, what 2-3 specific actions can division directors and NPDs take together to foster such programs?