Can Feds KISS?
description
Transcript of Can Feds KISS?
Can Feds KISS? Creating a Simple and Smart
Strategic Execution and Evaluation “System” Despite
Complex Mandates and Requirements
Environmental Evaluator’s Network Concurrent Sessions
June 23, 2011, 4-5 PMRoom 310
Liz Davenport and Thanh Vo DinhNational Oceanic and Atmospheric Administration
1
How a SMART planning, budgeting, execution, and evaluation system can better manage a complex portfolio
How better integration of performance monitoring, program evaluation, and assessing strategic progress can improve performance management
How Feds can KISS to ensure GPRA Modernization Act improves agency and government-wide performance
Network to Learn How Evaluation Can Aid an End-to-
End Management Process
2
NOAA manages about 12,800 employees, aided by about 4,000 contractors
NOAA has 6 Line Offices and 12 Staff Offices There are 127 mandates and authorities
applicable to NOAA’s mission NOAA’s responsibilities span all 50 states, Guam,
Puerto Rico, and the Pacific Island Region
Context for NOAA’s Strategy Execution and Evaluation (SEE)
Process
3
NOAA Strategic Plan
4
Here is SEE…under its first cycle of implementation...
What Is NOAA Creating in the Strategy Execution and Evaluation
Process?
5
6
1FocusAnnual
Guidance Memorandum
2Assess
Corporate Portfolio
3Request
Budget
4ReassessCorporate Portfolio
5Execute
Annual Operating
Plans
6EvaluateProgress
Request Budget to AchieveIntended Investment Outcomes
Evaluate Progress to Plan, Performance, Programs, and Strategic Priorities
Assess Portfolio to Plan Investments
Reassess Portfolio to Optimize Performance in
Operational Plans
Execute Operating Plans to Achieve Priority
Outcomes
AlignStrategic Plan
Implementation Plans
Strategy Execution & Evaluation Framework
Focus Attention to Achieve Priority Outcomes
From Compliance to Assimilation…Growing Utility of Evaluation
Compliance with performance evaluation requirements
Compliance and some performance data used to improve management
Compliance and sporadic use of requirements to “message” and “manage” performance to improve outcomes
Managers consistently use program evaluation requirements to define long-term outcomes, improve short-term success and achieve objectives
Agency success is strongly aided by assimilated program evaluation and a performance management framework that makes optimal use of performance data for management
7
Program Evaluation Requirement Driver
8
GPRA Modernization Act Authorizes Implementation to:
Foster coordinated, cross-cutting, government-wide planning ◦ Government-wide strategic goals with annual government-
wide (Federal) performance plan Engage Congress to identify management/performance
concerns◦ Agency strategic plans include Federal performance goals and
timed for release with changes in Administration Improve accountability for results and address
management weaknesses ◦ Agency and separate Federal performance plans aligned;
looks at past 5 years of performance data◦ Performance information better utilized due to greater
Administration and Congressional engagement ◦ Quarterly reviews reported at www.performance.gov
Adopt Standard, Federal Terms and Definitions Ensure Agency Clarity Regarding:
◦ Program◦ Performance Measurement◦ Performance Measures◦ Program Evaluation
Comply with New Standard:◦ Government-wide/cross-agency performance plan, with agreed
upon outcomes and accountability within a 2-year time frame
Performance Management and Evaluation “Unit” and “Paradigm”
9
Stop and Think!
10
How can we define “program” for agency performance planning and reporting and for Federal performance planning and reporting, given the need to link to and show progress in achieving Federal goals?
How can we do evaluations and use them to provide feedback to our Agency for these new requirements?
GAO Defined Types of Federal Program EvaluationsProcess (or Implementation) Evaluation – assesses extent to which a
program is operating as intended…assesses program activities’ conformance to statutory and regulatory requirements, program design, and professional standards or customer expectations
Outcome Evaluation – assesses extent to which a program achieves its outcome-oriented objectives…focuses on outputs and outcomes (including unintended effects) to judge program effectiveness but may also assess program process to understand how outcomes are produced
Impact Evaluation – (type of outcome evaluation) assesses net effect of a program by comparing program outcomes with an estimate of what would have happened in absence of the program…employed when external factors are known to influence program’s outcomes…to isolate program’s contribution to achievement of objectives
Cost-Benefit and Cost-Effectiveness Analyses – compare program’s outputs or outcomes with costs (resources expended) to produce them…when applied to existing programs, considered a program evaluation…cost-effectiveness analysis assesses the cost of meeting a single goal or objective and can be used to identify the least costly alternative for meeting that goal…aims to identify all relevant costs and benefits, usually expressed in dollar terms Performance Measurement and Evaluation, Definitions and Relationships, GAO 11-
616SP
Program Evaluation Typology
11
Can these program evaluations and performance measurements serve the new requirements at the same time serving Agency needs for management to meet strategic priorities? (National Ocean Policy…how will it be evaluated?)
Do we need anything else? Or what could be potential tools, approaches, or new paradigms for evaluation?
Stop and Think!
12
Theory of changes: Logic model for government-wide goals from collective actions
Performance accountability: www.performance.gov
Conflict resolution and coordination of capabilities between partners
Strategic evaluation in concert with performance measurement and program evaluation
Current Available Tools?
13
Thanks for letting us share SEE and speculate about GPRAMA impacts on evaluation
Thoughts, experiences , best practices on… How can we define program for agency performance planning and
reporting and for Federal performance planning and reporting given the need to link to and show progress in achieving Federal goals?
How can we do evaluations and use them to provide feedback to our Agency for these new requirements?
Can these program evaluations and performance measurements serve the new requirements at the same time serving Agency needs for management to meet strategic priorities? (National Ocean Policy…how will it be evaluated?)
Do we need anything else? Or what could be potential tools, approaches, or new paradigms for evaluation?
Discussion Questions
14
Backup Slides
15
What are your thoughts, experiences, best practices on…
1. GPRAMA requirements on your Agency’s evaluation and performance management?
2. Communicating evaluation results, using evaluation to facilitate, increase relationships/interaction with Congress and stakeholders?
3. Implementing GPRAMA requirements on evaluation within the current budget environment?
4. How are you building evaluation capacity? (Linking evaluation with performance management and evaluating success/failure of strategic goals/plans)?
Other Discussion Questions
16
• Effective compliance with evaluation requirements• Learning culture through evaluation feedback• Robust program management• Better investments through adaptive management• Valuable decision-making support tool• Transparent performance, outcomes, and results• Monitor progress toward achieving strategic goals • SMART performance targets• Improved accountability
Program Evaluation Framework Benefits
17
GPRAMA Agency Strategic Plan Changes That May Impact Evaluations
Former Process1. Revised at least every 3 years2. Covered at least a 5-year period3. Consulted with Congress4. Involved other stakeholders5. Submitted to OMB and Congress
Amended Process1. Revision every 4 years, approximately 1
year after a new Presidential term begins
2. Consultations with relevant Congressional committees at least every 2 years
3. Publication online and President and Congress notified
Amended Content4. Relationship to Federal government
priority goals5. Interagency coordination and
collaboration6. Identification of agency priority goals7. Description of incorporation of
Congressional input
18
Former Content 1. Mission statement 2. Strategic goals3. Strategies and resources4. Relationship to performance
goals5. External factors that could
significantly affect the goals6. Program evaluations
GPRAMA Agency Performance Plan Changes That May Impact Evaluations
Former Process1. Upcoming fiscal year2. No set time frame by OMB3. Submitted to OMB
Amended Process1. Covers 2 fiscal years
(current/upcoming)2. Concurrent with President’s Budget3. Available online; President and
Congress notified
19
Former Content1. Performance goals in objective,
quantifiable, measurable form2. Cover all program activities3. Strategies and resources4. Performance measures5. Basis for comparing actual
results with performance goals6. Means to verify/validate data
Amended Content1. Relationship to agency strategic and
priority goals, and Federal government performance goals
2. Coordination and collaboration3. Contributing programs and activities4. Milestones5. Goal leaders6. Balanced set of measures7. Data accuracy and reliability8. Major management challenges9. Low-priority programs
GPRAMA Agency Performance Plan Changes That May Impact Evaluations
GPRA Modernization Act…Agency Quarterly Priority Progress Reviews◦ At least quarterly, agency head, COO, and PIC
review with goal leaders progress toward each priority goal
◦ Review involves contributors in and out of Agency: Assess how programs and activities contribute to goal Categorize goals by risk of not being achieved For those at risk, identify strategies to improve
performance
20
GPRAMA Agency Performance Plan Changes That May Impact Evaluations
GPRA Modernization Act…Requires High Performance Goals with Expanded Performance Reporting◦ Set quarterly progress reviews◦ Require Congressional consultation and
engagement in performance management◦ Require a Performance Portal◦ Reduce duplicative performance reporting by
10%
21
GPRAMA Agency Performance Plan Changes That May Impact Evaluations GPRA Modernization Act Senate
Committee Suggests:◦Agency, OMB, and Relevant Congressional
Committee Review of Agency Meeting Goals
◦ If Goals Are Not Met: After 1st FY, require a Goal Performance Improvement
Plan If not met after 2nd FY, submit actions to improve
performance (statutory changes, resource transfer, etc.)
If not met after 3rd consecutive FY, must act within 60 days or face statutory changes or possible termination or reduction possible
22
GPRAMA Agency Performance Plan Changes That May Impact Evaluations
23
1. Strategic Plan must include how program evaluations were used to establish or revise strategic goals and a schedule for future evaluations
2. Performance reports must include summary findings of program evaluations completed during the fiscal year covered by the report
3. Evaluation will benefit performance planning and goal setting as well as quarterly reviews of priority goals