Overview of OPASI’s Division of Evaluation and Systemic Assessments (DESA) Deborah Guadalupe...
-
Upload
antonia-kennedy -
Category
Documents
-
view
225 -
download
1
description
Transcript of Overview of OPASI’s Division of Evaluation and Systemic Assessments (DESA) Deborah Guadalupe...
Overview of OPASI’sOverview of OPASI’sDivision of Evaluation Division of Evaluation
andandSystemic Assessments Systemic Assessments
(DESA)(DESA)Deborah Guadalupe Duran, Ph.D.Deborah Guadalupe Duran, Ph.D.
Chief, Systemic Assessments BranchChief, Systemic Assessments Branch
DESA: Division of Evaluation and Systemic Assessments • Evaluation Feedback • System Assessments (Sci of Sci)• Evaluability• Organization Performance
DRDA: Division of Resource Development and Analysis • Knowledge Management• Portfolio Analysis• Public Health Need• Centralized tools
DSC: Division of Strategic Coordination• Strategic Coordination-High Risk-High Impact• Trans-NIH Initiatives
gaps-potentials
data
-evi
denc
e
strengths-improvements
DRDADRDA
DSCDSC
DESADESA
OPASI Overview
Two branches
SystemicAssessments
Branch(SAB)
EvaluationBranch
(EB)
DESA Structure
(DESA)
Coordinate NIH wide evaluations and assessments
Provide technical assistance for required performance reporting
Facilitate the development of adaptive measures for science
Provide findings, integrated with portfolio analyses that foster scientific planning
SAB EB
DESA Mission
Evaluation Branch
Evaluate NIH-wide programs, key initiatives, and strategic objectives (formative and summative)
Track outcomes/recommendations
Maximize the effective use of the NIH Evaluation Set-Aside Program SAB EB
DESA - EB Mission
Systemic Assessments Branch Coordinate program performance activities across NIH
Comply and respond to federally-mandated performance reporting mechanisms• GPRA• PART• PAR• Scorecard
Foster the development of measures for • Adaptive science assessments• Organization assessments
Facilitate science of science activities
SAB EB
DESA – SAB Mission
• Strategic Plans System Level- HHS/IC/Office/Program
• Science of Science Management• Pre-evaluation Assessments• Visual Analytic Tools• e-Monitoring Systems
• Strategic Plans Project Level• Independent Evaluations• Evaluation Set-Aside Funds
• GPRA• PART• PAR• Audit• Biennial Report• Scorecard
Coordinating-Planning-Managing-Tracking-Reporting
SystemPerformance
& Assessments
IndependentExternal
EvaluationsRelated Processes
Resources
StrategicPlanning
& Evaluability
SAB SABEB
DESA Activities
Evaluation Branch Manage Evaluation Set-Aside Program which provides
dedicated funding for ICs and OD offices
Provide hands-on technical support and online guidance
Consult with ICs and OD offices on evaluation planning
Coordinate two trans-NIH committees to evaluate proposals
Report evaluation results SAB EB
DESA – EB Role
Systemic Assessments Branch Monitor NIH performance (not evaluation)
Report program performance information in lay language
Ensure Budget/Performance Integration
Participate in OMB / Secretary / HHS / Federal workgroups
Liaison to OMB, HHS, ICs performance issues
Develop System Assessments
Conduct internal assessments (evaluability)
Coordinate Science of Science activities
Strategic Planning
SAB EB
DESA – SAB Role
Systemic Assessments Branch: • Evaluability Assessments (internal)• Science of Science (internal)• Monitoring & Tracking Performance• Organization Performance Reporting
Evaluation Branch:• Funding for independent evaluations (external)
SystemicAssessments
BranchEvaluation
Branch
SAB/EB--related (ie IC Planning & Evaluation Community support) but unique functions
DESA Functions
DESA (SAB and EB) offer comprehensive evaluations/assessments – Internal and External
SAB–internal & organization assessments• Evaluability Assessments• Monitoring & Tracking Performance• System Assessments (Science of Science)• Adaptive evaluations for science
EB–external • Independent Evaluations of projects/programs/key initiatives
Internal(SAB)
External(EB)
Evaluations/Assessments
EvaluationExternal- best conducted by source outside
of the project/program
Types Formative and Summative Evaluations (Mid-term and End-of-Project Reviews) Needs assessments Process
Standard Methodologies
Standard Evaluation Processes
Program/project as the unit of analysis Upfront utilization focus / cause and effect Linear logic models Focus on achieving pre-planned goals Findings assessed in relation to plans Time specific Methodologies
• Static designs• Preference for quantitative data• Randomized Control Trials-Gold Standard
Conditions that Challenge Evaluation
High innovation Developmental High uncertainty Dynamic (complex) Emergent
SCIENCE and SCIENCE SYSTEMS
Science Programs
• New field• Developing theories & tools• Developing metrics & analytic methodologies• NIH among the leaders (national & international)
Program & Management
Organization
project
technologies
communication administration
program
humancapital
management
A pictorial representation System Assessments
System Premises
Systems neutrality: A system is functioning as observed for some reasons, fulfilling some functions.
In whose interests is a system functioning?
Who benefits?
What does the homeostasis offer?
In a well-functioning system…
no sub-system operates at maximum
dynamic/systems change• Relationships are interdependent and non-linear, rather than
simple and linear (cause-effect)• Transitions - worse before better
complex - science• Highly emergent (difficult to plan and predict)
Healthy System
Internal preparationfor external evaluations
Independentexternal evaluations
Organizational andsystem assessments
SAB/EB work together to coordinate assessments of program/project performance and NIH overall performance
SystemPerformance
& Assessments
IndependentExternal
EvaluationsRelated Processes
Resources
StrategicPlanning
& Evaluability
SAB SABEB
DESA Synergy
• Providing Technical Assistance• Teaching Strategic Planning course• Developing and implementing evaluability survey• Providing results feedback• Preparing for GPRA and PART• Creating visual analytic tools
• Offering resources• Providing technical assistance• Guiding application process• Reviewing proposals• Monitoring awards• Promoting innovative methodologies
• Monitoring and Reporting- PART- GPRA- Audits- PAR/MD&A
DESA--implemented through a number of steps/tasks
SystemPerformance
& Assessments
IndependentExternal
EvaluationsRelated Processes
Resources
StrategicPlanning
& Evaluability
SAB SABEB
DESA Implementation
Be prepared for •Evaluations•Performance reporting•Strategic planning
Utilize analysis tools for decision makers
Conduct•Needs Assessments• Feasibility Studies• Process evaluations• Outcome evaluations• Evidence related activities
Score well in• PART• GPRA• Audits• PAR/MD&A
SAB and EB supports ICs to
SystemPerformance
& Assessments
IndependentExternal
EvaluationsRelated Processes
Resources
StrategicPlanning
& Evaluability
SAB SABEB
DESA Outcomes
SAB SABEB
Demonstrates appropriate federal stewardship
Provides a systematic process for conducting evaluations/ assessments that produce evidence-based results, which can be used to improve or adapt programs and/or management of science comply with performance assessment requirements plan
StrategicPlanning
& Evaluability
IndependentExternal
EvaluationsRelated Processes
Resources
SystemPerformance
& Assessments
DESA Summation