STRATEGIC PLANNING IN AN EVIDENCE BASED ENVIRONMENT JESSICA NAPIER JUSTICE CENTER FOR EVIDENCE BASED...

STRATEGIC PLANNING IN AN EVIDENCE BASED ENVIRONMENT

JESSICA NAPIERJUSTICE CENTER FOR EVIDENCE BASED PRACTICE

Developed for WV SAG Committee RetreatsSeptember 22, 2011

Discussion Points

Evidence Based Practice: the basics What is evidence based practice? What is fidelity? Benefits of EBP

Grant Administration in an Evidence Based Environment The Grant Process Performance Measurement vs. Evaluation Key Terms

Reflection Transitioning to EBP

Evidence Base Practices: the basics

Evidence Based Practices: the basics

What is Evidence Based Practices? Evidence Based Practice (EBP) – definitions vary

widely, but there are guidelines or criteria that can be used to assess the quality of evidence “Evidence‐based practice (EBP) is the objective, balanced,

and responsible use of current research and the best available data to guide policy and practice decisions, such that outcomes for consumers are improved.”1

“A fundamental premise of the evidence-based movement is that the information used to establish what works must be trustworthy and credible.” (p.12) 2

Evaluations that use the scientific method (randomization, control and experiment groups, etc.) provide the most evidence about a program’s effectiveness.

1 Information obtained from DOJ; Institute of Corrections from http://cjinstitute.org/files/Community_Corrections_BoxSet_Oct09.pdf p. 11‐13.2 Przybylski’s (2008) What Works: Effective Recidivism Reduction and Risk-Focused Prevention Programs prepared for the Commission on Criminal and Juvenile Justice


Levels of

Research

Pyramid

Pyramid example adapted from: Pierce County Council Performance Audit Office http://www.co.pierce.wa.us/xml/abtus/plans/perf-audit/Evidence-Based_Practices_Report_Feb_27_2008.pdf

Evidence Based Practice Key Terms Fidelity – relates to “how well” a program was

implemented Program effectiveness

Performance measurements Feedback

Program integrity? Was program or model implemented as it was

designed? Innovation

Components or characteristics of successful/effective programs

Based on theory


Benefits of EBP Budget cuts = funding cuts by committees Want biggest bang for buck

Aids in making financial decisions Use practices that have proven to be effective

Data driven decision making allows for more efficient spending of funds

Fosters collaboration among programs Partnerships can be beneficial and save funds Productive partnerships also allow for programs

to apply for funds they normally would not have qualified for also.


Benefits of EBP (continued) Sustainability Plans

Place a limit on the amount of time or funds available to a program

Grantees to seek aid from other funding sources

Allows for more programs to receive funds over time therefore increasing the number of programs available to the public (funds are not wrapped up in same programs)

More funds for more programs More funds for more services

Federal Requirement (now or in future)


Grant Administration in an Evidence Based Environment


Starts at the beginning thinking through to the end Educate grantees about what is to be

expected in the application process Make resources available to grantees prior to

and during the Request for Proposal process Make use of scientific research on priority

areas, such as journals, government statistics, legitimate online resources These could be summarized in literature

reviews

Starts at the beginning thinking through to the end (continued) Review the federal required mandates for a

grant, and get to know the priority areas. Look at what is historically funded and why it is

funded Look at programs that receive the most funding,

do those programs need to be evaluated to determine if they are being effective?

Market the grants to a broader population Include a researcher in the development of the

application and/or RFP


Starts at the beginning thinking through to the end (continued) What goes in the Application/RFP: What are the

components? Lay out exactly what is expected and under what terms

Purpose of RFP – What is the problem? Miniature summary of literature – Statement of the

problem Budget guidelines Specific data collection guidelines (uniform instruments

will allow for comparisons across programs) Better performance measures = better data analysis

Monitoring requirements Sustainability plans Logic Model


Starts at the beginning thinking through to the end (continued) Scoring system for Applications/RFP Provide feedback to applicants so they know how

to improve their application in future Overall items to think about

Infusion of expertise Program staff could provide committee with

resources/literature reviews Involve researchers in the committee(s) and

throughout the grant process What if there is no research available on subject How do you find literature on a subject?


Performance Measurements vs. Evaluation

What are performance measurements?*

Concerned with gathering information to determine if a program is meeting its objectives

Focus on reporting program outcomes Information gathered can be used to improve program

What are evaluations?*

Assesses the effectiveness of a program in achieving its objectives

“Determines whether program outcomes(s) can be attributed to the program or other factors”

“Aims at program improvement through modification of program operation and/or design”

* Poulin, M. E., & Williamson. (2011). Evaluation and performance measurement 101: An introduction to key concepts. Justice Research and Statistics Association.

Performance Measurement vs. Evaluation

Performance Measurement vs. EvaluationAnswers the questions…

Looks to answer: Is the target

population being served?

Is implementation going as planned?

Are stated objectives being accomplished?

Looks at the same questions as performance measurement, plus… Can outcomes be

attributed to the program?

“Did one intervention have a greater impact on outcomes than another intervention?”

Is this program effective?

Performance Measurement * Evaluation *


Performance Measurement vs. EvaluationSimilarities and Differences

Information can be used for monitoring & improvement

There are a variety of ways and methods to design performance measurements & evaluations

Performance measurement generates evaluative information, but typically quantitative.

Evaluations use means to determine whether the outcomes of the program can be attributed to the program or other factors

Comparison or control groups Random assignment or

treatment Evaluation provides more

information regarding the programs effectiveness

Evaluations closely examine the quality of a program.

Similarities * Differences *



Why do Performance Measurements? *

Identify which grantees are meeting objectives

Facilitates agency planning and budgeting Helps to select the best grantee to meet a

service need Keeps programs on track with their

implementation Federal requirement


Evaluation time What is the point?

Uses performance measures to determine if goals & objectives are being met

Uses performance measures to provide information regarding program effectiveness or outcomes

What to expect? Evaluations use means to determine whether the

outcomes of the program can be attributed to the program or other factors 1

Unbiased evaluators; early in process Approximately 5% of Total Funding Stream Feedback/Data


Performance Measurement: A closer look

Elements of Well-Described Performance Measure *

Over the life of the grant, increase of 10% of participants that successfully completed program. Baseline will be the percent of program participants who successfully completed the preceding year. Successful program completion refers to individual who completes all the program requirements and pass the graduation exam.

* Poulin, M. E. (2010) Incorporating performance measurement and evidence-based practices in grants management. Justice Research and Statistics Association.


Over the life of the grant, increase of 10% of

participants that successfully completed program. Baseline will be the percent of program participants who successfully completed the preceding year. Successful program completion refers to individual who completes all the program requirements and pass the graduation exam.

Time frame Direction Target

Base

line

Key

Term


With Data Collection… Tells us whether goals are being met Aides in strategic planning

Tells us if funds are being spent wiselyIf needs are being met

Provides a metric for success/failure and basis for comparison (standardized)

Helps identify problems with implementation and if problems need to be addressed

With Data Collection…(continued) Self-assessment (allows program to

see progress or areas to improve) What gets measured gets done!

If results are not measured, success cannot be distinguished from failures.

If successes cannot be distinguished, they cannot be replicated.

If failures can not be identified, they cannot be corrected.

If success cannot be demonstrated, support cannot be secured.


Key Terms Associated with Performance Measurement

Process Measures (Output) Outcome Measures Goals Objectives Activities

Key Terms Examined

Process Measures (Output) Addresses program implementation;

directly tied to the activities.

Outcome Measurements Addresses the program accomplishments;

directly tied to the objectives.

The more specific the measure, the more reliable it will measure what it says it will measure.

Key Terms Examined1

Goals*

What is the end result of a project or what is anticipated to happen (outcome)

Conceptual and abstract Objectives *

Outcomes that are tangible, concrete, specific, measureable, and achievable, with a time element

Activities Is only one way to accomplish what you have

written. If your answer is yes, you have probably written an activity. Plan to do = Activity Want to do…= Objective* College of DuPage. Creating goals, objective, and activities. Available at http://www.cod.edu/grants/Creating_Goals.pdf

http://www.cod.edu/grants/Creating_Goals.pdf

http://www.cod.edu/grants/Creating_Goals.pdf

Reflection

Reflection

Looking to the future: Things to Consider Think about the desired outcome Start with the RFP

Infuse expertise into the grant process Lay out requirements in RFP

Educate applicants Offer online trainings if necessary

May consider a webinar or podcast for grant writing

Educational webinar or podcast for applicants on EBP

Make resources available via CD or online forum for applicants (up-to-date and cost efficient)

Reflection

Looking to the future: Things to Consider Fidelity Performance Measurements

Qualitative and Quantitative Make it meaningful Measure what you want to know

Data! What gets measured gets done If you are going to require data to be submitted, data

should be utilized, feedback should be given, reports (comparative: by agency or program)

Online resources!!!! Data requests can be submitted via online survey tools

into an excel spreadsheet

Reflection

Evaluation Does a program do what it says it will do? Unbiased researcher Timely & requires specific funds

Critical to determine if program is effective Must have measures in place from the

beginning

Transitioning to EBP: Example

Basic data request Filed away

Prevention? Instructional Time

No lesson plans Not EBP

Law Enforcement Presence

Expand data collection Update form Create database Make available online

Pre\Post test Satisfaction Survey Using EBP lessons

Standardization “what works” Some are little to no cost

to PRO Online available

Current PRO Adding SOME EBP

Conclusion

CD Grant Writing Resources Evidence Based Practices Resources Other Documents and Publications

Contact Information

[email protected]

558-8814 ext 53325

[email protected]

558-8814 ext 53338

Jessica Napier Stephen M. Haas

Important links:www.djcs.wv.gov

www.facebook.com/wvcjsacwww.twitter.com/wvcjsac

mailto:[email protected]




http://www.djcs.wv.gov/

http://www.facebook.com/wvcjsac

http://www.twitter.com/wvcjsac

Thank you

STRATEGIC PLANNING IN AN EVIDENCE BASED ENVIRONMENT JESSICA NAPIER JUSTICE CENTER FOR EVIDENCE BASED...

Documents

Transcript of STRATEGIC PLANNING IN AN EVIDENCE BASED ENVIRONMENT JESSICA NAPIER JUSTICE CENTER FOR EVIDENCE BASED...