STRATEGIC PLANNING IN AN EVIDENCE BASED ENVIRONMENT JESSICA NAPIER JUSTICE CENTER FOR EVIDENCE BASED...

34
STRATEGIC PLANNING IN AN EVIDENCE BASED ENVIRONMENT JESSICA NAPIER JUSTICE CENTER FOR EVIDENCE BASED PRACTICE Developed for WV SAG Committee Retreats September 22, 2011
  • date post

    19-Dec-2015
  • Category

    Documents

  • view

    213
  • download

    0

Transcript of STRATEGIC PLANNING IN AN EVIDENCE BASED ENVIRONMENT JESSICA NAPIER JUSTICE CENTER FOR EVIDENCE BASED...

Page 1: STRATEGIC PLANNING IN AN EVIDENCE BASED ENVIRONMENT JESSICA NAPIER JUSTICE CENTER FOR EVIDENCE BASED PRACTICE Developed for WV SAG Committee Retreats September.

STRATEGIC PLANNING IN AN EVIDENCE BASED ENVIRONMENT

JESSICA NAPIERJUSTICE CENTER FOR EVIDENCE BASED PRACTICE

Developed for WV SAG Committee RetreatsSeptember 22, 2011

Page 2: STRATEGIC PLANNING IN AN EVIDENCE BASED ENVIRONMENT JESSICA NAPIER JUSTICE CENTER FOR EVIDENCE BASED PRACTICE Developed for WV SAG Committee Retreats September.

Discussion Points

Evidence Based Practice: the basics What is evidence based practice? What is fidelity? Benefits of EBP

Grant Administration in an Evidence Based Environment The Grant Process Performance Measurement vs. Evaluation Key Terms

Reflection Transitioning to EBP

Page 3: STRATEGIC PLANNING IN AN EVIDENCE BASED ENVIRONMENT JESSICA NAPIER JUSTICE CENTER FOR EVIDENCE BASED PRACTICE Developed for WV SAG Committee Retreats September.

Evidence Base Practices: the basics

Page 4: STRATEGIC PLANNING IN AN EVIDENCE BASED ENVIRONMENT JESSICA NAPIER JUSTICE CENTER FOR EVIDENCE BASED PRACTICE Developed for WV SAG Committee Retreats September.

Evidence Based Practices: the basics

What is Evidence Based Practices? Evidence Based Practice (EBP) – definitions vary

widely, but there are guidelines or criteria that can be used to assess the quality of evidence “Evidence‐based practice (EBP) is the objective, balanced,

and responsible use of current research and the best available data to guide policy and practice decisions, such that outcomes for consumers are improved.”1

“A fundamental premise of the evidence-based movement is that the information used to establish what works must be trustworthy and credible.” (p.12) 2

Evaluations that use the scientific method (randomization, control and experiment groups, etc.) provide the most evidence about a program’s effectiveness.

1 Information obtained from DOJ; Institute of Corrections from http://cjinstitute.org/files/Community_Corrections_BoxSet_Oct09.pdf p. 11‐13.2 Przybylski’s (2008) What Works: Effective Recidivism Reduction and Risk-Focused Prevention Programs prepared for the Commission on Criminal and Juvenile Justice

Page 5: STRATEGIC PLANNING IN AN EVIDENCE BASED ENVIRONMENT JESSICA NAPIER JUSTICE CENTER FOR EVIDENCE BASED PRACTICE Developed for WV SAG Committee Retreats September.

Evidence Based Practices: the basics

Levels of

Research

Pyramid

Pyramid example adapted from: Pierce County Council Performance Audit Office http://www.co.pierce.wa.us/xml/abtus/plans/perf-audit/Evidence-Based_Practices_Report_Feb_27_2008.pdf

Page 6: STRATEGIC PLANNING IN AN EVIDENCE BASED ENVIRONMENT JESSICA NAPIER JUSTICE CENTER FOR EVIDENCE BASED PRACTICE Developed for WV SAG Committee Retreats September.

Evidence Based Practice Key Terms Fidelity – relates to “how well” a program was

implemented Program effectiveness

Performance measurements Feedback

Program integrity? Was program or model implemented as it was

designed? Innovation

Components or characteristics of successful/effective programs

Based on theory

Evidence Based Practices: the basics

Page 7: STRATEGIC PLANNING IN AN EVIDENCE BASED ENVIRONMENT JESSICA NAPIER JUSTICE CENTER FOR EVIDENCE BASED PRACTICE Developed for WV SAG Committee Retreats September.

Benefits of EBP Budget cuts = funding cuts by committees Want biggest bang for buck

Aids in making financial decisions Use practices that have proven to be effective

Data driven decision making allows for more efficient spending of funds

Fosters collaboration among programs Partnerships can be beneficial and save funds Productive partnerships also allow for programs

to apply for funds they normally would not have qualified for also.

Evidence Based Practices: the basics

Page 8: STRATEGIC PLANNING IN AN EVIDENCE BASED ENVIRONMENT JESSICA NAPIER JUSTICE CENTER FOR EVIDENCE BASED PRACTICE Developed for WV SAG Committee Retreats September.

Benefits of EBP (continued) Sustainability Plans

Place a limit on the amount of time or funds available to a program

Grantees to seek aid from other funding sources

Allows for more programs to receive funds over time therefore increasing the number of programs available to the public (funds are not wrapped up in same programs)

More funds for more programs More funds for more services

Federal Requirement (now or in future)

Evidence Based Practices: the basics

Page 9: STRATEGIC PLANNING IN AN EVIDENCE BASED ENVIRONMENT JESSICA NAPIER JUSTICE CENTER FOR EVIDENCE BASED PRACTICE Developed for WV SAG Committee Retreats September.

Grant Administration in an Evidence Based Environment

Page 10: STRATEGIC PLANNING IN AN EVIDENCE BASED ENVIRONMENT JESSICA NAPIER JUSTICE CENTER FOR EVIDENCE BASED PRACTICE Developed for WV SAG Committee Retreats September.

Grant Administration in an Evidence Based Environment

Starts at the beginning thinking through to the end Educate grantees about what is to be

expected in the application process Make resources available to grantees prior to

and during the Request for Proposal process Make use of scientific research on priority

areas, such as journals, government statistics, legitimate online resources These could be summarized in literature

reviews

Page 11: STRATEGIC PLANNING IN AN EVIDENCE BASED ENVIRONMENT JESSICA NAPIER JUSTICE CENTER FOR EVIDENCE BASED PRACTICE Developed for WV SAG Committee Retreats September.

Starts at the beginning thinking through to the end (continued) Review the federal required mandates for a

grant, and get to know the priority areas. Look at what is historically funded and why it is

funded Look at programs that receive the most funding,

do those programs need to be evaluated to determine if they are being effective?

Market the grants to a broader population Include a researcher in the development of the

application and/or RFP

Grant Administration in an Evidence Based Environment

Page 12: STRATEGIC PLANNING IN AN EVIDENCE BASED ENVIRONMENT JESSICA NAPIER JUSTICE CENTER FOR EVIDENCE BASED PRACTICE Developed for WV SAG Committee Retreats September.

Starts at the beginning thinking through to the end (continued) What goes in the Application/RFP: What are the

components? Lay out exactly what is expected and under what terms

Purpose of RFP – What is the problem? Miniature summary of literature – Statement of the

problem Budget guidelines Specific data collection guidelines (uniform instruments

will allow for comparisons across programs) Better performance measures = better data analysis

Monitoring requirements Sustainability plans Logic Model

Grant Administration in an Evidence Based Environment

Page 13: STRATEGIC PLANNING IN AN EVIDENCE BASED ENVIRONMENT JESSICA NAPIER JUSTICE CENTER FOR EVIDENCE BASED PRACTICE Developed for WV SAG Committee Retreats September.

Starts at the beginning thinking through to the end (continued) Scoring system for Applications/RFP Provide feedback to applicants so they know how

to improve their application in future Overall items to think about

Infusion of expertise Program staff could provide committee with

resources/literature reviews Involve researchers in the committee(s) and

throughout the grant process What if there is no research available on subject How do you find literature on a subject?

Grant Administration in an Evidence Based Environment

Page 14: STRATEGIC PLANNING IN AN EVIDENCE BASED ENVIRONMENT JESSICA NAPIER JUSTICE CENTER FOR EVIDENCE BASED PRACTICE Developed for WV SAG Committee Retreats September.

Performance Measurements vs. Evaluation

Page 15: STRATEGIC PLANNING IN AN EVIDENCE BASED ENVIRONMENT JESSICA NAPIER JUSTICE CENTER FOR EVIDENCE BASED PRACTICE Developed for WV SAG Committee Retreats September.

What are performance measurements?*

Concerned with gathering information to determine if a program is meeting its objectives

Focus on reporting program outcomes Information gathered can be used to improve program

What are evaluations?*

Assesses the effectiveness of a program in achieving its objectives

“Determines whether program outcomes(s) can be attributed to the program or other factors”

“Aims at program improvement through modification of program operation and/or design”

* Poulin, M. E., & Williamson. (2011). Evaluation and performance measurement 101: An introduction to key concepts. Justice Research and Statistics Association.

Performance Measurement vs. Evaluation

Page 16: STRATEGIC PLANNING IN AN EVIDENCE BASED ENVIRONMENT JESSICA NAPIER JUSTICE CENTER FOR EVIDENCE BASED PRACTICE Developed for WV SAG Committee Retreats September.

Performance Measurement vs. EvaluationAnswers the questions…

Looks to answer: Is the target

population being served?

Is implementation going as planned?

Are stated objectives being accomplished?

Looks at the same questions as performance measurement, plus… Can outcomes be

attributed to the program?

“Did one intervention have a greater impact on outcomes than another intervention?”

Is this program effective?

Performance Measurement * Evaluation *

* Poulin, M. E., & Williamson. (2011). Evaluation and performance measurement 101: An introduction to key concepts. Justice Research and Statistics Association.

Page 17: STRATEGIC PLANNING IN AN EVIDENCE BASED ENVIRONMENT JESSICA NAPIER JUSTICE CENTER FOR EVIDENCE BASED PRACTICE Developed for WV SAG Committee Retreats September.

Performance Measurement vs. EvaluationSimilarities and Differences

Information can be used for monitoring & improvement

There are a variety of ways and methods to design performance measurements & evaluations

Performance measurement generates evaluative information, but typically quantitative.

Evaluations use means to determine whether the outcomes of the program can be attributed to the program or other factors

Comparison or control groups Random assignment or

treatment Evaluation provides more

information regarding the programs effectiveness

Evaluations closely examine the quality of a program.

Similarities * Differences *

* Poulin, M. E., & Williamson. (2011). Evaluation and performance measurement 101: An introduction to key concepts. Justice Research and Statistics Association.

Page 18: STRATEGIC PLANNING IN AN EVIDENCE BASED ENVIRONMENT JESSICA NAPIER JUSTICE CENTER FOR EVIDENCE BASED PRACTICE Developed for WV SAG Committee Retreats September.

Performance Measurement vs. Evaluation

Why do Performance Measurements? *

Identify which grantees are meeting objectives

Facilitates agency planning and budgeting Helps to select the best grantee to meet a

service need Keeps programs on track with their

implementation Federal requirement

* Poulin, M. E., & Williamson. (2011). Evaluation and performance measurement 101: An introduction to key concepts. Justice Research and Statistics Association.

Page 19: STRATEGIC PLANNING IN AN EVIDENCE BASED ENVIRONMENT JESSICA NAPIER JUSTICE CENTER FOR EVIDENCE BASED PRACTICE Developed for WV SAG Committee Retreats September.

Evaluation time What is the point?

Uses performance measures to determine if goals & objectives are being met

Uses performance measures to provide information regarding program effectiveness or outcomes

What to expect? Evaluations use means to determine whether the

outcomes of the program can be attributed to the program or other factors 1

Unbiased evaluators; early in process Approximately 5% of Total Funding Stream Feedback/Data

Performance Measurement vs. Evaluation

Page 20: STRATEGIC PLANNING IN AN EVIDENCE BASED ENVIRONMENT JESSICA NAPIER JUSTICE CENTER FOR EVIDENCE BASED PRACTICE Developed for WV SAG Committee Retreats September.

Performance Measurement: A closer look

Elements of Well-Described Performance Measure *

Over the life of the grant, increase of 10% of participants that successfully completed program. Baseline will be the percent of program participants who successfully completed the preceding year. Successful program completion refers to individual who completes all the program requirements and pass the graduation exam.

* Poulin, M. E. (2010) Incorporating performance measurement and evidence-based practices in grants management. Justice Research and Statistics Association.

Page 21: STRATEGIC PLANNING IN AN EVIDENCE BASED ENVIRONMENT JESSICA NAPIER JUSTICE CENTER FOR EVIDENCE BASED PRACTICE Developed for WV SAG Committee Retreats September.

Performance Measurement: A closer look

Over the life of the grant, increase of 10% of

participants that successfully completed program. Baseline will be the percent of program participants who successfully completed the preceding year. Successful program completion refers to individual who completes all the program requirements and pass the graduation exam.

Time frame Direction Target

Base

line

Key

Term

Page 22: STRATEGIC PLANNING IN AN EVIDENCE BASED ENVIRONMENT JESSICA NAPIER JUSTICE CENTER FOR EVIDENCE BASED PRACTICE Developed for WV SAG Committee Retreats September.

Performance Measurement: A closer look

With Data Collection… Tells us whether goals are being met Aides in strategic planning

Tells us if funds are being spent wiselyIf needs are being met

Provides a metric for success/failure and basis for comparison (standardized)

Helps identify problems with implementation and if problems need to be addressed

Page 23: STRATEGIC PLANNING IN AN EVIDENCE BASED ENVIRONMENT JESSICA NAPIER JUSTICE CENTER FOR EVIDENCE BASED PRACTICE Developed for WV SAG Committee Retreats September.

With Data Collection…(continued) Self-assessment (allows program to

see progress or areas to improve) What gets measured gets done!

If results are not measured, success cannot be distinguished from failures.

If successes cannot be distinguished, they cannot be replicated.

If failures can not be identified, they cannot be corrected.

If success cannot be demonstrated, support cannot be secured.

Performance Measurement: A closer look

Page 24: STRATEGIC PLANNING IN AN EVIDENCE BASED ENVIRONMENT JESSICA NAPIER JUSTICE CENTER FOR EVIDENCE BASED PRACTICE Developed for WV SAG Committee Retreats September.

Key Terms Associated with Performance Measurement

Process Measures (Output) Outcome Measures Goals Objectives Activities

Page 25: STRATEGIC PLANNING IN AN EVIDENCE BASED ENVIRONMENT JESSICA NAPIER JUSTICE CENTER FOR EVIDENCE BASED PRACTICE Developed for WV SAG Committee Retreats September.

Key Terms Examined

Process Measures (Output) Addresses program implementation;

directly tied to the activities.

Outcome Measurements Addresses the program accomplishments;

directly tied to the objectives.

The more specific the measure, the more reliable it will measure what it says it will measure.

Page 26: STRATEGIC PLANNING IN AN EVIDENCE BASED ENVIRONMENT JESSICA NAPIER JUSTICE CENTER FOR EVIDENCE BASED PRACTICE Developed for WV SAG Committee Retreats September.

Key Terms Examined1

Goals*

What is the end result of a project or what is anticipated to happen (outcome)

Conceptual and abstract Objectives *

Outcomes that are tangible, concrete, specific, measureable, and achievable, with a time element

Activities Is only one way to accomplish what you have

written. If your answer is yes, you have probably written an activity. Plan to do = Activity Want to do…= Objective* College of DuPage. Creating goals, objective, and activities. Available at http://www.cod.edu/grants/Creating_Goals.pdf

Page 27: STRATEGIC PLANNING IN AN EVIDENCE BASED ENVIRONMENT JESSICA NAPIER JUSTICE CENTER FOR EVIDENCE BASED PRACTICE Developed for WV SAG Committee Retreats September.

Reflection

Page 28: STRATEGIC PLANNING IN AN EVIDENCE BASED ENVIRONMENT JESSICA NAPIER JUSTICE CENTER FOR EVIDENCE BASED PRACTICE Developed for WV SAG Committee Retreats September.

Reflection

Looking to the future: Things to Consider Think about the desired outcome Start with the RFP

Infuse expertise into the grant process Lay out requirements in RFP

Educate applicants Offer online trainings if necessary

May consider a webinar or podcast for grant writing

Educational webinar or podcast for applicants on EBP

Make resources available via CD or online forum for applicants (up-to-date and cost efficient)

Page 29: STRATEGIC PLANNING IN AN EVIDENCE BASED ENVIRONMENT JESSICA NAPIER JUSTICE CENTER FOR EVIDENCE BASED PRACTICE Developed for WV SAG Committee Retreats September.

Reflection

Looking to the future: Things to Consider Fidelity Performance Measurements

Qualitative and Quantitative Make it meaningful Measure what you want to know

Data! What gets measured gets done If you are going to require data to be submitted, data

should be utilized, feedback should be given, reports (comparative: by agency or program)

Online resources!!!! Data requests can be submitted via online survey tools

into an excel spreadsheet

Page 30: STRATEGIC PLANNING IN AN EVIDENCE BASED ENVIRONMENT JESSICA NAPIER JUSTICE CENTER FOR EVIDENCE BASED PRACTICE Developed for WV SAG Committee Retreats September.

Reflection

Evaluation Does a program do what it says it will do? Unbiased researcher Timely & requires specific funds

Critical to determine if program is effective Must have measures in place from the

beginning

Page 31: STRATEGIC PLANNING IN AN EVIDENCE BASED ENVIRONMENT JESSICA NAPIER JUSTICE CENTER FOR EVIDENCE BASED PRACTICE Developed for WV SAG Committee Retreats September.

Transitioning to EBP: Example

Basic data request Filed away

Prevention? Instructional Time

No lesson plans Not EBP

Law Enforcement Presence

Expand data collection Update form Create database Make available online

Pre\Post test Satisfaction Survey Using EBP lessons

Standardization “what works” Some are little to no cost

to PRO Online available

Current PRO Adding SOME EBP

Page 32: STRATEGIC PLANNING IN AN EVIDENCE BASED ENVIRONMENT JESSICA NAPIER JUSTICE CENTER FOR EVIDENCE BASED PRACTICE Developed for WV SAG Committee Retreats September.

Conclusion

CD Grant Writing Resources Evidence Based Practices Resources Other Documents and Publications

Page 33: STRATEGIC PLANNING IN AN EVIDENCE BASED ENVIRONMENT JESSICA NAPIER JUSTICE CENTER FOR EVIDENCE BASED PRACTICE Developed for WV SAG Committee Retreats September.

Contact Information

[email protected]

558-8814 ext 53325

[email protected]

558-8814 ext 53338

Jessica Napier Stephen M. Haas

Important links:www.djcs.wv.gov

www.facebook.com/wvcjsacwww.twitter.com/wvcjsac

Page 34: STRATEGIC PLANNING IN AN EVIDENCE BASED ENVIRONMENT JESSICA NAPIER JUSTICE CENTER FOR EVIDENCE BASED PRACTICE Developed for WV SAG Committee Retreats September.

Thank you