1

Evaluation Workshop 10/17/2014

Presenters:Jim Whittaker – KCP/Evaluation

Dr. Larry M. Gant – SSW EvaluationChristiane Edwards – SSW Evaluation

2

Session Overview

Goal: Continuous Improvement Objective 1. Identifying Evaluation Strategies

to Establish Evidence Based Practices Objective 2. Moving Activities from Emerging

to Best Practice through Evaluation Objective 3. Communication – Sites, State,

and SSW Wrap-Up Evaluation Round Table Walk away with evaluation tools to take your

work to the next level.

3

Evaluation – Data Collection – Assessments – Best Practices – Methodology??????????

4

Evaluation Goals

Short Term Goals. Within the next six months, the Evaluation Team will:

Provide technical assistance to University Program Partners in capturing service data within MI TRACKR.

Provide intensive regional MI TRACKR training sessions on quarterly basis.

Format data and/or data files- create a unified roster to prepare data for analysis, - conduct an initial analysis of program service data from 8th

and 9th grade cohort.

5

Short-Term Goals continued

Continuously update the MI GEAR UP website with relevant information, in particular, materials to support MI GEAR UP Program Partner evaluation strategies such as toolkits, templates, workshop resources and other relevant materials.

Provide technical assistance regarding:- Identifying Evaluation Strategies to Establish Evidence Based

Practices.- Moving Activities from Emerging to Best Practice through

Evaluation.- Communication of Best Practices

Assist MI GEAR UP Program Partners in developing evaluation and survey instruments directly aligned to service delivery objectives and goals.

6

Longer Term Evaluation Goals

Develop a detailed Evaluation Plan that directly reflects the state evaluation research questions.

Complete a detailed analysis of MI GEAR UP services in relationship to MI GEAR UP goals, objectives, and outcomes.

Develop a strategic plan for continuous program improvement through evaluation strategies

Publish academic articles to promote MI GEAR UP Best Practices.

7

Objective 1. Identifying Evaluation Strategies to Establish Evidence Based Practices

Review Emerging to Best Practice components.

Identify real life evaluation strategies

Evaluate these strategies and apply them to your MI GEAR UP programs.

Select one of your signature practices as either an Emerging, Promising or Best Practice.

Identify missing components.

8

Objective 1. Identifying Evaluation Strategies continued

Is your practice replicable? - Is there a lesson plan or template?

Is your practice supported by strong qualitative and quantitative data showing positive outcomes? - Is there a survey or other evaluation instrument to

provide the data?

9

Emerging Promising Best Practices• Incorporates qualities of

other positive/effective interventions.

• Is based on patterns that have been proven to lead to effective outcomes.

• Incorporates a process of continual quality improvement

• Has evaluation plan in place

Emerging Practice

• In addition to the qualities of an emerging practice, has strong qualitative and quantitative data showing positive outcomes

Promising Practice

• Has undergone a rigorous process of peer review and evaluation that indicates effectiveness

Best Practice

Source: “Emerging, Promising, and Best Practices Definitions” from the Kentucky Cabinet for Health and Family Services

10

Identify Real Life Evaluation Strategies

Learn to bake a delicious Whipping Cream Pound Cake

http://youtu.be/9wd4pJJfMUc

Can you replicate this pound cake with the information given in the clip? If not, what is missing?

11

Identify Real Life Evaluation Strategies

Have you ever booked a hotel room, bought a house or a car?

Identify evaluation strategies you would use to determine which hotel room, house or car to select?

12

Identify Real Life Evaluation Strategies (continued)

Are you using a Cost Benefit Analysis or Criterion Based Matrix to assess whether a purchase is worth your money?

a) Cost vs. Location

b) Cost vs. Size

c) Cost vs. Benefits

d) Cost vs. Amenities

e) Cost vs. Appearance

By using these strategies, you are actually conducting a small scale evaluation! NOW, LET’S TAKE THIS EXAMPLE AND APPLY IT TO MI GEAR UP.

13

Objective 1. Identify Evaluation Strategies to Establish Evidence-Based Practices.

Like a recipe, your lesson plan serves as the tool or blue print

for replication.

A lesson plan is one of the components for establishing a Best Practice.

If your practice is replicable, it can reliably produce strong qualitative and quantitative data supporting positive outcomes.

14

Objective 1. Identify Evaluation Strategies to Establish Evidence-Based Practices.

Using a Criterion Based Matrix or Cost Benefit Analysis for your practice can demonstrate significant positive outcomes.

Program Services Delivery Time & Money (PSDT & M) vs. the following program goals:

Increased understanding of objectives and goals Increased GPA Increased standardized test scores Increased understanding of college affordability Increased FAFSA completion rates Promotion to the next grade level Increased graduation rates

Information from the analysis will allow you to continuously improve the variables to achieve better outcomes.

15

Evaluation Program Workshop: Emerging/Promising/Best Practice

Please place your practice on the Emerging/Promising/ Best spectrum.

1. Identify which the practice components you currently have in place.

2. Identify which the practice components you would like to establish.

3. During the roundtable discussion, you may share which evaluation strategies you are currently using for continuous program improvement.

16

Objective 1 Worksheet. Identifying Evaluation Strategies to Establish Evidence-Based Practices 

Program Partner:__________________________________________

 

Signature Practice: ________________________________________

 

Directions: Place Your Practice on the Emerging/Promising/Best Practice spectrum.

Use information from your previously completed Characteristics of Promising/Best Practices document to complete the following:

 

1. Identify which of the seven Promising/Best Practice components you currently have in place and describe how those components apply to your practice.

2. Identify which of the seven Promising/Best Practice components you would like to establish and describe the necessary actions steps to implement them.

17

Take Your Selected Practice and Use It for Continuous Program Improvement.

1. A Promising Practice requires strong qualitative and quantitative data showing positive outcomes. Acquiring this data involves both a lesson plan for replicability and an evaluation tool to demonstrate positive outcomes.

2. Identify the components of a lesson plan to support the replicability of your practice.

3. Identify the components of an evaluation instrument to support qualitative and quantitative data supporting the activity as a promising practice.

18

Objective 2. Moving Activities from Emerging to Best Practice through Evaluation

Moving from Emerging to Best Practices is an ongoing activity. The Evaluation Team is available to provide the following assistance:

Development of lesson plan templates

Draft survey instruments and other evaluation tools

19

Objective 2 Worksheet. Moving Activities from Emerging to Best Practice through Evaluation.

Program Partner: ________________________________

 

Signature Practice: _______________________________

 

Directions: A lesson plan is one of the components for establishing a Best Practice. If your practice is replicable, it can reliably produce strong qualitative and quantitative data supporting positive outcomes. To effectively replicate a practice, a lesson plan should include the following components:

Goal: What the student will have learned by end of the activity. Objective: Measures for whether the goal has been achieved Activity: Step-by-step description of the practice Assessment: Measures whether goal has been achieved

1. Identify the lesson components for your signature practice:

20

Objective 2 Worksheet. Moving Activities from Emerging to Best Practice through Evaluation.

Program Partner: ________________________________

 

Signature Practice: _______________________________

 

Directions: Once you have a replicable practice, you will want to develop and implement an evaluation tool to capture qualitative and quantitative data supporting positive program outcomes.

 

2. Identify an evaluation tool to capture data from your practice and describe how the evaluation would be implemented:

21

Objective 3. Communication of Evaluation Results

Sites Ongoing program improvement Share program success with local stakeholders

State Share program success with national and state

stakeholders Inform programmatic decisions and resource

allocation SSW

Publish program outcomes.

22

Action Steps

1. Identify one to two signature program services that will be transformed to a Best Practice.

2. Continuously improve those services through evaluation strategies.

3. Develop and implement the lesson plan for replicability.

23

Action Steps

4. Develop evaluation tools, in cooperation with the Evaluation Team, to support your practice with quantitative and qualitative data.

5. Communicate your improvements and signature work with the Evaluation Team, so that the work may be published.

24

Wrap Up

25

Evaluation Roundtable Discussion

Handout for Discussion Questions is provided.