PLT PROGRAM EVALUATION

What do funders want to know and can you (provide the data to) give

it to them?

Norie Dimeo-EdigerCarla Rapp

Michelle Youngquist

HERE’S WHAT OUR STAKEHOLDERS SAID

We would like to know that your forest education programs for educators (PLT workshops, tours, etc.) are effective and

a good investment of our money.$$$$$$$$$$$$

HERE IS WHAT WE HEARD

Conduct an extensive (and expensive) statistically

valid evaluation.

HERE IS WHAT WE THOUGHT

I know that you believe you understand what you think I said, but I’m not sure you realize that what you heard is not what I meant.

Robert McCloskey

What did the stakeholders mean?

?????????

So………

Norie asked Pete and Paul.

Carla asked Larry, Winston and Bill.

Michelle asked Kevin, Bob and Mike.

What did they say????

I want to know that teachers and students “get it.”

-It would be nice to be able to track from grade to grade and year to year the attitudes and understanding of the students regarding our issues.

-I want you to go out and tape the teachers using the activities so we can see them in action.

Are they using the materials?

Has it changed (for the better) the way they teach about forests?   

I want to know enough people have responded to the questionnaire/survey that I can trust the results

Is PLT still the best way to deliver our message and how do we evaluate that?

How do we do what they want?

LEVELS OF EVALUATION Level 1. Reaction – What is the participants’

response to the project or activity?

Level 2. Learning – What did the participants learn?

Level 3. Behavior or Application – Did the participants’ learning affect their behavior?

Level 4. Results – Did participants’ behavior change move the original situation towards the objective

Source: Kirkpatrick (1994). Designing Evaluation for Education Projects – NOAA

LEVEL 1. REACTION – WHAT IS THE PARTICIPANTS’ RESPONSE TO THE PROJECT OR ACTIVITY?

LEVEL 2. LEARNING – WHAT DID THE PARTICIPANTS LEARN?

When asked to rate their content knowledge before/after the workshop the responses were:

BEFORE High 5 4 3 2 1 Low 1 5 8 10 5 1 AFTER High 5 4 3 2 1 Low 17 13 0 0 0

LEVEL 3. BEHAVIOR OR APPLICATION – DID THE PARTICIPANTS’ LEARNING AFFECT THEIR BEHAVIOR?

LEVEL 4. RESULTS – DID PARTICIPANTS’ BEHAVIOR CHANGE MOVE THE ORIGINAL SITUATION TOWARDS THE OBJECTIVE

Oil City is a Title I school in rural northwest Louisiana. In 2001, faced with declining enrollment and a possible closure, teachers and the administration worked together to create a school with an environmental focus.

Measurable AchievementThe success of the new environmental education focus and the

activities that sprang from it brought recognition to the school. In 2004–2005, the school was recognized by the Louisiana Association of Business and Industry as one of the “Top 10” most improved schools in Louisiana.

In 2006, the school received the National School Change Award sponsored by Fordham University Graduate School of Education, the American Association of School Administrators, and Pearson Education. Students have been invited to perform at the Louisiana Forestry Association Conference in Shreveport.

In 2006, a fourth-grade teacher at the school, Brenda Smith, was named a National Outstanding Educator for Project Learning Tree.

SOME EVALUATION METHODS THAT MIGHT ANSWER THEIR QUESTIONS

Focus groups Questionnaires or surveys Observation Case studies

FOCUS GROUPS Purpose-to explore a topic in depth

through group discussion Advantages-key info can be conveyed,

efficient way to get range and depth of info

Disadvantages-can be hard to analyze, need good facilitator, can be hard to schedule

QUESTIONNAIRES AND SURVEYS Purpose-to quickly/easily obtain a lot of

info from people in a non-threatening way.

Advantages-anonymous, inexpensive, large sample size, easy to compare and analyze

Disadvantages-might not get careful feedback, wording can bias response, may not tell full story, low return rate

OBSERVATION Purpose-to gather accurate info on how

people behave Advantages-generates data about

behavior not reported behavior, data collected in context

Disadvantages-time intensive, “observer effect”, small sample size

CASE STUDIES Purpose-to conduct comprehensive

examination through cross comparison of cases

Advantages-provides vivid imagery, and detailed understanding of the experience

Disadvantages-time intensive, detailed understanding of a particular participant results can not be generalized.

COMMUNICATION OF RESULTS What? To whom? How much? How

often? What language? eg. Edu-speak?

Communicating results should not be an after thought, but part of the evaluation plan.

SOME EXAMPLES OF WHAT WE’VE DONE Carla: Pre/Post “Viewpoints on the

Line” adaptation, video-taping teachers doing PLT

Norie: Professional 3rd party evaluation Michelle: Surveys, portfolio of teacher

& student work. Collecting, but:

You do need to communicate it to the stakeholders. How best to do that?

WHAT HAVE OTHERS EXPERIENCED

QUESTIONS?

Thank you! Norie Dimeo-Ediger, Oregon Forest Resources Institute

dimeo-ediger@ofri.org 971-673-2956 www.OregonForests.org

Carla L. Rapp, Georgia Forestry Association carla@gfagrow.org 478-992-8110 www.gfagrow.org

Michelle Youngquist, Idaho Forest Products Commission plt@idahoforests.org 208-334-4061 www.idahoforests.org

CARLA, MICHELLE & NORIE