Post on 01-Jan-2016
description
Dr. Carol AlbrechtResearch Team
EXAMPLE of EVALUATION RESEARCH
18 public school districts in the state of Texas were involved in Service Learning Projects. The State of Texas Service Learning Center hired Dr. Carol Albrecht and her students to evaluate this program. These power point slides outline the steps they tookto complete this evaluation.
We met with them to identify their objectives. They wanted to know how the program impacted public school children, teachers, community partners and parents of students.
Click here to see the surveys and
codebooks.
Click here to see the surveys and
codebooks.
An Old Chinese Proverb States:
I HEAR, I FORGETI SEE, I REMEMBER
I DO, I UNDERSTAND
By “doing” this project we learned and really understood some important
components of valid and reliable evaluation research.
Timing of the Pre-testTiming of the Pre-test
Many programs are ongoing, and this canMany programs are ongoing, and this can
have a major impact on the pre-test.have a major impact on the pre-test.
In our study, many of the students had In our study, many of the students had
already participated in a Service Learningalready participated in a Service Learning
activity at some point in their school years.activity at some point in their school years.
So, we didn’t have a true “pre” test. TheSo, we didn’t have a true “pre” test. The
““pre” test scores were contiminated bypre” test scores were contiminated by
prior participation.prior participation.
Timing of the Post-TestTiming of the Post-Test
The “May” EffectThe “May” EffectOutside environmental/social factors needOutside environmental/social factors need
to be considered. to be considered.
In our study, we discovered that both teachers and In our study, we discovered that both teachers and students were more negative about almost EVERYTHING students were more negative about almost EVERYTHING related to school at the end of the year. This may berelated to school at the end of the year. This may be
explained by two factors: explained by two factors:
First, they were just TIRED of school, and lookingFirst, they were just TIRED of school, and looking
forward to vacation.forward to vacation.
Second, they had just taken standardized tests, Second, they had just taken standardized tests, TAKS. These tests were stressful for both TAKS. These tests were stressful for both
teachers teachers and students.and students.
Head Start Program In the 1960’s the head start programwas launched. The objectivewas to increase the IQ scores of underrepresented
populations including children living in poverty.Early research showed that standardized tests of IQ
increased for several years, and the decreased, until there was no difference between the experimental and control groups. While some felt this was evidence for
discontinuing the program, parents came forward arguing that the researchers
weren’t using the right measurements.
Head Start Program
A group of researchers call The Perry Preschool Consortium, with The input of teachers and parents, identified(1) social, (2) educational and (3)
socioeconomic indicators that differentiated preschool participants from a control group up to 19 years after participation in the program. The differences were compelling.
Furthermore, this group arguedthat the decreasing IQ scoresactually provided evidence that environmental factors CAN influence IQ – both positively and negatively. Thus being In an “enriched” environment (i.e., the Head
Start Program) can increase IQ but then being transferred to an impoverished environment
(i.e., public schools in poor neighborhoods) can decrease IQ.
Schooling SuccessHigh School Graduation or EquivalentCollege or Vocational TrainingFunctional Competence
Ever Classified as Mentally RetardedTime Spent in Special Education
Social Responsibility
Ever Detained or ArrestedTeen Pregnancies
EmployedReceiving Welfare
Using focus groups and intensive interviews,
we looked to
(1) Teachers,
(2) Parents
(3) And student participants
as well as past research, to
help us identify valid and accurate indicators.
Analysis of this qualitative data indicated
(1) Some students did experience the desired impact.
(2) We needed to use “control” variables to accurately assess the impact of Service Learning.
The following control variables were all highly significantly related to EVERY outcome measurement.
Student Success
Results from focus groups with students and intensive interviews with teachers
indicated that these were valid indicators of the quality and quantity of
participations were related to outcomes.
Mean Score on Outcome Measurements by Amount of Time Student Planned or Participated in Project
Students Planned -Four Hours or More
Yes No
Attitudes Toward Attending CollegeAttitudes Toward SchoolProblem Solving SkillsLeadership Skills
12.81***19.84***17.01***17.15***
12.50***18.45***15.58***15.85***
Students Participated - Four Hours or More
Yes No
Attitudes Toward Attending CollegeAttitudes Toward SchoolProblem Solving SkillsLeadership Skills
12.89***19.54***16.91***17.04***
12.18***18.38***15.09***15.46***
***p<.0001
* p< .05 **p< 0.01 ***p< 0.0001
Mean Scores on Outcome Measurements by Sense of Ownership – Whether or Not Students Made Decisions
Students Made Decisions
Yes No
Attitudes Toward CollegeAttitudes Toward SchoolProblem Solving SkillsLeadership Skills
13.04***19.59***
17.26***p17.51***
12.36***18.84***15.58***15.81***
* p< .05 **p< 0.01 ***p< 0.0001
Click here to see the Power Point Presentation.
CHART 1. High School Students’ Perception of How Good They are at Speaking in Front of Groups by Whether or Not They Made Decisions about Service Learning Projects
(p <0.0001)
CHART 2. High School Students’ Perception of How Good They are at Finding Resources by Whether or Not They Made Decisions about Service Learning Projects
11%7%
44%
13%7%
28%24%
9%
43%
17%
0.00%10.00%20.00%30.00%40.00%50.00%60.00%70.00%80.00%90.00%
100.00%
StronglyDisagree
Disagree Undecided Agree Strongly Agree
Did
Did Not
(p <0.0001)
Beware of Social Desirabil
ity
In Evaluation Research, Participant Often Evaluate
a Program Positively – EVEN when the Program is “poor” and Ineffective.
It may not be seen as socially acceptable to do
otherwise.
Why did we become concerned?Why did we become concerned?
What are the “danger” signs? What are the “danger” signs?
How did we attempt to alleviate it? How did we attempt to alleviate it?
How did we modify the constructionHow did we modify the construction
of our surveys, our research design and of our surveys, our research design and
analysis of the data to deal with this analysis of the data to deal with this
problem?problem?
Danger SignsDanger Signs
Type of researchType of research
Past literature indicates that respondents tend to Past literature indicates that respondents tend to bebe
very positive when asked about their participation very positive when asked about their participation inin
a program even when it is a poor program. They a program even when it is a poor program. They don’ don’ want to believe they wasted their time, and want to believe they wasted their time, and they oftenthey often
feel an obligation to express appreciation for feel an obligation to express appreciation for thosethose
who implemented the program. who implemented the program.
Danger SignsDanger Signs
Self selection into the programSelf selection into the program
Students and teachers were not required to Students and teachers were not required to
participate in the program. Therefore the participate in the program. Therefore the programprogram
was more likely to attract participants who was more likely to attract participants who
already had positive attitudes toward thesealready had positive attitudes toward these
““types” of activities. types” of activities.
Danger SignsDanger Signs
Consistently high scores on everyConsistently high scores on every
aspect of the program – no variationaspect of the program – no variation
Response Set can occur. This is where respondents Response Set can occur. This is where respondents give you the same response (usually positive) give you the same response (usually positive)
without without seriously considering the question. seriously considering the question.
The “ceiling” effect is a similar problem. ThisThe “ceiling” effect is a similar problem. This
is when you get consistently highly positive is when you get consistently highly positive scoresscores
on the pre-test. In this case, there is little room on the pre-test. In this case, there is little room
for improvement in scores.for improvement in scores.
Check ListCheck List
Make participation voluntary and make answers anonymous or confidential.Make participation voluntary and make answers anonymous or confidential.
Vary negative/positive statements inVary negative/positive statements in
Your indexYour index
Avoid misleading/biased questionsAvoid misleading/biased questions
Make statements or questions very specificMake statements or questions very specific
Check List – continuedCheck List – continued
Make participation voluntary and make answers anonymous or confidential.Make participation voluntary and make answers anonymous or confidential.
Put “sensitive” questions at the endPut “sensitive” questions at the end
Ask how they would change program “under ideal circumstances”.Ask how they would change program “under ideal circumstances”.
Avoid (1) yes or (2) no answers – ask “degrees” of positive or negative.Avoid (1) yes or (2) no answers – ask “degrees” of positive or negative.
Ask for their input in improving the program – rather than simply Ask for their input in improving the program – rather than simply
evaluating the program for instance:evaluating the program for instance:
NOT – Is this a successful program, but rather - what factors increase or decrease the success NOT – Is this a successful program, but rather - what factors increase or decrease the success of this program.of this program.
Check ListCheck List
If possible, don’t evaluate your own programIf possible, don’t evaluate your own program
An “outsider” would tend to be more objective and participants would be more An “outsider” would tend to be more objective and participants would be more likely to provide unbiased answers.likely to provide unbiased answers.
Have a variety of participants evaluate the program soHave a variety of participants evaluate the program so
you can look for consistencies/inconsistencies in answers.you can look for consistencies/inconsistencies in answers. StudentsStudents TeachersTeachers Parents of participantsParents of participants Community PartnersCommunity Partners
Check List - continuedCheck List - continued
Use multi-methods so you can compare results across Use multi-methods so you can compare results across
to see if you get similar results and look for additionalto see if you get similar results and look for additional
insights. These could include:insights. These could include:
Focus groupsFocus groupsParticipant observationParticipant observationSurveys Surveys Intensive interviewsIntensive interviewsContent analysis Content analysis
Content analysis is especially important for researchers who Content analysis is especially important for researchers who identify tangible products (e.g., bushels of grains) as their identify tangible products (e.g., bushels of grains) as their outcomes. outcomes.
Check ListCheck List
Compare your program with other programsCompare your program with other programs
Compare across different levels of participation within Compare across different levels of participation within your sample to see if there are variationsyour sample to see if there are variations
Compare across different types of participation within Compare across different types of participation within your sample (i.e., in our study, we compared across your sample (i.e., in our study, we compared across types oftypes of
Service Learning projects).Service Learning projects).
Check ListCheck List
Compare across different “types” of participants (This Compare across different “types” of participants (This wouldwould
include males vs. females, parents vs. children, rural include males vs. females, parents vs. children, rural vs. vs.
urban dwellers). urban dwellers).
Compare scores across questions – especially questions Compare scores across questions – especially questions thatthat
measure the same outcomes.measure the same outcomes.
Compare answers across time Compare answers across time
fall vs. summer participants fall vs. summer participants The most important thing to remember here is to NOT just ask if
the program was successful, but rather, HOW and WHEN it is most successful.
Laboratory Experiments
Intensive Interviews
Examples of data Examples of data
that the program is producing the desired that the program is producing the desired
outcomesoutcomes
collected from Teachers collected from Teachers Using Telephone SurveysUsing Telephone SurveysUsing Focus Groups Using Focus Groups
Descriptive Statistics for Elementary and Middle/High School Service Learning Teachers: Descriptive Statistics for Elementary and Middle/High School Service Learning Teachers: Extent to Which Teachers Agree with the Following Statements about the Impact of Extent to Which Teachers Agree with the Following Statements about the Impact of
Service Learning in Their ClassroomService Learning in Their Classroom
ElementaryElementary Middle/High SchoolMiddle/High School
NumberNumber PercentPercent NumberNumber PercentPercent
Positive Addition to Classroom LearningPositive Addition to Classroom Learning Agree/Strongly AgreeAgree/Strongly Agree
Beneficial for the ALL StudentsBeneficial for the ALL Students Agree/Strongly AgreeAgree/Strongly Agree
Motivates Students to be InvolvedMotivates Students to be Involved Agree/Strongly AgreeAgree/Strongly Agree
Helps Students Learn CurriculumHelps Students Learn Curriculum Agree/Strongly AgreeAgree/Strongly Agree
Should be Required for All StudentsShould be Required for All Students Agree/Strongly AgreeAgree/Strongly Agree
7272
7575
6666
4747
4040
90.0090.00
93.7593.75
87.5087.50
58.7558.75
50.0050.00
7373 7474 6666 5656
5050
96.0596.05
97.3797.37
88.8988.89
73.6873.68
65.7965.79
Descriptive Statistics: Identification of GREATEST BENEFITS by Descriptive Statistics: Identification of GREATEST BENEFITS by Middle/High School and Elementary Service Learning TeachersMiddle/High School and Elementary Service Learning Teachers
ElementaryElementary Middle/High SchoolMiddle/High School
PercentPercent PercentPercent
Benefits for StudentsBenefits for Students Service to OthersService to Others Understanding of WorldUnderstanding of World Personal GrowthPersonal Growth Help Learn CurriculumHelp Learn Curriculum
50.0050.00 14.7114.71 30.89 30.89 4.414.41
32.4332.43 25.6825.68 32.4332.43 9.469.46
Benefits for TeachersBenefits for Teachers Student GrowthStudent Growth Service to OthersService to Others Involvement with StudentsInvolvement with Students Break in TAKSBreak in TAKS
82.2682.2616.1316.13 1.611.61
----
73.9173.9115.9415.94 7.257.25 2.902.90
As One Teacher Stated, “Service Learning is the most powerful and impactful thing I ever did in the classroom as a teacher. It hooked me, and I am a believer in the power.
Another Teacher Claimed, “I think this program has transcended anything that anyone expected when they began the program. It has extended beyond what they thought it could achieve.”
One Teacher Argued, “I could have never ever taught the lessons they learned about human nature.”
While Another Claimed, “It teaches kids the skills that are not book skills….skills like how to think, how to plan, how to organize, how to manage - stuff you can read about in a book, but until you do it, you don’t know you have the ability to do it.”
One Teacher Stated, “school is not as…engaging as when they learn through these projects…they are learning all of these things by action – their great public speaking skills, their writing skills, their marketing…”
Another Teacher Explained, “in the writing TAKS, we had to write with a prompt so it kind of helped with the writing and the reading TAKS too.”
Examples of data Examples of data
that the program is producing the desired that the program is producing the desired
outcomesoutcomes
collected from parents and community partners collected from parents and community partners using telephone surveysusing telephone surveysusing focus groupsusing focus groups
Descriptive Statistics For Parents and Community Partners;Descriptive Statistics For Parents and Community Partners;An Evaluation of the Service Learning ProgramAn Evaluation of the Service Learning Program
ParentsParents Community PartnersCommunity Partners
MeanMean RangeRange MeanMean RangeRange
Positive Addition to ClassroomPositive Addition to Classroom
Beneficial for All StudentsBeneficial for All Students
Motivates Students to be Involved Motivates Students to be Involved
Helps Agency Achieve GoalsHelps Agency Achieve Goals Is Valued by AgencyIs Valued by Agency
Is Valued by CommunityIs Valued by Community
*on 5 pts. Scale (SD to SA)*on 5 pts. Scale (SD to SA)**sample size is small**sample size is small
5.005.00
4.834.83
4.834.83
----
----
----
00
11
11
----
----
----
4.944.94
4.834.83
4.894.89
4.394.39
4.674.67
4.564.56
11
11
11
33
33
33
One Community Partner Described Their Relationship with the School, “We Actually came to the schools…and we were looking for assistance. It’s a great marriage. We are still married.”
And when Describing the Benefits for Students,“..we’ve watched students mature into more
socially aware students - much more mature. It’s amazing. It’s just amazing.”
Timing of data collection is important.
Selecting reliable/valid indicators is critically important. Spend some time doing this.
IF you are doing Evaluation Research, plan ways to reduce the impact of social desirability on your results.
Use multi-methods when feasibility to provide additional insights and greater support for your results.
Try to gather information from all the different groups that may be impacted by the program (i.e., parents, students etc.).
Dr. Carol Albrecht Assessment Specialist USU Ext carol.albrecht@usu.edu (979) 777-2421