Evaluation Research Kodi D. Havins AED 615 Fall 2006 Dr. Franklin.

Evaluation Evaluation ResearchResearch

Kodi D. HavinsKodi D. HavinsAED 615AED 615Fall 2006Fall 2006

Dr. FranklinDr. Franklin

Evaluation ResearchEvaluation Research

• Definitions:Definitions:– “ “Evaluation is the systematic Evaluation is the systematic

assessment of the worth or assessment of the worth or merit of some object”merit of some object”

– ““Evaluation is the systematic Evaluation is the systematic acquisition and assessment of acquisition and assessment of information to provide useful information to provide useful feedback about some object”feedback about some object”

(Trochim, 2006)(Trochim, 2006)

Evaluation ResearchEvaluation Research

• Basically is used to provide feedback on Basically is used to provide feedback on an event, organization, program, policy, an event, organization, program, policy, technology, person, activity, etc.technology, person, activity, etc.– All of these are called “object(s)”All of these are called “object(s)”

• It assesses information about the “object” It assesses information about the “object” • Used heavily in social sciences and Used heavily in social sciences and

government agenciesgovernment agencies• Usually outcomes influence Usually outcomes influence

decision-making processdecision-making process

Research DesignsResearch Designs

• 4 Major Types:4 Major Types:1.1. Survey ResearchSurvey Research

2.2. Case StudyCase Study

3.3. Field ExperimentField Experiment

4.4. Secondary Data AnalysisSecondary Data Analysis

(Garson, n.d.)(Garson, n.d.)

ProceduresProcedures

• 4 Strategies:4 Strategies:1.1.Scientific-Experimental ModelsScientific-Experimental Models

2.2.Management-Oriented Systems Management-Oriented Systems ModelsModels

3.3.Qualitative/Anthropological ModelsQualitative/Anthropological Models

4.4.Participant-Oriented ModelsParticipant-Oriented Models


1. Scientific-Experimental 1. Scientific-Experimental ModelsModels

• Take values and methods from the sciences (especially social sciences)

• Prioritize on the desirability of impartiality, accuracy, objectivity, and validity of the information generated


1. Scientific-Experimental 1. Scientific-Experimental ModelsModels

• Examples:Examples: Tradition of experimental and quasi-Tradition of experimental and quasi-

experimental designsexperimental designs Objectives-based research that comes Objectives-based research that comes

from educationfrom education Econometrically-oriented perspectives Econometrically-oriented perspectives

including cost-effectiveness and cost-including cost-effectiveness and cost-benefit analysisbenefit analysis

Recent articulation of theory-driven Recent articulation of theory-driven evaluationevaluation


2. Management-Oriented 2. Management-Oriented SystemsSystems

• 2 of the most common:2 of the most common:– Program Evaluation and Review Program Evaluation and Review

Technique (PERT) Technique (PERT) – Critical Path Method (CPM)Critical Path Method (CPM)


2. Management-Oriented 2. Management-Oriented SystemsSystems

Examples:Examples:

• Framework (“Logframe”) Model Framework (“Logframe”) Model – Used widely by U.S. Agency for Used widely by U.S. Agency for

International DevelopmentInternational Development


3. 3. Qualitative/AnthropologiQualitative/Anthropological Modelscal Models

• Emphasis on:Emphasis on:– The importance of observationThe importance of observation– The need to retain the The need to retain the

phenomenological quality of the phenomenological quality of the evaluation contextevaluation context

– The value of subjective human The value of subjective human interpretation in the evaluation processinterpretation in the evaluation process


3. 3. Qualitative/Anthropological Qualitative/Anthropological ModelsModelsExamples:Examples:

– The approaches known in evaluation as The approaches known in evaluation as naturalistic or “Fourth Generation” naturalistic or “Fourth Generation” EvaluationEvaluation

– The various qualitative schoolsThe various qualitative schools– Critical Theory and art criticism Critical Theory and art criticism

approachesapproaches– The “Grounded Theory” approaches of The “Grounded Theory” approaches of

Glaser and StraussGlaser and Strauss


4. Participant-Oriented 4. Participant-Oriented ModelsModels

• Emphasize the central Emphasize the central importance of the evaluation importance of the evaluation participantsparticipants


4. Participant-Oriented 4. Participant-Oriented ModelsModels

Examples:Examples:– Client-Centered Research Client-Centered Research – Stakeholder ApproachesStakeholder Approaches– Consumer-Oriented Research Consumer-Oriented Research


So, how do you So, how do you decide which one decide which one

to use???to use???

Things to keep in mind:Things to keep in mind:

• Great evaluators use ideas from Great evaluators use ideas from each of the 4 strategies as they need each of the 4 strategies as they need themthem

• Methodologies needed will and Methodologies needed will and should be variedshould be varied

– This should not be an easy processThis should not be an easy process

• There is no simple answer!!!There is no simple answer!!!


Types of Evaluations:Types of Evaluations:

• There are many different There are many different types of evaluationstypes of evaluations

Types of Evaluations:Types of Evaluations:

• 2 Important Types:2 Important Types:

1.1. Formative EvaluationFormative Evaluation

2.2. Summative EvaluationSummative Evaluation(Trochim, (Trochim,

2006)2006)

1. Formative Evaluation1. Formative Evaluation

• Purpose:Purpose:– To strengthen or improve the To strengthen or improve the

object being evaluatedobject being evaluated(Trochim, 2006)(Trochim, 2006)

• In other words, to provide In other words, to provide useful feedback to for the useful feedback to for the greater good of the “object”greater good of the “object”


• Includes several Evaluation TypesIncludes several Evaluation Types– Needs AssessmentNeeds Assessment– Evaluablitiy AssessmentEvaluablitiy Assessment– Structured ConceptualizationStructured Conceptualization– Implementation Evaluation Implementation Evaluation – Process EvaluationProcess Evaluation



• Questions generally asked:Questions generally asked:– What is the definition and scope of the What is the definition and scope of the

problem or issue, or what’s the question?problem or issue, or what’s the question?– Where is the problem and how big or Where is the problem and how big or

serious is it?serious is it?– How should the program or technology How should the program or technology

be delivered to address the problem?be delivered to address the problem?– How well is the program or technology How well is the program or technology

delivered?delivered?(Trochim, 2006)(Trochim, 2006)

2. Summative Evaluation2. Summative Evaluation

• Purpose:Purpose:– To examine the effects or To examine the effects or

outcomes of some objectoutcomes of some object(Trochim, 2006)(Trochim, 2006)

• In other words, to review In other words, to review the results/conclusions of the results/conclusions of an “object”an “object”


• Can be subdivided into different Can be subdivided into different categories:categories:

– Outcome EvaluationsOutcome Evaluations– Impact EvaluationImpact Evaluation– Cost-Effectiveness and Cost-Benefit Cost-Effectiveness and Cost-Benefit

Analysis Analysis – Secondary Analysis Secondary Analysis – Meta-AnalysisMeta-Analysis

(Trochim, (Trochim, 2006)2006)


• Questions generally asked:Questions generally asked:– What type of evaluation is feasible?What type of evaluation is feasible?– What was the effectiveness of the What was the effectiveness of the

program or technology?program or technology?– What is the net impact of the program?What is the net impact of the program?


LimitationsLimitations

• Limitations of this type of research Limitations of this type of research include the level of expertise and include the level of expertise and knowledge of the expert panels knowledge of the expert panels – If the evaluation is only as good as the If the evaluation is only as good as the

knowledge of the expert panelknowledge of the expert panel

Journal ArticleJournal Article

A Review of Subject Matter A Review of Subject Matter Topics Researched in Topics Researched in

Agricultural and Extension Agricultural and Extension EducationEducation

By: Rama B. Radhakrishna & By: Rama B. Radhakrishna & Wenwei XuWenwei Xu

A Review of Subject Matter Topics A Review of Subject Matter Topics Researched in Agricultural and Extension Researched in Agricultural and Extension EducationEducation

Retrieved from: Retrieved from:

Journal of Agriculture Education Journal of Agriculture Education

http://pubs.aged.tamu.edu/jae/http://pubs.aged.tamu.edu/jae/search/default.aspsearch/default.asp

Purpose of StudyPurpose of Study

• The purpose of the study was to The purpose of the study was to “examine subject matter topics “examine subject matter topics researched in agricultural education researched in agricultural education over a ten year period”over a ten year period”

ObjectivesObjectives

• The objectives of the study were to:The objectives of the study were to:1.1. Identify subject matter topics researched Identify subject matter topics researched

in agricultural extension education in the in agricultural extension education in the last decade (1986 – 1996)last decade (1986 – 1996)

2.2. Categorize subject matter topics Categorize subject matter topics published in the published in the Journal of Agricultural Journal of Agricultural EducationEducation and proceedings of the and proceedings of the National Agricultural Education Research National Agricultural Education Research Meeting over a ten year period (1986-Meeting over a ten year period (1986-1996)1996)

How the Evaluation Method How the Evaluation Method was usedwas used

• 2 sources were used to gain data 2 sources were used to gain data – Journal of Agricultural EducationJournal of Agricultural Education– Papers Presented at the National Papers Presented at the National

Agricultural Education Research Agricultural Education Research meetings (NAERM) from 1986-1996meetings (NAERM) from 1986-1996


• Resulted in 402 journal articles and Resulted in 402 journal articles and 451 papers451 papers


• Each paper was given a code numberEach paper was given a code number

• Each was reviewed and categorized Each was reviewed and categorized into relevant subject matter into relevant subject matter categoriescategories– 3 Criteria were used:3 Criteria were used:

•Title of StudyTitle of Study

•Central Theme and FocusCentral Theme and Focus

•Findings and conclusionsFindings and conclusions


• 25 subject-matter topics were 25 subject-matter topics were identifiedidentified

• The list of the 25 topics was given to The list of the 25 topics was given to a panel of experts for review and a panel of experts for review and validationvalidation


• Panel of ExpertsPanel of Experts– Asked to comment on:Asked to comment on:

1.1. The appropriateness of categoriesThe appropriateness of categories

2.2. Add or identify categories they thought Add or identify categories they thought that were left out or delete categories they that were left out or delete categories they felt did not fitfelt did not fit

3.3. Suggest whether some categories could be Suggest whether some categories could be combinedcombined


• Revised list of subject-matter Revised list of subject-matter categories contained 30 subject categories contained 30 subject matter topicsmatter topics

• Data was then summarized using Data was then summarized using frequencies and percentagesfrequencies and percentages

Some of the FindingsSome of the Findings

Top 5 Studies:Top 5 Studies:• 75 75 Secondary Agriculture ProgramsSecondary Agriculture Programs

StudiesStudies

• 70 70 Styles/Theory and CognitionStyles/Theory and Cognition Studies Studies

• 43 43 ProfessionalismProfessionalism Studies Studies

• 4242 Extension Extension StudiesStudies

• 38 38 Agriculture Mechanics/SafetyAgriculture Mechanics/Safety Studies Studies

Conclusions of StudyConclusions of Study

““Study provided information on Study provided information on subject matter topics investigated subject matter topics investigated by agricultural and extension by agricultural and extension educators, which in turn provides educators, which in turn provides perspectives about the research perspectives about the research efforts of the agricultural and efforts of the agricultural and extension education profession.”extension education profession.”

Summary of Evaluation Summary of Evaluation Research MethodResearch Method

•The evaluation method was used The evaluation method was used in this study to assess what in this study to assess what topics were researched in the topics were researched in the agricultural education field agricultural education field during a 10 year period.during a 10 year period.

Summary of Evaluation Summary of Evaluation Research MethodResearch Method

•The outcomes of this study will The outcomes of this study will provide useful feedback to those provide useful feedback to those in the professionin the profession

ReferencesReferences

• Radhkrishna, R.B. & Xu, W. (1997). A review of subject Radhkrishna, R.B. & Xu, W. (1997). A review of subject

matter topics researched in agricultural and matter topics researched in agricultural and extension extension education. education. Journal of Agricultural Journal of Agricultural EducationEducation. 38(3), 59-. 38(3), 59- 69.69.

• Trochim, W.M. (2006) Research methods knowledge Trochim, W.M. (2006) Research methods knowledge base. base.

Drake UniversityDrake University. Retrieved from: . Retrieved from: http://www.socialresearchmethods.net/kb/intreval.htm http://www.socialresearchmethods.net/kb/intreval.htm

on 10/26/06.on 10/26/06.

Any Any Questions??Questions??

??

Thank YouThank You

Evaluation Research Kodi D. Havins AED 615 Fall 2006 Dr. Franklin.

Documents

Transcript of Evaluation Research Kodi D. Havins AED 615 Fall 2006 Dr. Franklin.