Program Evaluation Strategies to Improve Teaching for Learning Rossi Ray-Taylor and Nora Martin...
-
Upload
weston-archbold -
Category
Documents
-
view
215 -
download
0
Transcript of Program Evaluation Strategies to Improve Teaching for Learning Rossi Ray-Taylor and Nora Martin...
![Page 1: Program Evaluation Strategies to Improve Teaching for Learning Rossi Ray-Taylor and Nora Martin Ray.Taylor and Associates MDE/NCA Spring School Improvement.](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649caf5503460f9497254f/html5/thumbnails/1.jpg)
Program Evaluation Strategies to Improve Teaching for Learning
Rossi Ray-Taylor and Nora MartinRay.Taylor and Associates
MDE/NCA Spring School Improvement Conference2008
![Page 2: Program Evaluation Strategies to Improve Teaching for Learning Rossi Ray-Taylor and Nora Martin Ray.Taylor and Associates MDE/NCA Spring School Improvement.](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649caf5503460f9497254f/html5/thumbnails/2.jpg)
Introductions
Overview for the session
![Page 3: Program Evaluation Strategies to Improve Teaching for Learning Rossi Ray-Taylor and Nora Martin Ray.Taylor and Associates MDE/NCA Spring School Improvement.](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649caf5503460f9497254f/html5/thumbnails/3.jpg)
Evaluation comfort levels
0
5
10
15
20
25
30
conductingevaluation
beingevaluated
notcomfortable
somewhatcomfortable
verycomfortable
![Page 4: Program Evaluation Strategies to Improve Teaching for Learning Rossi Ray-Taylor and Nora Martin Ray.Taylor and Associates MDE/NCA Spring School Improvement.](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649caf5503460f9497254f/html5/thumbnails/4.jpg)
Research driven school design
Rigor, relevance, relationships
![Page 5: Program Evaluation Strategies to Improve Teaching for Learning Rossi Ray-Taylor and Nora Martin Ray.Taylor and Associates MDE/NCA Spring School Improvement.](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649caf5503460f9497254f/html5/thumbnails/5.jpg)
![Page 6: Program Evaluation Strategies to Improve Teaching for Learning Rossi Ray-Taylor and Nora Martin Ray.Taylor and Associates MDE/NCA Spring School Improvement.](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649caf5503460f9497254f/html5/thumbnails/6.jpg)
Evaluation can provide information about
• Definition and evidence of the problem (needs assessment)
• Contextual information about the factors related to the problem and solutions
• The input factors and resources• The interventions, processes and
strategies employed• Outcomes, results and effectiveness
![Page 7: Program Evaluation Strategies to Improve Teaching for Learning Rossi Ray-Taylor and Nora Martin Ray.Taylor and Associates MDE/NCA Spring School Improvement.](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649caf5503460f9497254f/html5/thumbnails/7.jpg)
Program evaluation can be a form of action research
![Page 8: Program Evaluation Strategies to Improve Teaching for Learning Rossi Ray-Taylor and Nora Martin Ray.Taylor and Associates MDE/NCA Spring School Improvement.](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649caf5503460f9497254f/html5/thumbnails/8.jpg)
Elements of a sound evaluation
• Clear statement of what the project is intended to do and why– Needs assessment– Theory of action– Clear, measurable statement of goal attainment
• Appropriate evaluation methods and tools • Transparency
![Page 9: Program Evaluation Strategies to Improve Teaching for Learning Rossi Ray-Taylor and Nora Martin Ray.Taylor and Associates MDE/NCA Spring School Improvement.](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649caf5503460f9497254f/html5/thumbnails/9.jpg)
Evaluation can lay the ground work for accountability
![Page 10: Program Evaluation Strategies to Improve Teaching for Learning Rossi Ray-Taylor and Nora Martin Ray.Taylor and Associates MDE/NCA Spring School Improvement.](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649caf5503460f9497254f/html5/thumbnails/10.jpg)
Key concepts
• Identify the total system impact of program and financial decisions
• Institutions led by data and evidence of results
![Page 11: Program Evaluation Strategies to Improve Teaching for Learning Rossi Ray-Taylor and Nora Martin Ray.Taylor and Associates MDE/NCA Spring School Improvement.](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649caf5503460f9497254f/html5/thumbnails/11.jpg)
Types of Data
• Achievement
• Demographic
• Program
• Perception
• Process
• Costs
![Page 12: Program Evaluation Strategies to Improve Teaching for Learning Rossi Ray-Taylor and Nora Martin Ray.Taylor and Associates MDE/NCA Spring School Improvement.](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649caf5503460f9497254f/html5/thumbnails/12.jpg)
Qualities of Data & Information• Time series, repeated data collection–
how does the effect change over time?
• Cohort analysis – overall effect on one group over time
• Benchmarking & standards – makes the data relative, establishes context and comparables
![Page 13: Program Evaluation Strategies to Improve Teaching for Learning Rossi Ray-Taylor and Nora Martin Ray.Taylor and Associates MDE/NCA Spring School Improvement.](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649caf5503460f9497254f/html5/thumbnails/13.jpg)
• Triangulation – looking for information in multiple indicators
• Pattern & trend analysis
• Leading & lagging indicators
![Page 14: Program Evaluation Strategies to Improve Teaching for Learning Rossi Ray-Taylor and Nora Martin Ray.Taylor and Associates MDE/NCA Spring School Improvement.](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649caf5503460f9497254f/html5/thumbnails/14.jpg)
Practical matters
• Accuracy
• Reliability
• Validity
• Accessibility
![Page 15: Program Evaluation Strategies to Improve Teaching for Learning Rossi Ray-Taylor and Nora Martin Ray.Taylor and Associates MDE/NCA Spring School Improvement.](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649caf5503460f9497254f/html5/thumbnails/15.jpg)
Evaluation
Employ evaluation strategies from the very beginning of a project and assemble and review effectiveness data. State measurable outcomes, document processes and review progress throughout the life of the project.
![Page 16: Program Evaluation Strategies to Improve Teaching for Learning Rossi Ray-Taylor and Nora Martin Ray.Taylor and Associates MDE/NCA Spring School Improvement.](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649caf5503460f9497254f/html5/thumbnails/16.jpg)
“Where outcomes are evaluated without knowledge of implementation, the results seldom provide a direction for
action because the decision maker lacks information about what produced the outcomes (or lack of
outcomes).” Michael Quinn Patten
Quoted in “Data Analysis for Comprehensive Schoolwide Improvement”,
Victoria L. Bernhardt, 1998
![Page 17: Program Evaluation Strategies to Improve Teaching for Learning Rossi Ray-Taylor and Nora Martin Ray.Taylor and Associates MDE/NCA Spring School Improvement.](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649caf5503460f9497254f/html5/thumbnails/17.jpg)
Central Evaluation Questions
• Did the program / project do what was intended?– Did the project stick to the plan?
There may be valid reasons for varying from the plan – if so what are they?
![Page 18: Program Evaluation Strategies to Improve Teaching for Learning Rossi Ray-Taylor and Nora Martin Ray.Taylor and Associates MDE/NCA Spring School Improvement.](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649caf5503460f9497254f/html5/thumbnails/18.jpg)
• What is the theory of change for the project?
• Why is this project being carried out this way and why is it judged to be the most appropriate way?– These questions are important because
they help when judging the impact of changes in the plan.
![Page 19: Program Evaluation Strategies to Improve Teaching for Learning Rossi Ray-Taylor and Nora Martin Ray.Taylor and Associates MDE/NCA Spring School Improvement.](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649caf5503460f9497254f/html5/thumbnails/19.jpg)
• What is the context for the project?– Issues include past trends, local politics,
resource distribution, threats, opportunities, strengths and weaknesses
![Page 20: Program Evaluation Strategies to Improve Teaching for Learning Rossi Ray-Taylor and Nora Martin Ray.Taylor and Associates MDE/NCA Spring School Improvement.](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649caf5503460f9497254f/html5/thumbnails/20.jpg)
• What actually happened during the course of the project?
• Who was served by the project? Why?
• Who was not served by the project? Why?
• Was the project valued by the intended audience?
![Page 21: Program Evaluation Strategies to Improve Teaching for Learning Rossi Ray-Taylor and Nora Martin Ray.Taylor and Associates MDE/NCA Spring School Improvement.](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649caf5503460f9497254f/html5/thumbnails/21.jpg)
• What were the inputs, resources – both real e.g., financial and material – and intellectual?
• What was the “cost” of the project?
![Page 22: Program Evaluation Strategies to Improve Teaching for Learning Rossi Ray-Taylor and Nora Martin Ray.Taylor and Associates MDE/NCA Spring School Improvement.](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649caf5503460f9497254f/html5/thumbnails/22.jpg)
• What were the results /outcomes?• Were goals and objectives met?• What were the intended and unintended
consequences?• What was the impact on the overall system?• Was there a process impact – did the project
result in a change in the way that business is done?
![Page 23: Program Evaluation Strategies to Improve Teaching for Learning Rossi Ray-Taylor and Nora Martin Ray.Taylor and Associates MDE/NCA Spring School Improvement.](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649caf5503460f9497254f/html5/thumbnails/23.jpg)
A successful project begins and ends with a good evaluation design
![Page 24: Program Evaluation Strategies to Improve Teaching for Learning Rossi Ray-Taylor and Nora Martin Ray.Taylor and Associates MDE/NCA Spring School Improvement.](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649caf5503460f9497254f/html5/thumbnails/24.jpg)
How do these questions improve teaching for learning?
![Page 25: Program Evaluation Strategies to Improve Teaching for Learning Rossi Ray-Taylor and Nora Martin Ray.Taylor and Associates MDE/NCA Spring School Improvement.](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649caf5503460f9497254f/html5/thumbnails/25.jpg)
The best and most “sticky”, lasting interventions have the following
components
• They are based in research and evidence
• They are locally constructed and customized
![Page 26: Program Evaluation Strategies to Improve Teaching for Learning Rossi Ray-Taylor and Nora Martin Ray.Taylor and Associates MDE/NCA Spring School Improvement.](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649caf5503460f9497254f/html5/thumbnails/26.jpg)
• They are targeted to a clear view of the problem to be solved
• They are built for sustainability
• They are “owned” – not just a matter of compliance
• They are designed to build local capacity
![Page 27: Program Evaluation Strategies to Improve Teaching for Learning Rossi Ray-Taylor and Nora Martin Ray.Taylor and Associates MDE/NCA Spring School Improvement.](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649caf5503460f9497254f/html5/thumbnails/27.jpg)
Understand change and apply research about change to
improve teaching for learningChange takes trust
Change takes building relationshipsChange takes endurance (time)
Change takes knowledge of research
![Page 28: Program Evaluation Strategies to Improve Teaching for Learning Rossi Ray-Taylor and Nora Martin Ray.Taylor and Associates MDE/NCA Spring School Improvement.](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649caf5503460f9497254f/html5/thumbnails/28.jpg)
Examine policies and practices that serve as barriers and those
that serve as catalysts to achievement
![Page 29: Program Evaluation Strategies to Improve Teaching for Learning Rossi Ray-Taylor and Nora Martin Ray.Taylor and Associates MDE/NCA Spring School Improvement.](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649caf5503460f9497254f/html5/thumbnails/29.jpg)
Evaluate
Audit the systemMeasure results
Change to goal not just awareness or implementation
![Page 30: Program Evaluation Strategies to Improve Teaching for Learning Rossi Ray-Taylor and Nora Martin Ray.Taylor and Associates MDE/NCA Spring School Improvement.](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649caf5503460f9497254f/html5/thumbnails/30.jpg)
Meta Evaluation
Use meta evaluation strategies to look for results across projects and interventions
![Page 31: Program Evaluation Strategies to Improve Teaching for Learning Rossi Ray-Taylor and Nora Martin Ray.Taylor and Associates MDE/NCA Spring School Improvement.](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649caf5503460f9497254f/html5/thumbnails/31.jpg)
Evaluation readiness
Identify and clearly state program goals and outcomes
Transform these goals into measurable objectives
Define program theory and supporting research
![Page 32: Program Evaluation Strategies to Improve Teaching for Learning Rossi Ray-Taylor and Nora Martin Ray.Taylor and Associates MDE/NCA Spring School Improvement.](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649caf5503460f9497254f/html5/thumbnails/32.jpg)
Develop the formative and summative evaluation plan
Develop the plan to gather data, deploy evaluation resources and gather information from and report to stakeholders
![Page 33: Program Evaluation Strategies to Improve Teaching for Learning Rossi Ray-Taylor and Nora Martin Ray.Taylor and Associates MDE/NCA Spring School Improvement.](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649caf5503460f9497254f/html5/thumbnails/33.jpg)
Design for continuous feedback and transparency
![Page 34: Program Evaluation Strategies to Improve Teaching for Learning Rossi Ray-Taylor and Nora Martin Ray.Taylor and Associates MDE/NCA Spring School Improvement.](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649caf5503460f9497254f/html5/thumbnails/34.jpg)
Consider system evaluation policies and expectations
At the time of proposal initiatives, programs and projects should be designed to include program evaluation
![Page 35: Program Evaluation Strategies to Improve Teaching for Learning Rossi Ray-Taylor and Nora Martin Ray.Taylor and Associates MDE/NCA Spring School Improvement.](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649caf5503460f9497254f/html5/thumbnails/35.jpg)
“It’s easy to make judgments – that’s evaluation. It’s easy to ask questions about impact – that’s evaluation. It’s easy to disseminate reports –
that’s evaluation. What’s hard is to put all those pieces together in a meaningful whole which tells
people something they want to know and can use about a matter of importance. That’s
evaluation.” HalcolmQuoted in “Data Analysis for Comprehensive Schoolwide Improvement”,
Victoria L. Bernhardt, 1998
![Page 36: Program Evaluation Strategies to Improve Teaching for Learning Rossi Ray-Taylor and Nora Martin Ray.Taylor and Associates MDE/NCA Spring School Improvement.](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649caf5503460f9497254f/html5/thumbnails/36.jpg)
Program evaluation planning document Goal: What does this project intend to achieve? State goals in observable terms. Limit the project to 3-5 overall goals.
Objective
State project objectives in measurable terms
Measure/Criteria
How will the objective be measured? What tools or methods will be used?What are the criteria for success?
Evaluation activities & questions
What activities and tasks need to be accomplished to carry out evaluation of this objective?
State the evaluation questions
Time line
When will data be collected and reported?
Person responsible
Who is responsible for this task?
Resources and budget
What is the budget and what other resources will be needed?
![Page 37: Program Evaluation Strategies to Improve Teaching for Learning Rossi Ray-Taylor and Nora Martin Ray.Taylor and Associates MDE/NCA Spring School Improvement.](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649caf5503460f9497254f/html5/thumbnails/37.jpg)
In the beginning you think. In the end you act. In between you negotiate the possibilities. Some people move from complexity to simplicity and on into catastrophe. Others move from simplicity to
complexity and onward into full scale confusion. Simplification makes action possible in the face of overwhelming complexity. It
also increases the odds of being wrong. The trick is to let a sense of simplicity inform our thinking, a sense of complexity inform our
actions, and a sense of humility inform our judgments…” Michael Quinn Patten (p. 143 Bernhardt)
Quoted in “Data Analysis for Comprehensive Schoolwide Improvement”,
Victoria L. Bernhardt, 1998
![Page 38: Program Evaluation Strategies to Improve Teaching for Learning Rossi Ray-Taylor and Nora Martin Ray.Taylor and Associates MDE/NCA Spring School Improvement.](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649caf5503460f9497254f/html5/thumbnails/38.jpg)
Resources
• Center for Evaluation & Education Policy www.ceep.indiana.edu
• The Evaluation Center; Western Michigan University
![Page 39: Program Evaluation Strategies to Improve Teaching for Learning Rossi Ray-Taylor and Nora Martin Ray.Taylor and Associates MDE/NCA Spring School Improvement.](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649caf5503460f9497254f/html5/thumbnails/39.jpg)
More Resources
• What Works in Schools Translating Research into Action by Robert J. Marzano
• Data Analysis for Comprehensive Schoolwide Improvement; Victoria L. Bernhardt
![Page 40: Program Evaluation Strategies to Improve Teaching for Learning Rossi Ray-Taylor and Nora Martin Ray.Taylor and Associates MDE/NCA Spring School Improvement.](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649caf5503460f9497254f/html5/thumbnails/40.jpg)
More Resources
• The “Data Wise” Improvement Process: Eight steps for using test data to improve teaching and learning, by Kathryn Parker Boudett et al in Harvard Education Letter, January/February 2006 Volume 22, Number 1.
![Page 41: Program Evaluation Strategies to Improve Teaching for Learning Rossi Ray-Taylor and Nora Martin Ray.Taylor and Associates MDE/NCA Spring School Improvement.](https://reader036.fdocuments.us/reader036/viewer/2022062423/56649caf5503460f9497254f/html5/thumbnails/41.jpg)
Rossi Ray-Taylor, PhD [email protected]
Nora Martin, [email protected]
Ray.Taylor and Associates2160 S. Huron Parkway, Suite 3
Ann Arbor, Michigan 48104www.raytaylorandassoc.org