Models for Evaluating MSP Projects Evaluation of Professional Development Programs MSP Regional...
-
Upload
gyles-atkinson -
Category
Documents
-
view
213 -
download
0
Transcript of Models for Evaluating MSP Projects Evaluation of Professional Development Programs MSP Regional...
![Page 1: Models for Evaluating MSP Projects Evaluation of Professional Development Programs MSP Regional Conference Dallas, Texas February 7, 2007 Norman L. Webb.](https://reader035.fdocuments.us/reader035/viewer/2022070400/56649e265503460f94b1536a/html5/thumbnails/1.jpg)
Models for Evaluating MSP ProjectsEvaluation of Professional Development Programs
MSP Regional ConferenceDallas, Texas
February 7, 2007
Norman L. WebbSenior Research Scientist
Wisconsin Center for Education Research
University of WisconsinMadison, Wisconsin
![Page 2: Models for Evaluating MSP Projects Evaluation of Professional Development Programs MSP Regional Conference Dallas, Texas February 7, 2007 Norman L. Webb.](https://reader035.fdocuments.us/reader035/viewer/2022070400/56649e265503460f94b1536a/html5/thumbnails/2.jpg)
![Page 3: Models for Evaluating MSP Projects Evaluation of Professional Development Programs MSP Regional Conference Dallas, Texas February 7, 2007 Norman L. Webb.](https://reader035.fdocuments.us/reader035/viewer/2022070400/56649e265503460f94b1536a/html5/thumbnails/3.jpg)
Program StagesPlanningImplementation
Impact
![Page 4: Models for Evaluating MSP Projects Evaluation of Professional Development Programs MSP Regional Conference Dallas, Texas February 7, 2007 Norman L. Webb.](https://reader035.fdocuments.us/reader035/viewer/2022070400/56649e265503460f94b1536a/html5/thumbnails/4.jpg)
First Steps in Initiating an EvaluationFinding an evaluator
External/internal
Independent/within programCost for evaluationScope for evaluation
![Page 5: Models for Evaluating MSP Projects Evaluation of Professional Development Programs MSP Regional Conference Dallas, Texas February 7, 2007 Norman L. Webb.](https://reader035.fdocuments.us/reader035/viewer/2022070400/56649e265503460f94b1536a/html5/thumbnails/5.jpg)
Evaluation for Program Planning
![Page 6: Models for Evaluating MSP Projects Evaluation of Professional Development Programs MSP Regional Conference Dallas, Texas February 7, 2007 Norman L. Webb.](https://reader035.fdocuments.us/reader035/viewer/2022070400/56649e265503460f94b1536a/html5/thumbnails/6.jpg)
Compatibility between Program Design and
Evaluation Design
![Page 7: Models for Evaluating MSP Projects Evaluation of Professional Development Programs MSP Regional Conference Dallas, Texas February 7, 2007 Norman L. Webb.](https://reader035.fdocuments.us/reader035/viewer/2022070400/56649e265503460f94b1536a/html5/thumbnails/7.jpg)
No causation without manipulation
![Page 8: Models for Evaluating MSP Projects Evaluation of Professional Development Programs MSP Regional Conference Dallas, Texas February 7, 2007 Norman L. Webb.](https://reader035.fdocuments.us/reader035/viewer/2022070400/56649e265503460f94b1536a/html5/thumbnails/8.jpg)
Experimental and Quasi-Experimental Designs Pre-Post Test Pre-Post Test with a Control Group Post Test Only Post Test with a Control Group Random Assignment Multiple Levels
Teacher – StudentSchool – Student
Trend Analysis—Baseline, Periodic Assessments
![Page 9: Models for Evaluating MSP Projects Evaluation of Professional Development Programs MSP Regional Conference Dallas, Texas February 7, 2007 Norman L. Webb.](https://reader035.fdocuments.us/reader035/viewer/2022070400/56649e265503460f94b1536a/html5/thumbnails/9.jpg)
Professional Development Program Design
Selection of Teachers All Volunteers Purposeful
Selection of Schools All Volunteers Purposeful Stratified
![Page 10: Models for Evaluating MSP Projects Evaluation of Professional Development Programs MSP Regional Conference Dallas, Texas February 7, 2007 Norman L. Webb.](https://reader035.fdocuments.us/reader035/viewer/2022070400/56649e265503460f94b1536a/html5/thumbnails/10.jpg)
Evaluation of Program Theory
Research Based Logic
![Page 11: Models for Evaluating MSP Projects Evaluation of Professional Development Programs MSP Regional Conference Dallas, Texas February 7, 2007 Norman L. Webb.](https://reader035.fdocuments.us/reader035/viewer/2022070400/56649e265503460f94b1536a/html5/thumbnails/11.jpg)
![Page 12: Models for Evaluating MSP Projects Evaluation of Professional Development Programs MSP Regional Conference Dallas, Texas February 7, 2007 Norman L. Webb.](https://reader035.fdocuments.us/reader035/viewer/2022070400/56649e265503460f94b1536a/html5/thumbnails/12.jpg)
Simon, 1997 Mathematics Teaching Cycle
![Page 13: Models for Evaluating MSP Projects Evaluation of Professional Development Programs MSP Regional Conference Dallas, Texas February 7, 2007 Norman L. Webb.](https://reader035.fdocuments.us/reader035/viewer/2022070400/56649e265503460f94b1536a/html5/thumbnails/13.jpg)
Logic Model Framework for Listing Program Activities
![Page 14: Models for Evaluating MSP Projects Evaluation of Professional Development Programs MSP Regional Conference Dallas, Texas February 7, 2007 Norman L. Webb.](https://reader035.fdocuments.us/reader035/viewer/2022070400/56649e265503460f94b1536a/html5/thumbnails/14.jpg)
Professional Development Activities
Curriculum and Instruction
ACTIVITIES D4)MTLs support other teachers
D5)Teachers -after school weekly classes
D6)Teachers -one-week summer institute
E4) E5) E6)Teachers -bi-weekly grade level meetings
F4)MTLs monthly two-day workshops on leadership
F5) F6)
![Page 15: Models for Evaluating MSP Projects Evaluation of Professional Development Programs MSP Regional Conference Dallas, Texas February 7, 2007 Norman L. Webb.](https://reader035.fdocuments.us/reader035/viewer/2022070400/56649e265503460f94b1536a/html5/thumbnails/15.jpg)
Professional Development Interim Outcomes
Curriculum and Instruction
INTERIM I4)MTLs’ beliefs, attitudes, and practices become aligned with reflective teaching
I5)Teachers build confidence in reflective teaching of mathematics
I6)Teachers independently use reflective teaching strategies
J4)MTLs gain leadership skills and recognize teachers’ growth in reflective teaching
J5)Teachers build deeper understanding of content and student thinking
J6)
K4) K5)Teachers gain skills at recognizing student mathematical thinking and experiment with techniques
K6)Teachers make instructional and curriculum decisions based on students’ understanding
L4) L5)Teachers evaluate students’ understanding based on viable model of how mathematical understanding develops
L6)Teachers learn to manage reflective teaching classroom
![Page 16: Models for Evaluating MSP Projects Evaluation of Professional Development Programs MSP Regional Conference Dallas, Texas February 7, 2007 Norman L. Webb.](https://reader035.fdocuments.us/reader035/viewer/2022070400/56649e265503460f94b1536a/html5/thumbnails/16.jpg)
Program Implementation and Evaluation
![Page 17: Models for Evaluating MSP Projects Evaluation of Professional Development Programs MSP Regional Conference Dallas, Texas February 7, 2007 Norman L. Webb.](https://reader035.fdocuments.us/reader035/viewer/2022070400/56649e265503460f94b1536a/html5/thumbnails/17.jpg)
Guskey’s Five Critical Levels of Evaluating Professional Development
1. Participants’ Reactions, 2. Participants’ Learning, 3. Organization Support for Change, 4. Participants’ Use of New Knowledge and Skills5. Student Learning Outcomes.
Guskey, T. R. (2000). Evaluating professional development. Thousand Oaks, CA: Corwin Press. As reported by Monsaas and McGee-Brown, Evaluating Professional Learning Communities Using Mixed, Quantitative and Qualitative Methodology, MSP Evaluation Summit, September 2005. CD available from Consortium for Building Evaluation Capacity, Utah State University.
![Page 18: Models for Evaluating MSP Projects Evaluation of Professional Development Programs MSP Regional Conference Dallas, Texas February 7, 2007 Norman L. Webb.](https://reader035.fdocuments.us/reader035/viewer/2022070400/56649e265503460f94b1536a/html5/thumbnails/18.jpg)
Some Issues Related to Evaluating the Implementation of Professional Development
Time Frame Proof-of-Concept Studies Breadth vs. Depth Formative Studies Aligned Measurement Instruments Living with Imperfection Analytic Horizon Mismatch with Funding
![Page 19: Models for Evaluating MSP Projects Evaluation of Professional Development Programs MSP Regional Conference Dallas, Texas February 7, 2007 Norman L. Webb.](https://reader035.fdocuments.us/reader035/viewer/2022070400/56649e265503460f94b1536a/html5/thumbnails/19.jpg)
Frequently Used Methods in Evaluation of Professional Development Programs
Case Studies Experimental Designs Multi-Variant Designs Trend Analyses
![Page 20: Models for Evaluating MSP Projects Evaluation of Professional Development Programs MSP Regional Conference Dallas, Texas February 7, 2007 Norman L. Webb.](https://reader035.fdocuments.us/reader035/viewer/2022070400/56649e265503460f94b1536a/html5/thumbnails/20.jpg)
Frequently Used Tools in Evaluating Professional Development Programs
Surveys Observations Interviews Focus Groups Teacher Logs Content Knowledge Tests
![Page 21: Models for Evaluating MSP Projects Evaluation of Professional Development Programs MSP Regional Conference Dallas, Texas February 7, 2007 Norman L. Webb.](https://reader035.fdocuments.us/reader035/viewer/2022070400/56649e265503460f94b1536a/html5/thumbnails/21.jpg)
Less Frequently Used Tools
Content Coverage Maps Concept Maps Student Work Samples
![Page 22: Models for Evaluating MSP Projects Evaluation of Professional Development Programs MSP Regional Conference Dallas, Texas February 7, 2007 Norman L. Webb.](https://reader035.fdocuments.us/reader035/viewer/2022070400/56649e265503460f94b1536a/html5/thumbnails/22.jpg)
Program Impact
![Page 23: Models for Evaluating MSP Projects Evaluation of Professional Development Programs MSP Regional Conference Dallas, Texas February 7, 2007 Norman L. Webb.](https://reader035.fdocuments.us/reader035/viewer/2022070400/56649e265503460f94b1536a/html5/thumbnails/23.jpg)
Math Grade 5
-10
-5
0
5
10
15
AlACiMa Cohort I Schools Other schools
Q1
5%
median
95%
Q3
Value-Added Effect on Mathematics Scores from Grade 4 to Grade 5, 2003-2005
![Page 24: Models for Evaluating MSP Projects Evaluation of Professional Development Programs MSP Regional Conference Dallas, Texas February 7, 2007 Norman L. Webb.](https://reader035.fdocuments.us/reader035/viewer/2022070400/56649e265503460f94b1536a/html5/thumbnails/24.jpg)
Spanish Grade 5
-10
-5
0
5
10
15
AlACiMa Cohort I Schools Other Schools
Q1
5%
median
95%
Q3
Value-Added Effect on Spanish Scores from Grade 4 to Grade 5, 2004-2005
![Page 25: Models for Evaluating MSP Projects Evaluation of Professional Development Programs MSP Regional Conference Dallas, Texas February 7, 2007 Norman L. Webb.](https://reader035.fdocuments.us/reader035/viewer/2022070400/56649e265503460f94b1536a/html5/thumbnails/25.jpg)
Grade 5Subject Matter
AlACiMa Parameter Estimate
Standard Error t Value Pr > │t │
Mathematics 1.26 0.68 1.86 0.06
Spanish -0.07 0.58 -0.12 0.90
Value Added Effect of AlAciMa Schools for Grade 5 by Content Area for Cohort 1 2004-2005
![Page 26: Models for Evaluating MSP Projects Evaluation of Professional Development Programs MSP Regional Conference Dallas, Texas February 7, 2007 Norman L. Webb.](https://reader035.fdocuments.us/reader035/viewer/2022070400/56649e265503460f94b1536a/html5/thumbnails/26.jpg)
600
620
640
660
680
700
720
740
760
2000 2001 2002 2003 2004 2005
Year
Scal
e Sc
ore 4th
8th10th
MMSD Mean Math Scale Scores for Grades 4, 8, and 10 2000-2005
![Page 27: Models for Evaluating MSP Projects Evaluation of Professional Development Programs MSP Regional Conference Dallas, Texas February 7, 2007 Norman L. Webb.](https://reader035.fdocuments.us/reader035/viewer/2022070400/56649e265503460f94b1536a/html5/thumbnails/27.jpg)
0
100
200
300
400
500
600
700
800
900
1000
2000 2001 2002 2003 2004 2005
Year
Scal
e Sc
ore
MMSD Math Mean, First and Third quartile, and 5th and 95th Percentile Scores for Grade 8 2000-2005
![Page 28: Models for Evaluating MSP Projects Evaluation of Professional Development Programs MSP Regional Conference Dallas, Texas February 7, 2007 Norman L. Webb.](https://reader035.fdocuments.us/reader035/viewer/2022070400/56649e265503460f94b1536a/html5/thumbnails/28.jpg)
n = 1761
n = 1726
n = 1729n = 1697
n = 1666n = 1706
-0.1
0
0.1
0.2
0.3
0.4
2000 2001 2002 2003 2004 2005
Year
Eff
ect S
ize*
Normalized Scale Score Effect Size Between MMSD and WI for Math Grade 8 2000-2005
![Page 29: Models for Evaluating MSP Projects Evaluation of Professional Development Programs MSP Regional Conference Dallas, Texas February 7, 2007 Norman L. Webb.](https://reader035.fdocuments.us/reader035/viewer/2022070400/56649e265503460f94b1536a/html5/thumbnails/29.jpg)
Evaluators Need to be Realistic
Program design will change(27 of 40 evaluators, 68%)
Collecting student data is challenging Access to school, teachers, and students can
be difficult There is a trade off between collecting data
and writing about results Qualitative and quantitative data collections
requires a balance
![Page 30: Models for Evaluating MSP Projects Evaluation of Professional Development Programs MSP Regional Conference Dallas, Texas February 7, 2007 Norman L. Webb.](https://reader035.fdocuments.us/reader035/viewer/2022070400/56649e265503460f94b1536a/html5/thumbnails/30.jpg)
What Has Worked for MSP Evaluators
Continual Communication with Stakeholders
Conducting Case Studies Analysis of Teacher-Learning Logs a
Value to Stakeholders
![Page 31: Models for Evaluating MSP Projects Evaluation of Professional Development Programs MSP Regional Conference Dallas, Texas February 7, 2007 Norman L. Webb.](https://reader035.fdocuments.us/reader035/viewer/2022070400/56649e265503460f94b1536a/html5/thumbnails/31.jpg)
Final Thoughts
It is not the direction of the wind, but the set of the sails that guides the boat.