Post on 19-Dec-2015
Using Classroom Vignettes to Measure Teachers’
Instructional Practices in Mathematics
Brian Stecher, Vi-Nhuan Le, Laura Hamilton, et al.
RAND
Mosaic II Project Overview
Research Question: What is the relationship between student achievement and exposure to reform-oriented curriculum and instruction?• In the context of NSF Local Systemic Change and
Systemic Reform Initiatives• Mathematics and science• 3-year longitudinal tracking of students• Grades 3-4-5 or 6-7-8• Multiple measures of teaching practices,
including vignette-based survey items• Multiple-choice and open-ended measures of
achievement
The Challenge of Measuring Instructional Practice
Common methods for measuring instructional practice have disadvantages that can limit their validity
METHOD DISADVANTAGES
Observations Complex, time consuming, expensive, subject to observer bias, difficult to summarize, etc.
Surveys Retrospective, memory and time distortions, subject to self-report bias, lack of shared understanding of reform terminology, etc.
Logs Limited scope, self-report bias, lack of shared understanding of reform terminology, etc.
Study a New Measure of Reform-Oriented Teaching
Develop vignette-based survey items
• Clear, situated descriptions of instructional events
• Core mathematics and science curriculum
• Admit reform-oriented teacher actions
Examine the validity of vignettes as a measure of instructional practice
• Consistency of teacher responses across “parallel” vignettes
• Relationship of vignettes with other measures of instruction, including surveys, logs, and observations
Focus on 4th grade mathematics
Rational for Vignette-Based Approach
Attempting to measure “intention” to engage in reform-oriented teaching based on responses to specific, hypothetical, familiar situations with clear behavioral alternatives.
ADVANTAGES DISADVANTAGES
• More realistic situations and choices
• Minimizes problems of reform terminology
• Represents a standardized situation
• Imposes standardized responses
• Greater reading burden
• Difficult to develop
• Verisimilitude?
• Unknown validity
Steps in Developing the Vignettes
Convened a panel of mathematics experts
Used NCTM standards and other documents to create a framework • Dimensions of reform-oriented mathematics
curriculum and instruction • Framed in behavioral terms
Selected two common 4th grade mathematics topics as setting
• Area/perimeter
• Two-digit multiplication
Format of the Vignettes
Instructions on how to respond
A context section specifying:• Length of the unit • Prior learning history of the students
Four separate but linked instructional problems: • Introducing the lesson• Responding to student error• Reconciling different approaches• Choosing learning objectives
Response options that represented a range of low- and high-reform actions• Likelihood of responding in this manner?
Designed to be “parallel” across the two topics
18. You are almost at the end of the unit on multiplying two-digit numbers. You ask students to work in pairs orgroups to solve the following problem.
Each school bus can hold 41 people. How many people can go on a field trip if the district has 23 busesavailable? Find the answer and demonstrate why it is correct.
After working on the problem for a while, you ask each group if they will share their work.
The first group says the answer is 943, and they use the standard algorithm to show how they obtained thisresult.
41
x 23__________ 123
82__________ 943
The second group says the answer is 943, and they explain that they broke the numbers into tens and onesand used the distributive property.
41 x 23 = (40 +1) x (20 + 3) = (40 x 20) + (40 x 3) + (1 x 20) + (1 x 3) = 800 + 120 + 20 + 3 = 943
After praising both groups for using eff ective strategies, how likely are you to do each of the following in responseto these two explanations?
(Circle One Response in Each Row)
Very unlikely Somewhatunlikely
Somewhatlikely
Verylikely
a. Ask the class if they can think of another way to solvethe problem
1 2 3 4
b. Suggest that the class check the results by using acalculator
1 2 3 4
c. Tell them the first group’s method is faster1 2 3 4
d. Tell them they are both right and move on to the nextproblem
1 2 3 4
e. Have a classroom discussion about the diff erencesbetween the two approaches
1 2 3 4
Setting Classroom Context
Imagine you are teaching mathematics to a fourth-grade class. The students are familiar with place value and with the multiplication facts up to 10 times 10. You are about to begin a week-long unit on multiplication involving larger numbers.
Introducing the Problem Scenario
You ready to start the unit by introducing multiplication of two-digit numbers. How likely are you to do each of the following as an initial activity for the unit?
• Ask students what they know about multiplying two-digit numbers
• Have students work in groups to solve problems such as finding the number of inches in 20 feet
• Demonstrate the standard algorithm for multiplying 2 two-digit numbers
Note: Options Continue
Responding to Student Error Scenario
Part way through unit… Students know standard algorithm…
You ask your students to figure out how many miles Jerry can drive on 17 gallons of gas if Jerry’s car goes 23 miles per gallon of gas.
You ask Leah if she will share her solution. She says “I multiplied 20 times 20 and got 400. This works because 23 is three more than 20 and 17 is three less than 20.” How likely are you to do the following?
• Ask Leah if subtracting 3 miles per gallon is the same as adding 3 gallons of gas
• Explain that 23 x 17 is actually 391. Then pose another similar problem
Note: Options Continue
Reconciling Different Approaches Scenario
Almost at end of unit…
You ask students to work in groups to find how many people can go on a field trip if the district has 23 buses available, and each school bus can hold 41 people.
The first group says the answer is 943, and uses the standard algorithm [figure]. The second group gives the same answer, but explain that they broke the numbers into tens and ones and used the distributive property [figure]. How likely are you to do the following?
• Have a classroom discussion about the differences between the two approaches
• Tell them that the first group’s method is faster Note: Options Continue
Choosing Learning Objectives Scenario
If you were to design a unit on two-digit multiplication, how much emphasis would you place on each of the following objectives?
• Students will understand how place value relates to two-digit multiplication
• Students will be able to use place-value blocks or graph paper to represent two-digit multiplication
• Students will be able to define the terms ‘factor’ and ‘product’
Note: Options Continue
Study Methods
Sample
• 80 fourth-grade teachers in 20 schools from a district participating in the Local Systemic Change program
• Observed 39 teachers from 13 schools
Measures of instruction
• Surveys
• Logs
• Vignettes
• Classroom observations
Creating Vignette-Based Scores of Reform-Oriented Teaching
Assigned a value to each response option (low- to high-reform)
Create two scales
• Reform Inclinations
• Standardized sum of teachers’ answers to the high-reform response options (N = 27 items)
• Euclid
• Euclidean distance of each teacher from an “ideal” high reform teacher (N = 51 items)
.80)(á =
.86)(á =
Distribution of Vignette-Based Score of Reform Teaching (Euclid)
Other Measures from Survey, Logs and Observation
Processes: Extent to which teachers emphasized NCTM-endorsed cognitive processes (N = 5 items)
Reform: Frequency with which the teachers engage in reform-oriented practices (N = 9 items)
Discussion: Amount of time the class engaged in dialogue about mathematical thinking and understanding (N = 4 items)
Seatwork: Extent to which students engaged in seatwork and other low-reform activities (N = 2 items)
Understanding: Extent to which the teacher facilitated deeper mathematical understanding (N = 5 items)
Findings Regarding Vignette-Based Measures
Responses are stable across “parallel” mathematical contexts
“Reform Inclination” scale correlates with several survey and log measures (but not observational measures)
“Euclidean” scale correlates with observational measures (but not survey and log measures)
Mixed evidence regarding validity of vignette-based scales
Next Steps
Examine relationships of vignette-based measures with student outcomes, particularly achievement
Improve the quality of vignettes through “think aloud” interviews with teachers
Investigate how length and level of detail in vignettes affect how teachers respond
Explore the applicability of vignettes in other contexts