A learning progressions approachMeasuring teachers' use of formative assessment:Brent Duckor (SJSU)Diana Wilmot (PAUSD)Bill Conrad & Jimmy ScheRrer (SCCOE)Amy Dray (UCB)

Why formative assessment?Black and Wiliam (1998) report that studies of formative assessment (FA) show an effect size on standardized tests between 0.4 and 0.7, far larger than most educational interventions and equivalent to approximately 8 months of additional instruction;Further, FA is particularly effective for low achieving students, narrowing the gap between them and high achievers, while raising overall achievement; Enhancing FA would be an extremely cost-effective way to improve teaching practice; Unfortunately, they also find that FA is relatively rare in the classroom, and that most teachers lack effective FA skills.

Measuring Effective FA practice: Toward a cycle of psychometric inquiryDefine constructs in multi-dimensional spaceDesign items and observations protocolIterate scoring strategy in alignment construct mapsApply measurement model to validate teacher and item-level claims and to warrant inferences about effective practice

teachers who practice assessment for learning know and can understand and articulate in advance of teaching the achievement targets that their students are to hit;inform their students about those learning goals, in terms that students understand;translate classroom assessment results into frequent descriptive feedback for students, providing them with specific insights as to how to improve;continuously adjusting instruction based on the results of classroom assessments.

Research suggests

Good formative feedbackTypes of feedback (Butler, 1998; Butler & Neuman, 1995; Hattie & Timperley, 2007; Kluger & DeNisi, 1996); Level of specificity and task relatedness (Tunstall & Gipps, 1996; Ames, 1992; Dweck, 1986); The next steps required of students (Butler & Neuman, 1995) can influence the effectiveness of formative assessment on classroom learning.

The Zone of study

Items DesignOutcome SpaceMeasurementModelReliabilityValidityConstruct Map

Phase 1: Construct Mapping

Knowledge of formative assessment

Mapping a Route

Skills Allocationat Soliciting responses level

Chart1

6

3

1

Allocation

Knows what you do, not yet sure why you need it

Sheet1

Allocation

Posing6

Pausing3

Pouncing1

To resize chart data range, drag lower right corner of range.

Construct MapItem DesignMeasurementModelReliabilityValidityOutcome Space

Phase 2: Items DesignAlign items (tasks, prompts, scenarios) with the levels on the construct map.Building items to map onto a refined set of formative assessment practices.Consider various item types and delivery platforms.Review an example of a simulated scenario with focus on turns of talk.Consider content and construct validity, as well as inter-rater reliability in construction and use of items.

Consider the context: The Lesson CycleInitial framing of a question within a HIGH LEVEL Task.Study the enactment of formative assessment implementation within a science classroom engaged in HL tasks in ecology.

The evolution of a TaskThe Enacted TaskStein, Remillard, & Smith (2007)

Delivery Platforms for data collectionTraditional paper and pencil questionsClassroom based authentic tasksLesson planning/enactment/reflectionsLesson study of best formative assessment practicesInnovative adaptive virtual scenarios (AVS)Video episodesWeb-based virtual platforms

Adaptive VIRTUAL scenariosAbility to share a range of novice through expert formative assessment practice in video format.Pause videos throughout teacher enactment to measure teachers level of sophistication.

Potential items within Simulated SCENARIOs Pause video and ask the teacher:How would you characterize this initial move? What do you notice about the teachers questioning strategy?How might you compare one questioning strategy with another?What would you do next?

Replay video and show teachers follow up moves.

Pause and ask the teacher what they think:How would you negotiate this student response?What kind of question would you pose next?

Sample of teacher responsesInitial move was literal, more open ended.Teachers response to students was: too directed towards the right answer, thoughtfully provoking misconceptions, open enough to provoke deeper conceptual knowledge across the classroomI would have asked a question like,[a question that uses student thinking as a basis]

Administration of toolsAdminister tools with widest range of teachers possible pre-service Induction yearsVeteran 5-9 yearsVeteran 10+ yearsUse adaptive technology to capture teachers zone of proximal development in their expertise of formative assessment practicesExamine relationship (if any) between scores on multiple tools

Construct MapItems DesignMeasurementModelReliabilityValidityOutcome Space

Phase 3: Defining the outcome spaceLink each item response back to the levels on the construct map.Using scoring guides to capture granularity of formative assessment practices.Consider various types e.g. rubrics, observation protocols, codingProvide exemplars of practice to assist in scoring protocols.Consider content and construct validity, as well as inter-rater reliability in construction and use of scoring guides.

Problem of Practice:Formative assessmentTeachers have difficulty scaffolding student thinking and reasoning through discourse As a result, the cognitive demand of a task often declines during implementation (e.g., TIMSS, QUASAR)

A Tool for Measuring Teachers use of most-formative assessmentInitiate participation in classroom discussionsRespond to students contributions during a discussionScherrer & Stein (In Press)

An Example of How to Apply the Codes in ContextTeacher: What can you tell me about this shape?Juan: It has 4 right angles.Teacher: What else can you tell me?Kayla: It is a rectangle.Teacher: Okay, a rectangle. Why do you think it is a rectangle?Kayla: It has 4 sides.Teacher: Are all shapes that have 4 sides rectangles?Yasmin: It could also be a quadrilateral.Teacher: Wait. I am asking if all shapes with 4 sides are rectangles.Launch

Collect

Repeat, Uptake

Uptake-Literal

Terminal, Reinitiate

Scoring the CodesTeacher: What can you tell me about this shape?Juan: It is a quadrilateral.Teacher: What else can you tell me?Kayla: It is a rectangle.Teacher: Okay, what else?Yasmin: It has four right angles.Teacher: Okay, what about this shape over here? What can you tell me about this one? Launch

Collect

Collect

Launch

In this example, the teacher did not do anything with the student responses.+1

+0

+0

+1

Scoring the CodesTeacher: What can you tell me about this shape?Juan: It is a quadrilateral.Teacher: What else can you tell me?Kayla: It is a rectangle.Teacher: Okay, Juan said this shape is a quadrilateral, and Kayla said it is a rectangle. What is similar about quadrilaterals and rectangles? Launch

Collect

Connect

In this example, the teacher asked an open-ended question, gathered an additional response to that question, and then connected the two responses.+1

+1

+2

Connecting codes to response levels on the construct mapResponse LevelsLaunch, Collect, Connect, UptakeLaunch, Collect, UptakeLaunch, Collect, LiteralLaunch, Literal, LiteralLiteral, Literal, LiteralScoring designationsL4L3L2L1L0

Construct MapItems DesignOutcome SpaceReliabilityValidityMeasurement Model

Phase 4: applying measurement modelsCross reference qualitative construct maps with technically calibrated Wright Maps using IRT.Employ person and item statistics to check on model fitConsider various types of measurement models including facets.Provide individual and group level data on progress.Consider internal structure validity evidence for construct maps, in addition to checks on reliability of tools.

Berkeley Evaluation And Assessment Research CenterItem response theory can model a learning progression within a particular domain. For example:KSC: Knowledge of science contentKST: Knowledge of student thinkingKFA: Knowledge of formative assessment

Berkeley Evaluation And Assessment Research Center

Case study: Developing an integrated assessment system (DIAS) for teacher educationPamela Moss, University of Michigan Mark Wilson, University of California, BerkeleyGoal is to develop an assessment system that:focuses on teaching practice grounded in professional and disciplinary knowledge as it develops over time;addresses multiple purposes of a broad array of stakeholders working in different contexts; andcreates the foundation for programmatic coherence and professional development across time and institutional contexts.

Case study: DIASThe research team identified the ways in which student teachers can learn to use formative and summative assessment to guide their students learning.Developed a construct map that outlined a progression of learning in Assessment.Described the different aspects of Assessment, such as:Identifying the mathematical target to be assessed;Understanding the purposes of the assessment;Designing appropriate and feasible tasks (such as end of class checks); Developing accurate inferences about individual student and whole class learning.

Case study: DIASThe research team collected data from student teachers enrolled in the elementary mathematics teacher education program at the University of Michigan.Designed scoring guides based on our construct map.Coded videotapes from over 100 student teachers as they conducted lessons in the classroom.Coded associated collected data, such as lesson plans and reflections, since these documents contain information about what the student teachers hope to learn from the assessment(s), and what they infer about the students in their classroom.Using item response methods to determine which aspects of assessment practice are easier or more difficult for the student teachers and to thereby inform the teacher education program.

SynopsisIncredible partnershipFilling an important educational research spaceIdentified the assessment spaceFocus on the contentEmphasis on student thinkingContributions to instrumentation and methodologyMarry qualitative and quantitative data using IRT frameworkNext steps: pilot study

Contact InformationBill ConradJimmy ScherrrerSanta Clara County Office of Educationjimmy_scherrer@sccoe.org408-453-4332 (Office)510-761-2007 (Cell)Bill_Conrad@sccoe.org

Brent Duckor, Ph.D.Diana Wilmot, Ph.D.Assistant ProfessorCoordinator, Research & EvaluationCollege of EducationPalo Alto Unified School DistrictSan Jose State Universitydwilmot@pausd.orgbrent.duckor@sjsu.eduAmy Dray, Ph.D. UC Berkeley Graduate School of Educationadray@berkeley.edu

A learning progressions approachMeasuring teachers' use of formative assessment:Brent Duckor (SJSU)Diana Wilmot (PAUSD)Bill Conrad & Jimmy ScheRrer (SCCOE)Amy Dray (UCB)

**

*Call and response with audience. rorschach test

*Stiggins (2002)*Black & Wiliam (1998)*Coffey and associates (2011) have drawn attention to the notion of attentiveness and how it might condition the quality of formative feedback provided to students in the science classroom.

A particularly recent development in the literature is the focus on how teachers use of formative feedback is conditioned, in part, by their attentiveness to students thinking. This implies a degree of subject matter competence in understanding how students are likely to think about a topic, for example, the notion of mass, volume, and density in the science curriculum.

*We situate our project on the most-formative side of the assessment continuum. Specifically, we are interested in how teachers use language as a tool to assess their students thinking and reasoning, and then how they use their students thinking and reasoning to co-construct knowledge. *Duckor, Draney & Wilson (2009)*SCK: Subject content knowledge e.g. Biology; Ecosystems, AdaptationKST: Knowledge of student thinking e.g. prior knowledge, preconceptions, misconceptions, learning progressionsKFA: Knowledge of formative assessment*This slide illustrates a learning progression for gaining competency with FA practice over a teachers lifetime. *This slide illustrates a potential (generic) FA skills mix that world be observable at Soliciting level

What about bounce?

*Duckor, Draney & Wilson (2009)*Note that Coffey et al. (2011) found low level questioning around science content and that teachers did not have opportunity to demonstrate range of (posing question) choices in teaching enactments*What we know: We know different tasks position students into different types of thinkers they provide different opportunities to learn (Doyle, 1983; Stein & Smith, 1998). Although there is a relationship between the cognitive demand of a task and student learning (e.g., Stein & Lane, 1996), student learning really depends on how tasks are worked on in the classroom (Hiebert & Wearne, 1993). Teachers play a critical role in determining how tasks are worked on; they mediate the relationship between tasks and student achievement. Even with highly prescriptive curricula, there is great variation in the enactment of tasks (Collopy, 2003; Rowan & Correnti, 2009; Stein, Remillard, & Smith, 2007 ).

There is a growing body of literature reporting transformations of tasks as they unfold in a series of temporal phases (e.g., Stein, Grover, and Henningsen, 1996; Tarr, Chavez, Reys, & Reys, 2006; Stigler & Hiebert, 2004). Two phases that are often discussed are the written task (tasks as they appear in the curricular materials) and the enacted task (tasks as worked on by students). Close examination of classroom practice reveals the cognitive demand of the written task often diminishes during enactment.

Specifically, we are interested in how teachers assess and scaffold student thinking and reasoning during the enactment of high-level, cognitively demanding tasks.

The classroom activities that occur during the enactment phase most directly influence how students experience the content and what they learn (Carpenter & Fenemma, 1988; Wittrock, 1986).

It is how the teacher guides students through the enactment of the task that enables or constrains opportunities to learn. One can compare this experience to a guided tour of a museum. It is not the exhibits in and of themselves that produce the learning; rather, it is how one interacts with the exhibits that determines what will be learned. Likewise, in the classroom, the tasks in and of themselves do not produce the learning; rather, it is how students interact with the tasks that determines what will be learned.

*Traditional paper and pencil questions

Homegrown diagnostic disrict

Classroom based authentic tasks:

Pre-service items design e.g. PACT (Tier I) and BTSA (Tier II) for licensure

Innovative scenario-based interactions

NIH Genetic Conseling (Roter, et. al., 2009)*Duckor, Draney & Wilson (2009)*How we can capture classroom practice with respect to how teachers are enacting most-formative types of assessments. Today, I am going to specifically discuss how our project builds off work some of my colleagues and I have done at the Learning Research and Development Center with regard to coding classroom discourse.

*

Evidence suggests teachers have difficulty negotiating this type of discourse; thus, they have difficulty properly scaffolding student thinking and reasoning as lesson unfold in a series of temporal phases (e.g., Stein, Grover, and Henningsen, 1996; Tarr, Chavez, Reys, & Reys, 2006; Stigler & Hiebert, 2004). Two phases that are often discussed are the written task (tasks as they appear in the curricular materials) and the enacted task (tasks as worked on by students). Close examination of classroom practice reveals the cognitive demand of the written task often diminishes during enactment.

A classic example of this unintended reduction in cognitive demand comes from the 1999 TIMSS video study. Focusing on the US, 17% of the problems presented were classified as making connections - problems with high cognitive demand (Stigler & Hiebert, 2004). However, a second coding of the lessons that focused on the enactment of these tasks told a different story. The second coding of the lessons revealed US teachers implemented none of the cognitively demanding tasks in the way in which they were intended. Instead, US teachers transformed the tasks into procedural exercises or simply supplied the students with the answer to the task. The much-publicized QUASAR Project reported similar findings (Stein, Grover, & Henningsen, 1996).*

The The coding scheme was developed at the University of Pittsburgh, Learning Research and Development Center by Kevin Ashley, Richard Correnti, Moddy McKeown, Peg Smith, Mary Kay Stein, James Chisholm, and Jimmy Scherrer. *Although this example is being used to illustrate various codes, it is instructive of how a teacher can assess student understanding (Why do you think it is a rectangle?) and then scaffold the reasoning in an attempt to build a generalization (Are all shapes that have 4 sides rectangles?). *For example, in this excerpt, the teacher asks an open-ended question, collects two additional responses to that question, then moves to another Launch without doing anything with the responses to the original question. We would score these collects as 0. This is actually a common pattern exhibited by many teachers first trying to orchestrate whole-group discussion of an higher-level task. *In this little excerpt, used for illustrative purposes, we score the series of interactions a +4. We give the teacher 1 point for asking an open-ended question, another point for collecting an additional response to that question, and two points for opening up a position where the students have the opportunity to connect the responses. I want to restrict your attention to the collect code. In this example, because of its position, we assign it a value of 1. However, if we change the codes around this particular move, the value may change. *This slide re-focuses audience on where we are in the cycle of psychometric inquiry**Return to the three possible constructscontent knowledge, knowledge of student thinking, and knowledge of formative assessment. Item response theory models offer the possibility of exploring relationships between these three constructs. KSC: Subject content knowledge e.g. Biology; EcosystemsKST: Knowledge of student thinking e.g. prior knowledge, preconceptions, misconceptions, learning progressionsKFA: Knowledge of formative assessment including generic moves and strategies e.g. posing, pausing, bouncing, tagging, binning

Re-examining the **These can be modeled as three distinct domainswhere learning in one area does not affect learning in the other.*Or, the relationship between subject content knowledge, knowledge of student thinking, and knowledge of formative assessment can be modeled like thisthe constructs are still separate but now are staggered. *Or, the relationship can be even more complexwith different pathways where learning in one area (such as understanding student thinking) depends on the progression of learning in another area, such as subject content knowledge.

These different ways of modeling thinking depend on many thingswe need to assess each construct separately, we need to get a lot of data, and we need sophistica**How well do the student teachers understand the mathematical target they are assessing? Is that target appropriate--grade appropriate, or aligned with the lesson that was taught? Do the student teachers understand the purposes of the particular assessment? What information do they choose to collect from students on which they base their inferences (what evidence do they have about student learning)? Are those inferences warranted and accurate?**This research effort represents an incredible partnership among a broad group of stakeholders including a county office of education (SCCOE); research institutions (San Jose State and UC Berkeley); and school district teams.This work fills a research niche that has not been studied by linking progressions in student learning with progressions in teacher use of instructional practices like the use of formative assessment.There has been a narrowing of focus for this assessment work to most formative assessment and to the contextual use of questioning clusters by teachers that supports the improvement of students abilities to solve more complex real world problems.There has been a focus of the content to science content with an emphasis on student understanding and ability to use and apply ideas related to ecosystems.There is an emphasis on supporting formative assessment where teachers use questioning schema to elicit student thinking in the solution of real world and more complex problems.The work narrows the focus on instrumentation that uses both qualitative and quantitative scoring tools that measure teacher progression in the use of formative assessment.The work marries both qualitative and quantitative data within an IRT framework.Our next steps will be to plan and implement a pilot study in selected schools to better understand the four key components of this research initiative.**

*