Pinball Car Race Pilot Task - Assessment at SRI · Pinball Car Race Paper/Pencil Task. Pilot Study...

31
2014 Pinball Car Race Pilot Task Items, Rubrics, Student Work Samples, & Item Statistics SRI Education Division Assessment Team Daisy Rutstein Ron Fried Reina Fujii Geneva Haertel Terry Vendlinski Cindy Ziker Patricia Schank Amy Hafter Michelle Huynh Copyright © 2014 SRI International Performance-based Science Assessment Task Aligned to Next Generation Science Standards Evidence Centered Design Scenario-based

Transcript of Pinball Car Race Pilot Task - Assessment at SRI · Pinball Car Race Paper/Pencil Task. Pilot Study...

Page 1: Pinball Car Race Pilot Task - Assessment at SRI · Pinball Car Race Paper/Pencil Task. Pilot Study Results: Items, Scoring Rubrics, Calibrated Student Work Samples, and Item Statistics

2014 Pinball Car Race Pilot Task

Items, Rubrics, Student Work Samples, & Item Statistics

SRI Education Division Assessment Team • Daisy Rutstein • Ron Fried • Reina Fujii • Geneva Haertel • Terry Vendlinski • Cindy Ziker • Patricia Schank • Amy Hafter • Michelle Huynh

Copyright © 2014 SRI International

Performance-based Science Assessment Task Aligned to Next Generation Science Standards

Evidence Centered Design Scenario-based

Page 2: Pinball Car Race Pilot Task - Assessment at SRI · Pinball Car Race Paper/Pencil Task. Pilot Study Results: Items, Scoring Rubrics, Calibrated Student Work Samples, and Item Statistics

Contents

Background ................................................................................................................................................. 3

Orientation to Pinball Car Race (Not Scored) ............................................................................................ 4

Item 1 .......................................................................................................................................................... 5

Rubric for Item 1 ..................................................................................................................................... 6

Item 1: Percentage of students at each score point (N=156 students) .................................................... 6

Student Sample Responses for Item 1 .................................................................................................... 6

2-Point Sample Responses .................................................................................................................. 6

1-Point Sample Responses .................................................................................................................. 7

0-Point Sample Responses .................................................................................................................. 7

Item 2 (Not Scored) .................................................................................................................................... 8

Item 2: Percentage of students who selected each option (N=156 students) .......................................... 9

Item 3 ........................................................................................................................................................ 10

Rubric for Item 3 ................................................................................................................................... 11

Item 3: Percentage of students at each score point (N=156 students) .................................................. 11

Student Sample Responses for Item 3 .................................................................................................. 11

2-Point Sample Responses ................................................................................................................ 11

1-Point Sample Responses ................................................................................................................ 12

0-Point Sample Responses ................................................................................................................ 12

Item 4 ........................................................................................................................................................ 13

Rubric for Item 4 ................................................................................................................................... 14

Item 4: Percentage of students at each score point (N=156 students) .................................................. 14

Student Sample Responses for Item 4 .................................................................................................. 14

2-Point Sample Responses ................................................................................................................ 14

1-Point Sample Responses ................................................................................................................ 15

0-Point Sample Responses ................................................................................................................ 16

Item 5 ........................................................................................................................................................ 17

Rubric for Item 5 ................................................................................................................................... 18

Item 5: Percentage of students at each score point (N=156 students) .................................................. 18

Student Sample Responses for Item 5 .................................................................................................. 19

Copyright © 2014 SRI International Page 1

Page 3: Pinball Car Race Pilot Task - Assessment at SRI · Pinball Car Race Paper/Pencil Task. Pilot Study Results: Items, Scoring Rubrics, Calibrated Student Work Samples, and Item Statistics

2-Point Sample Responses ................................................................................................................ 19

1-Point Sample Responses ................................................................................................................ 19

0-Point Sample Responses ................................................................................................................ 20

Item 6 (Parts a and b are scored together) ................................................................................................. 21

Rubric for Items 6a and 6b.................................................................................................................... 22

Item 6a and 6b: Percentage of students at each score point (N=156 students) ..................................... 22

Student Sample Responses for Items 6a and 6b ................................................................................... 23

2-Point Sample Responses ................................................................................................................ 23

1-Point Sample Responses ................................................................................................................ 24

0-Point Sample Responses ................................................................................................................ 24

Item 7 (Parts a and b are scored together) ................................................................................................. 25

Rubric for Items 7a and 7b.................................................................................................................... 26

Items 7a and 7b: Percent of students at each score point (N=156 students) ......................................... 26

Student Sample Responses for Items 7a and 7b ................................................................................... 27

3-Point Sample Responses ................................................................................................................ 27

2-Point Sample Responses ................................................................................................................ 28

1-Point Sample Responses ................................................................................................................ 28

0-Point Sample Responses ................................................................................................................ 29

Item 8 (Dropped from assessment) ........................................................................................................... 30

Task Complete .......................................................................................................................................... 30

Copyright © 2014 SRI International Page 2

Page 4: Pinball Car Race Pilot Task - Assessment at SRI · Pinball Car Race Paper/Pencil Task. Pilot Study Results: Items, Scoring Rubrics, Calibrated Student Work Samples, and Item Statistics

Pinball Car Race Paper/Pencil Task

Pilot Study Results: Items, Scoring Rubrics, Calibrated Student Work Samples, and Item Statistics

Background The Pinball Car Race is a middle school science assessment task designed to test a student’s knowledge of science content and science practices, including scientific reasoning and the process skills involved in designing and interpreting results of an experimental investigation. An Evidence Centered Design (ECD) approach was used to develop this task, which focuses on knowledge of forms of energy in the physical sciences, specifically, potential and kinetic energy. The student selects a hypothesis and manipulates the variable of interest, while controlling the other variables, to test the hypothesis, interpret the results presented and draw conclusions. For formative purposes, scores on individual parts of the task may be used to provide diagnostic evidence of a student’s ability to draw conclusions about an experiment. The Pinball task provides a student with two attempts to design an experiment. Scores may be calculated to diagnose the level of skill a student demonstrates on each of the two attempts, providing an opportunity to determine whether student performance changes.

This document contains information from the pilot of this assessment task. The task was originally piloted as a paper/pencil version. A rubric for the paper/pencil version is provided below as well as item statistics from the pilot study of the paper/pencil version conducted in December of 2013. Samples of calibrated student work from the paper/pencil pilot are included.

Based on the results of this paper/pencil pilot study, the task was revised and additional items were added. The revised task appears online (see URL for Pinball Race Car task: http://assessment.sri.com/performancetask/pinball/start.php). Rubrics for this revised version appear in the “Pinball Car Race Online Task: Item Screenshots & Rubrics” document.

Copyright © 2014 SRI International Page 3

Page 5: Pinball Car Race Pilot Task - Assessment at SRI · Pinball Car Race Paper/Pencil Task. Pilot Study Results: Items, Scoring Rubrics, Calibrated Student Work Samples, and Item Statistics

Orientation to Pinball Car Race (Not Scored) This page introduces the idea of a Pinball Car Race. The student is shown several representations of the Pinball Car Race.

Copyright © 2014 SRI International Page 4

Page 6: Pinball Car Race Pilot Task - Assessment at SRI · Pinball Car Race Paper/Pencil Task. Pilot Study Results: Items, Scoring Rubrics, Calibrated Student Work Samples, and Item Statistics

Item 1 On this page, the student is shown images of the spring hitting the car, causing the car to move. Students are asked to describe the transfer of energy from the spring to the car.

Copyright © 2014 SRI International Page 5

Page 7: Pinball Car Race Pilot Task - Assessment at SRI · Pinball Car Race Paper/Pencil Task. Pilot Study Results: Items, Scoring Rubrics, Calibrated Student Work Samples, and Item Statistics

Rubric for Item 1 The student should indicate that when the spring hits the car the potential energy becomes kinetic energy. When the spring comes to a rest, it no longer has potential or kinetic energy. The student may also indicate that the car loses kinetic energy as it slows down.

2 points Student indicates that energy is transferred from the spring to the car when there is contact between the two AND that the moving car has kinetic energy.

1 point Student indicates that energy is transferred from the spring to the car when there is contact between the two OR that the moving car has kinetic energy.

0 points Student’s response NEITHER indicates that energy is transferred from the spring to the car when there is contact between the two NOR indicates that the moving car has kinetic energy.

Item 1: Percentage of students at each score point (N=156 students) Missing 0 1 2 16.0% 21.2% 39.7% 23.1%

Student Sample Responses for Item 1

2-Point Sample Responses For sample responses 1 and 2, the student indicates that energy is transferred from the spring to the car when there is contact between the two and that the moving car has kinetic energy.

Sample Response 1:

Sample Response 2:

The energy is transfered from the spring to the car because when the spring is pulled back, the coils are being pulled in a way they normally aren't. This potential energy is then transfered to the car and becomes kinetic energy

When the spring hits the car, it transfers the potential energy into kinetic energy, causing the car toward the direction it was hit.

Copyright © 2014 SRI International Page 6

Page 8: Pinball Car Race Pilot Task - Assessment at SRI · Pinball Car Race Paper/Pencil Task. Pilot Study Results: Items, Scoring Rubrics, Calibrated Student Work Samples, and Item Statistics

1-Point Sample Responses For sample response 3, the student indicates that energy is transferred from the spring to the car when there is contact between the two but does not indicate the energy of the car is kinetic energy. In sample response 4, the student indicates that the moving car has kinetic energy but does not discuss the transfer of energy from the spring to the car.

Sample Response 3:

Sample Response 4:

0-Point Sample Responses For sample responses 5 and 6, the student’s response neither indicates that energy is transferred from the spring to the car when there is contact between the two nor that the moving car has kinetic energy.

Sample Response 5:

Sample Response 6:

When the spring is released it pops out pushing the car transfering energy for the car to move.

When the spring is released, then the car has kinetic energy

The spring has built up pressure

The energy pushes the car forward and I don't know what type of energy it is

Copyright © 2014 SRI International Page 7

Page 9: Pinball Car Race Pilot Task - Assessment at SRI · Pinball Car Race Paper/Pencil Task. Pilot Study Results: Items, Scoring Rubrics, Calibrated Student Work Samples, and Item Statistics

Item 2 (Not Scored) Here, a student picks one of the characteristics of the spring to use in her or his hypothesis. The student explains why the characteristic was selected in terms of how it would affect the potential energy of the spring. The student is not scored on item 2, as any hypothesis is acceptable.

Copyright © 2014 SRI International Page 8

Page 10: Pinball Car Race Pilot Task - Assessment at SRI · Pinball Car Race Paper/Pencil Task. Pilot Study Results: Items, Scoring Rubrics, Calibrated Student Work Samples, and Item Statistics

Item 2: Percentage of students who selected each option (N=156 students)

Variable Selected Percentage of students who selected each option More coils 45.5% Less coils 10.3%

Thicker wire 21.8% Thinner wire 3.8%

Multiple* 18.6%

*In the paper/pencil pilot, students were able to select multiple options. This was changed in later versions of the item as we limited the number of variables the student could test to 1.

Copyright © 2014 SRI International Page 9

Page 11: Pinball Car Race Pilot Task - Assessment at SRI · Pinball Car Race Paper/Pencil Task. Pilot Study Results: Items, Scoring Rubrics, Calibrated Student Work Samples, and Item Statistics

Item 3 Once a student has picked a hypothesis, he or she is asked to explain why that hypothesis was chosen. The student is allowed to go back and change his or her hypothesis. Information from item 2 will be used in scoring item 3.

Copyright © 2014 SRI International Page 10

Page 12: Pinball Car Race Pilot Task - Assessment at SRI · Pinball Car Race Paper/Pencil Task. Pilot Study Results: Items, Scoring Rubrics, Calibrated Student Work Samples, and Item Statistics

Rubric for Item 3 Student should state how the spring characteristic that was selected would increase or decrease the potential energy of the spring. The student response should match the student’s hypothesis. For example, if the student hypothesizes that the car would travel farther with springs that have more coils, then the student should indicate that the greater the number of coils, the more potential energy the spring could store. Note that this item assesses a student’s understanding of how the hypothesis relates to the experiment he or she is conducting. It is not focused on the specific science content of the hypothesis.

2 points The student’s explanation matches his or her hypothesis AND indicates that he or she understands the cause and effect relationship between the car and the spring. This explanation must go beyond just restating the hypothesis.

1 point The student’s explanation matches the hypothesis but does not go beyond just restating the hypothesis.

0 points The student’s explanation does not match the hypothesis.

Item 3: Percentage of students at each score point (N=156 students) Missing 0 1 2

1.3% 5.1% 22.4% 71.2%

Student Sample Responses for Item 3

2-Point Sample Responses For sample responses 1 and 2, both students selected “More coils” as their hypothesis in item 2. The student explanations match his or her hypothesis and indicate that he or she understands the cause and effect relationship between the car and the spring.

Sample Response 1:

Sample Response 2:

the more coils on the spring the more energy forced onto the car

When the coils unravel, it will have more energy if there were more coils and it will push the car further.

Copyright © 2014 SRI International Page 11

Page 13: Pinball Car Race Pilot Task - Assessment at SRI · Pinball Car Race Paper/Pencil Task. Pilot Study Results: Items, Scoring Rubrics, Calibrated Student Work Samples, and Item Statistics

1-Point Sample Responses For sample responses 3 and 4, both students selected “More coils” as their hypothesis in item 2. The students’ explanations match their hypotheses in item 2, but do not go beyond just restating their hypotheses.

Sample Response 3:

Sample Response 4:

0-Point Sample Responses For sample response 5, the student chose “Thicker Wire” as his or her hypothesis in item 2. The student’s explanation does not match the hypothesis. There is no statement about how the thicker wire will contribute to the car going farther. For sample response 6, the student chose “More coils” as his or her hypothesis in item 2. The student’s explanation does not match the hypothesis and does not address the distance the car travels but rather makes a statement about force.

Sample Response 5:

Sample Response 6:

I think this hypothesis is true because the more coils, the more distance it gets to push the car/bounce back

because more coils will allow the car to go further distances

because more energy is consumed in the spring

Because the coils would make it longer but have the same amount of force.

Copyright © 2014 SRI International Page 12

Page 14: Pinball Car Race Pilot Task - Assessment at SRI · Pinball Car Race Paper/Pencil Task. Pilot Study Results: Items, Scoring Rubrics, Calibrated Student Work Samples, and Item Statistics

Item 4 In item 4, the student is asked to select settings for each of the springs. These settings allow the student to test his or her hypothesis.

Copyright © 2014 SRI International Page 13

Page 15: Pinball Car Race Pilot Task - Assessment at SRI · Pinball Car Race Paper/Pencil Task. Pilot Study Results: Items, Scoring Rubrics, Calibrated Student Work Samples, and Item Statistics

Rubric for Item 4 Scores will reflect whether the selected settings are appropriate to the hypothesis. A student should be using different settings for the independent variable and should keep the control variable settings the same in all three trials. The independent variable and control variable should match the student’s hypothesis. The independent variable is the variable that the student included in the hypothesis; the control variable is the variable the student will need to hold constant.

2 points The student varies the appropriate variable (i.e., the variable that matches the student’s hypothesis) AND keeps the other variable constant.

1 point The student varies both variables OR the student varies the incorrect variable (i.e., the variable that does not match the student hypothesis) but keeps the other variable constant.

0 points The student keeps both variables constant.

Item 4: Percentage of students at each score point (N=156 students) Missing 0 1 2

4.4% 2.6% 35.9% 57.1%

Student Sample Responses for Item 4

2-Point Sample Responses For sample response 1, the student chose “More coils” as his or her hypothesis in item 2, which means that the characteristic of the “Number of Coils” is the appropriate variable to vary. The student kept “Thickness of the Wire” constant.

Sample Response 1:

Copyright © 2014 SRI International Page 14

Page 16: Pinball Car Race Pilot Task - Assessment at SRI · Pinball Car Race Paper/Pencil Task. Pilot Study Results: Items, Scoring Rubrics, Calibrated Student Work Samples, and Item Statistics

1-Point Sample Responses For sample response 2, the student chose “More coils” as his or her hypothesis in item 2, which means that the characteristic of the “Number of Coils” is the appropriate variable to vary. However, the student varied “Thickness of the Wire” instead, which is incorrect.

Sample Response 2:

In sample response 3, the student varied both variables instead of keeping one of the variables constant.

Sample Response 3:

Copyright © 2014 SRI International Page 15

Page 17: Pinball Car Race Pilot Task - Assessment at SRI · Pinball Car Race Paper/Pencil Task. Pilot Study Results: Items, Scoring Rubrics, Calibrated Student Work Samples, and Item Statistics

0-Point Sample Responses For sample response 4, the student kept both variables constant, which is incorrect.

Sample Response 4:

Copyright © 2014 SRI International Page 16

Page 18: Pinball Car Race Pilot Task - Assessment at SRI · Pinball Car Race Paper/Pencil Task. Pilot Study Results: Items, Scoring Rubrics, Calibrated Student Work Samples, and Item Statistics

Item 5 In this item, the student explains the reason for choosing the spring settings. The student’s hypothesis selected in item 2 is not considered when scoring this item. The student’s explanation is an indication of whether he or she understands the concept of a control variable and an independent variable.

Copyright © 2014 SRI International Page 17

Page 19: Pinball Car Race Pilot Task - Assessment at SRI · Pinball Car Race Paper/Pencil Task. Pilot Study Results: Items, Scoring Rubrics, Calibrated Student Work Samples, and Item Statistics

Rubric for Item 5 A student response should indicate why the use of different settings for the independent variable allows the student to determine what would happen under different conditions. The student response should also indicate the need to keep the control variable constant in all three cases to ensure that any differences in the hypothesized outcome are not due to variation in the control variable. The student response must indicate that the student has knowledge of independent and control variables.

2 points

A student’s response indicates that the use of different settings for the independent variable allows the student to observe what would happen under different conditions, AND a student’s response indicates that keeping the control variable the same for all three springs isolates the effect of the independent variable. Explanations must go beyond just restating the settings.

1 point

A student’s response indicates that the use of different settings for the independent variable allows the student to observe what would happen under different conditions OR a student’s response indicates that keeping the control variable the same for all three springs isolates the effect of the independent variable. Explanations must go beyond just restating the settings.

0 points

A student’s response does not indicate that the use of different settings for the independent variable allows the student to observe what would happen under different conditions AND a student’s response does not indicate that keeping the control variable the same for all three springs isolates the effect of the independent variable.

Item 5: Percentage of students at each score point (N=156 students) Missing 0 1 2

3.2% 37.8% 22.4% 36.5%

Copyright © 2014 SRI International Page 18

Page 20: Pinball Car Race Pilot Task - Assessment at SRI · Pinball Car Race Paper/Pencil Task. Pilot Study Results: Items, Scoring Rubrics, Calibrated Student Work Samples, and Item Statistics

Student Sample Responses for Item 5

2-Point Sample Responses For sample responses 1 and 2, the student responses indicate that the use of different settings for the independent variable (the number of coils) allows the student to observe what would happen under different conditions. The responses also indicate that the student needs to keep the control variable (the thickness of the wire) the same.

Sample Response 1:

Sample Response 2:

1-Point Sample Responses For sample response 3, the student’s response only indicates that the independent variable (the number of coils) needs to be varied, but does not address the concept of a control variable. For sample response 4, the student indicates that he or she needs to keep a variable constant, but his or her response does not address the need to vary one setting in order to test the independent variable.

Sample Response 3:

Sample Response 4:

my hypothesis is testing the number of coils so I need different number of coils. I do not test the thickness of the wire so I keep that the same

I think that if I keep the wire thickness the same and only change the amount of coils it will show what amount of coils is best

because if I try with different number of coils, I can observe if the car will travel faster with very few or some coils, or like I said with many coils

my hypothesis was that if there were thicker wires it will have more power. I therefore kept the number of coils the same

Copyright © 2014 SRI International Page 19

Page 21: Pinball Car Race Pilot Task - Assessment at SRI · Pinball Car Race Paper/Pencil Task. Pilot Study Results: Items, Scoring Rubrics, Calibrated Student Work Samples, and Item Statistics

0-Point Sample Responses For sample response 5, the student indicates that he or she used different types of wire but does not indicate which characteristic was changed and which characteristic was held constant. For sample response 6, the student makes a prediction, rather than provides an explanation of why the selected settings were appropriate.

Sample Response 5:

Sample Response 6:

these settings will help me test my hypothesis because there are several types of wire

spring 1 is the one I think will go faster vs. 2 & 3

Copyright © 2014 SRI International Page 20

Page 22: Pinball Car Race Pilot Task - Assessment at SRI · Pinball Car Race Paper/Pencil Task. Pilot Study Results: Items, Scoring Rubrics, Calibrated Student Work Samples, and Item Statistics

Item 6 (Parts a and b are scored together) In items 6a and 6b, the student was given the hypothesis “the more coils, the farther the car will move.” The items require the student to determine if the data displayed “support,” “contradict,” or “do not provide enough information about his or her hypothesis.” In the data display, both the thickness of the wire and the number of coils were varied. The student should recognize that the results do not provide enough information about the hypothesis because each spring had different settings for both of the variables. This means that the effects could not be attributed to one variable or the other. Items 6a and 6b were scored together and the results of that scoring are presented below.

Copyright © 2014 SRI International Page 21

Page 23: Pinball Car Race Pilot Task - Assessment at SRI · Pinball Car Race Paper/Pencil Task. Pilot Study Results: Items, Scoring Rubrics, Calibrated Student Work Samples, and Item Statistics

Rubric for Items 6a and 6b A student’s answers should indicate a correct interpretation of the results of the experiment. In this case, the results do not provide information about the hypothesis. Since both variables were varied, it cannot be determined if the effect of the experiment is due to the number of coils or the thickness of the wire.

2 points Student correctly selects “do not provide information” AND indicates that you can’t tell which variable had an effect on the car/energy.

1 point

Student selects “does not provide information” AND has an incorrect explanation or no explanation. OR Student selects “contradict” AND indicates that the spring with few coils (Spring 1) makes the car go farther.

0 point

Student incorrectly selects “support” or “contradict” AND provides an explanation that inappropriately relates the results to the hypothesis. OR Student selects multiple options in 6a (i.e. “support” AND “contradict”) regardless of their explanation.

Percentage of students who selected each option in Item 6a (N=156 students)

Missing Supports Contradicts Does not provide information Multiple

3.2% 5.8% 41.0% 47.4% 2.6%

Item 6a and 6b: Percentage of students at each score point (N=156 students) Missing 0 1 2

3.9% 17.9% 39.1% 39.1%

Copyright © 2014 SRI International Page 22

Page 24: Pinball Car Race Pilot Task - Assessment at SRI · Pinball Car Race Paper/Pencil Task. Pilot Study Results: Items, Scoring Rubrics, Calibrated Student Work Samples, and Item Statistics

Student Sample Responses for Items 6a and 6b

2-Point Sample Responses For sample responses 1 and 2, the student correctly selects that the results do not provide information about the hypothesis. The student includes an explanation that it wasn’t possible to tell which variable has an effect on the distance the car moved.

Sample Response 1:

Sample Response 2:

These results do not provide information about his hypothesis

these do not provide information because he should have kept the wire thickness the same to test amount of coils

These results do not provide information about his hypothesis

Two variables are changed - the thickness of the wire *and* the number of coils. So, I can not find info. About how the number of coils affects how far the car goes. The thickness of the wire might also be affecting it.

6a.

6b.

6a.

6b.

Copyright © 2014 SRI International Page 23

Page 25: Pinball Car Race Pilot Task - Assessment at SRI · Pinball Car Race Paper/Pencil Task. Pilot Study Results: Items, Scoring Rubrics, Calibrated Student Work Samples, and Item Statistics

1-Point Sample Responses For sample response 3, the student response indicates that “these results contradict his hypothesis” rather than “these results do not support his hypothesis.” However, the explanation demonstrates that the student has some understanding of how the data are related to the hypothesis. For sample response 4, the student correctly indicates that the results do not provide information about the hypothesis but does not explain what is wrong with the choice of coils.

Sample Response 3:

Sample Response 4:

0-Point Sample Responses For sample response 5, the student indicates that the data support the hypothesis and his or her explanation does not discuss the relationship between the data and the hypothesis. For sample response 6, the student says that the results contradict the hypothesis but the response does not explain why the results contradict the hypothesis.

Sample Response 5:

Sample Response 6:

These results contradict his hypothesis

the less coils was the ones that went farther

These results do not provide information about his hypothesis

I chose the wrong coils

These results support his hypothesis

because it show now what type of coils affect the distance

These results contradict his hypothesis

The classmate's hypothesis was wrong, his/hers hypothesis didn't show support his/hers results.

6a.

6b.

6b.

6a.

6b.

6a.

6b.

6a.

Copyright © 2014 SRI International Page 24

Page 26: Pinball Car Race Pilot Task - Assessment at SRI · Pinball Car Race Paper/Pencil Task. Pilot Study Results: Items, Scoring Rubrics, Calibrated Student Work Samples, and Item Statistics

Item 7 (Parts a and b are scored together) On this page, students are given a second opportunity to reflect on the experiment. A student’s answer should correctly interpret the results of the experiment. In this case, the results support the hypothesis because as the wire becomes thicker, the car moved farther. All other variables remain the same so the cause must be due to the thickness of the wire.

Copyright © 2014 SRI International Page 25

Page 27: Pinball Car Race Pilot Task - Assessment at SRI · Pinball Car Race Paper/Pencil Task. Pilot Study Results: Items, Scoring Rubrics, Calibrated Student Work Samples, and Item Statistics

Rubric for Items 7a and 7b A student’s answers should correctly interpret the results of the experiment. In this case, the results support the hypothesis. As the wire got thicker the car moved farther. All other variables remained the same. Therefore, the cause of the car moving farthest must be due to the thickness of the wire.

3 points The student work shows: (1) student correctly selects “support her hypothesis”; (2) student indicates that the data shows that the thicker the wire the farther the car traveled; AND (3) student indicates that this cause could not be due to other variables.

2 points The student work shows 2 of the 3: (1) Student correctly selects “support her hypothesis”; (2) student indicates that the data shows that the thicker the wire the farther the car traveled; (3) student indicates that this cause could not be due to other variables.

1 point The student work shows 1 of the 3: (1) Student correctly selects “support her hypothesis”; (2) student indicates that the data shows that the thicker the wire the farther the car traveled; (3) student indicates that this cause could not be due to other variables.

0 points

Student incorrectly selects “contradict” OR “do not provide information” AND does not provide an appropriate explanation that relates the results to the hypothesis. OR Student selects multiple options in 7a (i.e. “support” AND “contradict”) regardless of their explanation.

Percent of students who selected each option in Item 7a (N=156 students)

Missing Supports Contradicts Does not provide information Multiple

2.6% 91.4% 0.7% 4.0% 1.3%

Items 7a and 7b: Percent of students at each score point (N=156 students) Missing 0 1 2 3

4.5% 7.7% 15.4% 59.0% 13.4%

Copyright © 2014 SRI International Page 26

Page 28: Pinball Car Race Pilot Task - Assessment at SRI · Pinball Car Race Paper/Pencil Task. Pilot Study Results: Items, Scoring Rubrics, Calibrated Student Work Samples, and Item Statistics

Student Sample Responses for Items 7a and 7b

3-Point Sample Responses For sample responses 1 and 2, the student correctly chooses the results “support her hypothesis” and explains that only the thickness of the wire changed, and the thickest spring made the car go farther. The student also indicates that this cause could not be due to other variables.

Sample Response 1:

Sample Response 2:

These results support her hypothesis

her hypothesis was that the thicker the wire the farther the car would travel. She kept the number of coils constant but changed the thickness of the wire in each spring. The spring with the thickest wire propelled the car the farthest

These results support her hypothesis

these springs only change once variable and that is the variable that is being tested. The average distance increases the thicker wire is, which was her hypothesis

7a.

7b.

7a.

7b.

Copyright © 2014 SRI International Page 27

Page 29: Pinball Car Race Pilot Task - Assessment at SRI · Pinball Car Race Paper/Pencil Task. Pilot Study Results: Items, Scoring Rubrics, Calibrated Student Work Samples, and Item Statistics

2-Point Sample Responses For sample responses 3 and 4, the student response indicates that the results “support her hypothesis.” In sample response 3, the student indicates that only one variable changed but does not describe how the data relates to the hypothesis. In sample response 4, the student explains how the data relates to the hypothesis but does not indicate that since one variable was kept constant, any change must be due to the thickness of the wire.

Sample Response 3:

Sample Response 4:

1-Point Sample Responses For sample responses 5 and 6, the students correctly choose that the results “support her hypothesis,” but neither explanation correctly relates the data to the hypothesis.

Sample Response 5:

Sample Response 6:

These results support her hypothesis

this supports her hypothesis because she kept the amount of coils the same and only changed the wire thickness

These results support her hypothesis

the thicker the wire the further it moved

These results support her hypothesis

the data supports her hypothesis

These results support her hypothesis

less coils and thin wire do better than same coils and thin wires.

7a.

7b.

7b.

7a.

7b.

7a.

7b.

7a.

Copyright © 2014 SRI International Page 28

Page 30: Pinball Car Race Pilot Task - Assessment at SRI · Pinball Car Race Paper/Pencil Task. Pilot Study Results: Items, Scoring Rubrics, Calibrated Student Work Samples, and Item Statistics

0-Point Sample Responses For sample response 7, the student states that the data both “support” and “contradict her hypothesis.” The student’s explanation does not indicate how the results relate to the hypothesis. For sample response 8, the student incorrectly selects that the results “contradict her hypothesis” and indicates that the hypothesis is wrong without explaining why.

Sample Response 7:

Sample Response 8:

These results support her hypothesis; These results contradict her hypothesis

The thick wire supports it, but the same coils do not.

These results contradict her hypothesis

my hypothesis is wrong

7b.

7a.

7b.

7a.

Copyright © 2014 SRI International Page 29

Page 31: Pinball Car Race Pilot Task - Assessment at SRI · Pinball Car Race Paper/Pencil Task. Pilot Study Results: Items, Scoring Rubrics, Calibrated Student Work Samples, and Item Statistics

Item 8 (Dropped from assessment) Item was dropped from the pilot analysis and revised in later iterations for the online version due to the need for refinements.

Task Complete

Copyright © 2014 SRI International Page 30