Formative Assessment: A Method Formative Assessment: A Method to Close the Feedback Loopto Close the Feedback Loop

Eugenia Etkina, Graduate School of Education (GSE)

Rutgers University

Florida International University

April 8, 2005

Supported in part by NSF Grant DUE #0241078

Rutgers PER GroupRutgers PER Group

David Brookes (Physics, Student) Suzanne Brahmia (Physics, Staff) Eugenia Etkina (GSE, Professor) Mike Lawrence (High School Physics Teacher) Sahana Murthy (GSE, Post Doc) David Rosengrant (GSE, Student) Maria Ruibal (GSE, Student) Alan Van Heuvelen (Physics, Professor) Aaron Warren (Physics, Student)

http://paer.rutgers.edu/scientificabilities/http://paer.rutgers.edu/PT3

OutlineOutline

• Assessment• Formative vs summative• Three steps of formative assessment• Self-assessment• Scientific abilities and assessment rubrics• Examples of assessment tasks and student

performance

We all do assessmentWe all do assessment

Summative

Formative

Formative AssessmentFormative Assessment

• Enhances learning during learning.

• Gains are the largest reported for an educational intervention (Black and Wiliam, 1998).

Components of formative Components of formative assessmentassessment

• Teacher giving feedback to the students• The teacher and students taking action

to improve learning during learning• Self - assessment

Three essential stepsThree essential steps

Where are you now?

Where do you

need to go?How can you

get there?

Role of feedbackRole of feedback

Feedback

Grade

Written comments

Grade and comments

E

F

F

E

C

T

Negative

Positive

Zero

What are our goals?What are our goals?

• Conceptual understanding• Problem solving• Scientific abilities

Formative assessment projectFormative assessment project

• What are some scientific abilities?• Activities in which students develop abilities• Sub-abilities• Assessment of abilities through rubrics• Rubrics as guidelines and for self-assessment• Implementing activities with rubrics (3 steps of FA)• Using rubrics to score• Do students improve?

What are some scientific abilities?What are some scientific abilities?

• Ability to represent a process in multiple ways• Ability to design an experimental investigation (an

observational experiment; a testing experiment; an investigation to solve a problem)

• Ability to collect and analyze experimental data• Ability to construct and modify explanations• Ability to evaluate all of the above

All of them are content dependent!!!

ActivitiesActivities

• Multiple representation tasks (D. Rosengrant, A. Van Heuvelen, E. Etkina)

• Experimental design tasks (S. Murthy, M. Ruibal, E. Etkina)

• Anomalous data tasks (D. Brookes, M. Lawrence, E. Etkina)

• Video problems tasks (D. Brookes, E. Etkina)

• Evaluation tasks (A. Warren, A. Van Heuvelen)

http://paer.rutgers.edu/scientificabilities

http://paer.rutgers.edu/pt3

Activity: design an experiment to solve Activity: design an experiment to solve a problema problem

Experiment 1Design an experiment to determine the coefficient of static friction between your shoe and the carpet.

Experiment 2Design an experiment to determine the specific heat of the given unknown object.

Sub-abilitiesSub-abilities

Ability to design an experiment to solve a problemSome sub-abilities

• To suggest multiple experiments to accomplish desired goals.• To design a reliable experiment that solves the problem.• To identify assumptions made in the procedure.• To determine specifically the way assumptions might affect the

outcome.• To identify sources of experimental uncertainties. • To evaluate specifically how experimental uncertainties may affect

data.• To evaluate results by an independent method.

ExperimentsExperiments

Design two independent experiments to determine the coefficient of static friction between

your shoe and the carpet.

Include in your report for each experiment:a) an outline of the experimental procedure and a labeled sketch;b) a mathematical procedure to solve the problem; c) assumptions about the objects, interactions, and processes used in the math

procedure and how these assumptions could affect the outcome;d) sources of experimental uncertainty, how these could affect the result and

how you could minimize them.e) Perform the experiment and record your observations in an appropriate

format. What is the outcome of your experiment? f) Compare the two values you obtained. Explain the difference using

assumptions and uncertainties. g) Describe a real life situation in which you need to figure out things similar to

this experiment.

Design two independent experiments to determine the specific heat of the given unknown object.

Scientific abilities rubricsScientific abilities rubrics• Contain descriptors of different levels of development of scientific abilities.

Scale of 0-3. Very high inter-rate reliability.

• Can be used as guidelines for students, grading lab reports, self-assessment of lab reports by students, research on developing and assessing scientific abilities.

• Lab tasks contain guidelines which match rubrics.

SCORE ABILITY

0 1 2 3

To design a reliable experiment that solves the problem

The experiment does not solve the problem.

The experiment attempts to solve the problem but due to the nature of the design, the data will not lead to an accurate solution.

The experiment attempts to solve the problem but due to the nature of the design, there is a moderate chance that the data will not lead to an accurate solution.

The experiment solves the problem and has a high likelihood of producing data that will lead to an accurate solution

Developing rubricsDeveloping rubrics

• Identifying important sub-abilities• Writing descriptors (scale 0-3)• Finding clear wording• Scoring student work• Discussing the items with a disagreement• Revising wording

Name

(9 rows)

Able to Able to Able to Able to Able to

Suzanne 3 1 1 3 2

Sahana 3 1 3 2 2

Using rubricsUsing rubrics

SCORE

ABILITY0 1 2 3

To identify assumptions made in the chosen procedure

No attempt is made to identify any assumptions.

An attempt is made to identify assumptions, but most are missing, described vaguely, or incorrect.

Most assumptions are correctly identified

All assumptions are correctly identified.

Design two independent experiments to determine the specificheat of the given unknown object.

Student writing samples

… the block and water will reach equilibrium after 10 minutes … no heat goes in and out of the calorimeter…minimal heat is lost to the environment during transfer of the block … the temperature inside the beaker is homogenous SCORE: 3

… No heat exchange between system and surroundings.

SCORE: 1

Using rubricsUsing rubrics

Design two independent experiments to determine the specificheat of the given unknown object.

SCORE

ABILITY

0 1 2 3

To determine specifically the way in which assumptions might affect the results

No attempt is made to determine the effects of assumptions.

An attempt is made to determine the effects of some assumptions, but most are missing, incorrect or described vaguely

Effects of most assumptions are determined correctly, though a few contain minor errors, inconsistencies, or omissions.

The effects of all assumptions are correctly determined

… if the heat is lost from the block during transfer, the specific heat obtained in the experiment would be smaller than what it should be.

SCORE: 2

… the assumptions will affect the results, the actual value may be different from the experimental one.

SCORE: 1

ImplementationImplementation

• Two large enrollment introductory algebra-based classes -- 500 and 190 students, 30 TAs

• Interactive engagement - ISLE curriculum• Labs are integrated [L-r-l]• Lab sections of 25 students who work in groups of 3 or 4• TA Training, one hour per week: TAs perform and analyze

design tasks• Students perform and write a report in a 3-hour lab

Pilot Study (spring 2004)Pilot Study (spring 2004)

Ability 1: To design a reliable experiment that solves the

problem

0

5

10

15

20

25

INITIAL FINAL

Score 0 Score 1 Score 2 Score 3

Ability 2: To choose a productive mathematical procedure

0

5

10

15

20

25

INITIAL FINAL

Score 0 Score 1 Score 2 Score 3

Ability 3: To communicate details of the experiment completely

0

5

10

15

20

25

INITIAL FINALScore 0 Score 1 Score 2 Score 3

Ability 4: To evaluate the effects of experimental uncertainties

0

5

10

15

20

25

INITIAL FINAL2Score0 Score1 Score2 Score3

•Students do not use rubrics for self assessment

•35 students randomly selected from 4 sections

•Scored student

responses from

week 3 (initial) and

week 10 (final)

•Found significant

improvement

Student writing samplesStudent writing samples

Week 3 Week 10

New Study (Fall 2004) New Study (Fall 2004) A typical lab sessionA typical lab session

Students download lab tasks and rubrics from course webpage

Student report: brainstorming

Student report: self-assessment scores

Student report: outcomes, effect of

assumptions, uncertainties

Student report: data and analysis Groups perform experiment;

collect, record and analyze dataTA

Students perform self-assessment of their experiment and analysisTA

Groups brainstorm solutions for task, decide best designTA

FindingsFindings

Scores on Scientific Abilities -- Expt 1

0

0.5

1

1.5

2

2.5

3

Design expt to solve problemUse equipment effectively

Evaluate results by indept. ...Choose mathematical proce...Identify assumptions in proc...How assumptions affect resultIdentify sources of uncertaintyEvaluate effects of uncertainties

Minimize uncertainty

Record data in meaningful wayAnalyze data appropriately

STUDENTTEACHER

N=102

FindingsFindings

Scores on Scientific Abilities - -Expt. 2

0

0.5

1

1.5

2

2.5

3

Design expt to solve problemUse equipment effectivelyEvaluate results by indep..

Choose mathematical pro...Identify assumptions in p...How assumptions affect r...Identify sources of uncert...Evaluate effect of uncertai...

Minimize uncertainty

Record data in meaningfu...Analyze data appropriately

STUDENT

TEACHER

N=102

FindingsFindings

Experiment 1

0

0.5

1

1.5

2

2.5

3

Identifyassumptions in

procedure

Howassumptionsaffect result

Evaluateeffects of

uncertainties

Minimizeuncertainty

STUDENT TEACHER

Experiment 2

0

0.5

1

1.5

2

2.5

3

Identifyassumptions in

procedure

Howassumptionsaffect result

Evaluate effectof

uncertainties

Minimizeuncertainty

STUDENT TEACHER

Initial scores

0

0.5

1

1.5

2

2.5

3

Design reliableexperiment

Choose mathematicalprocedure

Evaluate effects ofuncertainties

Scores

00.5

11.5

22.5

3

Design expt to solve problemUse equipment effectively

Evaluate results by indept...Choose productive mathe...Identify assumptions in p...How assumptions affect re...Identify sources of uncert...Evaluate how uncertainties...

Minimize uncertainty

Record data in meaningfu...Analyze data appropriately

TA 1 TA 2

FindingsFindings

N=47

What did we learn?What did we learn?

• It is possible to implement and assess open-ended design tasks in large enrollment class.

• Assessment rubrics serve as goals for writing new design tasks.

• Preparation of TA matters.• Evaluating assumptions and uncertainties is very

difficult for the students. • Abilities are content-dependent.• Students’ self-assessment is closer to ours on the

abilities that they developed. • Students’ self-assessment is farther from ours on the

abilities that they did not develop.

Ability to represent phenomena in Ability to represent phenomena in multiple waysmultiple ways

Free-body diagrams: where do you need to go?An elevator is slowing down An elevator is slowing down on

on its way up: its way down:

y a

y

a

Earth, cable

F Earth on elevator

F cable on elevatory F cable on elevator

F Earth on elevatory

Rubric for self assessementRubric for self assessement

Scientific Ability

0 1 2 3

Ability to construct free-body diagrams

No FBD is drawn

FBD is constructed but contains major errors such as an incorrect relative length, wrong direction, extra incorrect force vectors, a missing force vector, or a force exerted on a different object.

FBD contains no errors in forces but lacks a key feature such as labels or forces are mislabeled or axes are missing or mis labeled.

The diagram contains no errors and each force is labeled, axes are present and labeled.

Free-body diagrams: How to get there?

Formative assessment task in Formative assessment task in lecturelecture

Draw a free-body diagram for

the ball in the air..

Where are you now?

2

4

1

3

Fball on person

Fperson on ball

FEarth on ball

Fperson on ball

FEarth on ball

FEarth on ball

Fperson on ball

Do students actually use FBDs?Do students actually use FBDs?

Scientific Ability 0 1 2 3

Free-BodyDiagram

Norepresentation isconstructed.

FBD is constructedbut contains majorerrors such as anincorrect relativelength, wrongdirection, extraincorrect forcevectors or a forcevector is missing.

FBD contains noerrors in forcesbut lacks a keyfeature such aslabels or forces aremislabeled or axesare absent ormislabeled.

The diagramcontains no errors,each force islabeled so that it isclearly understoodwhat each forcerepresents. Axesare labeled.

SCORE 2

Ability to test an explanationAbility to test an explanation

Ability to make a prediction based on a relationship/explanation

Ability to revise the explanation based on the results of the experiment

Where do you need to go?

How to get there? How to get there?

Scientific Ability

0 1 2 3

Is able to make a reasonable prediction based on a relationship or explanation.

No attempt to make a prediction is made.

A prediction is made but it doesn’t follow from the relationship or explanation being tested, or it ignores or contradicts some of the assumptions inherent in the relationship or explanation.

A prediction is made that follows from the relationship or explanation and incorporates the assumptions, but it contains minor errors, inconsistencies or omissions.

A correct prediction is made that follows from the relationship or explanation and incorporates the assumptions.

Ability to revise the explanation or Ability to revise the explanation or assumptions based on the results of the assumptions based on the results of the

experimentexperiment

•Make a prediction about an outcome of a particular experiment•Explain why you made the prediction.•Watch the experiment, record the outcome, compare to the prediction.•Revise the explanation or the assumptions if necessary.

How to get thereHow to get there

Is able to revise the explanation of a prediction, based on the results of an experiment.

No attempt is made to explain the outcome of the experiment, revise explanation or assumptions.

An attempt is made to explain the outcome and revise the previous explanation or assumptions but is (a) mostly incomplete and/or

(b) based on incorrect reasoning.

The revision of the previous explanation or assumptions is partially complete and correct, yet still lacking in some relevant details.

The revision of the explanation or assumptions is explained completely and correctly.

Ability to design an experiment to Ability to design an experiment to solve a problemsolve a problem

1. Video problems.

2. Design experiments with scaffolding.

3. Design experiments with no scaffolding (practicals)

The ENDThe END

Thank you!