Technology-Facilitated Formative Assessment in Physical Science Connected Classrooms: Case Studies

Karen E. Irving Vehbi A. Sanalan Melissa L. Shirley

The Ohio State University

ASTE International Conference, St. Louis, MO January 10-13, 2008

The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R305K050045 to The Ohio State University.  The opinions expressed are those of the authors and do not represent views of the U.S. Department of Education.

01/11/2008

2

Project team• Principal Investigators

– Douglas T. Owens, OSU

– Stephen L. Pape, UF– Karen E. Irving, OSU– Louis Abrahamson,

BEF– Frank Demana, OSU

• Post doctoral researcher– Vehbi Sanalan, OSU &

Erzincan University, Turkey

• Consultants– Christy Kim Boscardin, UCLA– Joan Herman, UCLA– Jeremy Roschelle, SRI

• Project Coordinator– Mike Kositzke, OSU

• GRAs– Sedat Ucar; Sukru Kaya;

Melissa Shirley; Clare Bell; Ugur Baslanti; Sharilyn Owens; Hyesook Shin; Gonul Sakiz

01/11/2008

3

The potential of the connected classroom includes…• Multiple interconnected representations • Conceptual development through activity-

based learning experiences• Immediate anonymous feedback• Public displays of class knowledge• Technology-facilitated formative

assessment• Improved self-regulated learning

01/11/2008

4

CCMS project overview• Interdisciplinary professional development

and research project

• Algebra I and Physical Science

• First year algebra data

• Classroom connectivity technology

• Summer Institute – training

• T3 conference follow-up (2 years)

01/11/2008

5

Purpose & research questions• Purpose

• To report preliminary findings on science teacher experiences in connected classrooms

• Research Questions• How do physical science teachers implement

connected classroom technology in their teaching?

• Does connected classroom technology facilitate formative assessment?

01/11/2008

6

Technology integration: Social theory & practicality• Archer’s social theory

– Connected classroom as a cultural change• Morphostasis/ morphogenesis

• Doyle & Ponder’s practicality concept– Instrumentality

• Does the innovation fit the context of school?– Congruence

• Is the innovation compatible with teacher practice?– Costs/Benefits

• Is it worth the effort?

01/11/2008

7

Pedagogy of connected classrooms: Formative assessment

• The process used by teachers and students to recognize and respond to students’ learning in order to enhance that learning, during the learning. (Bell& Cowie, 2001)

• The gathering of assessment data & teacher use of assessment information to modify their work to improve their teaching effectiveness. (Black, 1995; Black & Wiliam, 1998).

01/11/2008

8

Formative assessment

• Formative assessment correlates with increased student achievement

– Analyzed over 40 studies on formative assessment

– All of the studies reported significant and often substantial achievement gains with positive effect sizes ranging from 0.4 to 0.7

(Black, Harrison, Lee, Marshall, & Wiliam, 2003)

01/11/2008

9

Difficulties implementing formative assessment

• Studies show formative assessment use by classroom teachers is one of the weakest aspects of teacher practice.

(Daws & Singh, 1996; Assessment Reform Group, 1999)

01/11/2008

10

Participants

• First physical science cohort: 9 teachers

• Purposively selected 3 teachers for preliminary analysis

• Criteria for selection– Success in implementation– Diversity of classroom situations

01/11/2008

11

School demographicsTeacher* State # students School Diversity % low SES

Ms. D TX 856Grades 7-8

57% Hispanic35% Black 5% White3% Asian/Pac Islander<1% Native American

82%

Ms. C OH 662Grades 6-8

51% White36 % Black11% Multiracial 2% Unspecified

72%

Ms. S OH 897 Grades 9-12

98% White 8%

*pseudonyms www.Greatschools.net

01/11/2008

12

Teacher preparation

*Pseudonyms

Teacher* UG degree Year Graduated

Graduate degree

Years Teaching

Ms. D Animal Science

2000 None 5

Ms. C Elementary Education

1987 Master’s Curriculum & Instruction

18

Ms. S Secondary Education

2003 None 3

01/11/2008

13

Students

*pseudonyms

Teacher* Class Student Characteristics

Student gender

Ms. D 7th grade science

Regular classes

M

Ms. C 8th grade science

Lowest achieving in the 8th grade; many inclusion & special ed

M & F

Ms. S 9th grade physical science

Regular classes

M & F

01/11/2008

14

Participants - Tech skills & support

*pseudonyms

Teacher* Tech Skills Tech support Classroom technology

Ms. D Experienced computer user

T3 mentor offsite

Only teacher in school using Navigator

Navigator

SmartBoard

Projector

Ms. C Technology noviceNo calculator skills initially

T3 mentors in bldg

Math teachers using Navigator

Navigator

SmartBoard

Projector

Ms. S Prior use of Probe ware; prior use of graphing calculator

T3 mentors in bldg

Math teachers & 1 science teacher using Navigator

Navigator

SmartBoard

Projector

01/11/2008

15

Data sources

• Telephone interviews (audio & notes) – Autumn – Spring

• Classroom observations (video & notes)– 2 days– 2 class periods

• Post observation teacher interviews• Student focus groups• Surveys and measures

01/11/2008

16

Coding – Practicality

• Instrumentality – context– Students– Technology support system– Social support

• Congruence– Successes/Challenges– Perceived ease of adoption

• Cost/Benefits– Why adopt? Why not?

*(Doyle & Ponder, 1977)

01/11/2008

17

Coding: Formative assessment typology

• Communicate task criteria– What has to be done– Quality criteria

• Gather data– Questions– Observes

• Interpret data– Critiques

• Change instructional practice– Suggestions for improved practice– Supplies information/ corrects/makes counter-suggestion

(Torrance and Pryor, 2001)

01/11/2008

18

Instrumentality - context

• Differences– Students– Schools – SES– Curriculum topics– Technology skills– Teacher preparation – Teaching experience

• Similarities– Mentoring– Social support– Hardware support– Risk takers– Persistence

01/11/2008

19

Congruence – inquiry teaching• Teachers found connected classroom

technology congruent with their prior teaching practices– My curriculum is to use the FOSS* kits. [There are] a

lot of hands on labs with those. So, basically, every time we do anything with data and graphing, we just automatically implement it through the Navigator™.

(Ms. C 3050.TIAu06.P23)

*Full Option Science System

01/11/2008

20

Congruence - assessment

I am on the data team for my district ….We really noticed …that kids are not analyzing the graphs….We had terrible data as an entire district.

So our district goal … is every week … having kids graph and analyze lines in graphs….This is what my students are...doing through Navigator™.

(Ms. C 3050.TIAu06.P29)

01/11/2008

21

Congruence - assessment

And [learning checks] are great because you can… put it by specific objectives and see where you are weak…and who is specifically weak in what area. That just narrows it down to each student by each objective.

(Ms. D 3026.TIAu07.P54)

01/11/2008

22

Costs

I got a late start. And trying to use it every day is hard.

And …equipment management is hard.

So once I felt more comfortable, [then] the more I used it of course….

It’s not hard to adapt, but it takes a little time to adapt your lessons.

(Ms. D 3026.TISp07.P52)

01/11/2008

23

Costs

It’s quite a bit of planning before to get set up and ready. And during class…you have to …change your mind a lot. Because even if you may have tested stuff …, there’s [going to] be issues that don’t work. And you may have to…switch gears right in the middle of your lesson ….So, it is [important] to be quick on your feet.

(Ms. D 3026.TISp07.P75)

01/11/2008

24

Benefits

• Connected classroom technology revealed new information about student learning. – Informal methods of ‘reading’ their classes to

determine the success of their lessons provided inadequate data

– Connected classroom technology gathered data from almost every student in the classroom, rather than just a few.

01/11/2008

25

Benefits – knowing more about student learning

It is just a great tool because you can narrow it down by topic and by student and you can really work with that student on what they are having trouble with and you can see if it is… one or two kids or if it is all of your kids. …It is great on…figuring out why they are missing something… or if it is… something you need to re-teach.

(Ms. D 3026.TIAu07.P72)

01/11/2008

26

Benefits – knowing more about student learning• Kids, especially by this age level, the seventh or eighth

grade, they have learned to play a game. They can know nothing and appear that they know a lot. I think it really brought to light for me that I did not have a good sense of what they are learning by looking at them.

(Ms. C 3050.TISp07.P90)

• I have taught for 18 years and I have been in seventh grade science for about 15 of the 18, and there are things that I have always been really sure that I think kids have understood completely. Now I see what they are thinking, and I am like, whoa, I am just amazed.

(Ms. C 3050.TISp07.P40)

01/11/2008

27

They didn’t know they didn’t know.

I am surprised. I will give a quiz which I think, okay, they should know this. This is pretty basic stuff. And it comes back and some kids on one question, half of them got it wrong. And I am like, whoa.

(Ms. D 3026.TISp07.P67)

01/11/2008

28

Gathering data – questioning in the CCWe were doing earth and sun relationships… revolution versus rotation….We did a learn check and … I didn’t put it up on the screen. But I was able to look on [my screen] and see who was making those mistakes still…. So it helped me because I could pinpoint, without embarrassing them, … who was still mixing those concepts up.

(Ms. C 3050.TIAu06.P47)

01/11/2008

29

Data interpretation – student learning

Connected classroom technology serves as a platform for science data collection and interpretation as well as pedagogical data collection and interpretation.

01/11/2008

30

Data interpretation – science inquiryThere was really strange data that showed up once we

aggregated the class. …the data showed up, and [also] the child’s name.

So I looked at the girl, and I said, “Do you know what you did? Look at this.”

And she said, “Oh, I put the temperature in before the time.”

And I said, “Yes, and what’s wrong with that?”

And she said, “The x information goes first. And I did it backwards”…

But from that she learned what she was doing wrong. I would have never caught that if we were just having a classroom discussion.

(Ms. C3050.TIAu06.P33)

01/11/2008

31

Rapid feedback – just in timeIt really opens my eyes and tells me that ok, they didn’t understand this. It really helps me quickly learn that I need to go back over a few things rather than move on.

‘Cause usually when you give a paper and pencil quiz, and you get it back, and you grade it, it’s like a day before you realize they didn’t understand this question at all.

(Ms. D 3026.TISp07.P67)

01/11/2008

32

Increasing student engagement

If I send them a learn check or a quick poll, I immediately go to screen capture. And every time…I walk by my computer, I just tap the refresh. It gives me an idea where they are. And actually it works really well to keep them on task because they really buy into the idea that I know what they are doing all the time.

(Ms.C 3050.TISp07.P45)

01/11/2008

33

Increasing student engagement

That is great because … every kid is involved… They like seeing that they are involved and it is confidential so they do not feel weird about jumping out and saying who they are if they are not sure about it. But …the kids just love it.

(Ms. D 3026.TIAu07.P56)

01/11/2008

34

Results - for these case studies…

• Connected classroom technology can be integrated in a variety of classroom contexts.

• Strong social support for innovative technology use provides an environment that supports connected classrooms.

• Teacher practice was congruent with CC practice. • Teachers integrate connected classroom

technology into both traditional and inquiry teaching practices.

01/11/2008

35

Results – for these case studies…

• Connected classroom technology helps teachers know more about their students.– More students respond – Rapid feedback – Aggregated class data in a public or private

forum– Discussion of right and wrong answers– Monitor on-task behavior

01/11/2008

36

Next steps…

• Continued analysis of classroom video– Track formative assessment cycles

• Look for complete and incomplete FA cycles– Define task; gather data; evaluate; plan teaching

– Evaluation without data gathering?

– Evaluation with partial data gathering?

• Look for evidence of teacher interactive and preplanned FA• Look for evidence of teacher informed decision making

• Year 2 data collection– Second cohort of science teachers

01/11/2008

37

Paper & Contact Information • Paper

– 5 paper copies of full article available today– Electronic version available at:

• www.ccms.osu.edu/ASTE2008/PSCaseStudy.pdf

Contact Information – Karen Irving Irving.8@osu.edu – Vehbi Sanalan Sanalan.1@osu.edu– Melissa Shirley Shirley.37@osu.edu

Physical Science Connected Classrooms: Case Studies

Karen E. Irving Vehbi A. Sanalan Melissa L. Shirley

The Ohio State UniversityASTE International Conference, St. Louis, MO

January 10-13, 2008

01/11/2008

39

Rapid feedback – just in timeFrom the student perspective, I feel like if they can get that instant feedback, instead of a couple of days later, when they don’t remember what they chose anyway, I feel like this way they really understand it a lot better and they can ask questions right away instead of that time lapse where they forget everything.

(Ms. S 3064.POI07)

01/11/2008

40

Benefits – knowing more about student learning It also allows me to see right away, what things they know and what things they are not catching on to. Like … that one learning check today, I could tell right away that they still cannot tell the difference between a distance graph and a position graph and their meaning.

(Ms. C 3050.poi07.P29)