Barrow Evidence Computerized Algebra Instruction December 2007

Technology’s Edge: The Educational Benefits of Computer-Aided Instruction

Lisa BarrowFederal Reserve Bank of Chicago

Lisa MarkmanPrinceton University

Cecilia RousePrinceton University and NBER

Student achievement is critical for individuals and society

• U.S. math skills have been increasing (NAEP), but proficiency levels remain low.

• Math skills may explain a large portion of wage inequality (Grogger, 1996; Murnane, Willet, & Levy, 1995)

School districts are turning to advances in computers to…

– Reduce administrative burden;

– Compensate for poor teacher content knowledge (especially in districts that report difficulty recruiting and retaining teachers, particularly in math and science);

– Allow more individualized student attention; students can progress at own pace.

Evidence on effectiveness of CAI is surprisingly weak

– Poorly defined computer use.

– Little use of randomized controlled study design.

In a 2001 review of the literature, Larry Cuban (2001, p. 179) concludes,

“When it comes to higher teacher and student productivity and a transformation of teaching and learning … there is little

ambiguity. Both must be tagged as failures. Computers have been oversold

and underused, at least for now.”

Computer-Aided Instruction (CAI) Treatment:

• Typically used in large urban districts;• Software and hardware package designed

to deliver one-on-one instruction;• Software described as “meeting National

Council of Teachers of Mathematics standards”;

• Software may be configured to state standards;

• Includes classroom management tools.

The Program

• 5 components per lesson:– Pretest– Review– Lesson– Cumulative review– Comprehensive test

• Required to achieve certain degree of mastery before advancing;

• Teachers can monitor student progress.

The design of this experiment…

• Randomly assign 8th and 9th grade classes in three large urban districts to be taught using computer-aided algebra and pre-algebra instruction.

• Assess the impact on statewide tests and tests designed to target algebra and pre-algebra skills.

• Note that the computer use was well-defined.

Research Design: Within-school random assignment to

CAI or traditional instruction at the classroom level in three districts

Research design addresses two forms of selection bias:

• Non-random assignment of students to CAI;• Non-random assignment of teachers to CAI.

Table 1: Characteristics of the Districts in Our Study

Top 100 Districts

3 Districts Combined District 1 District 2 District 3

# Students 112,807 62,507 ~68,000 ~22,000 ~97,000

% Female 48.8 49.4 49.7 48.8 49.3

% Native American 0.6 0.5 0.1 0.1 1.0

% Asian 7.1 3.1 1.9 0.8 4.4

% African American 28.1 69.5 93.6 40.3 59.4

% Hispanic 34.1 16.2 1.1 54.3 18.0

Implementation of Random Assignment

• Schools provided us with class schedules of pre-algebra and algebra classes;

• Given option of eliminating particular teachers or classes from the experiment;

• Randomization information provided to schools after students’ classes were scheduled.

Teacher Period 1 Period 2 Period 3 Period 4

A Alg 1 Alg 1A Alg 1

B Alg 1 Alg 1A Alg 1A

C Alg 1A Alg I

D Alg I Alg 1 Alg 1A

Randomized Evaluation of a Computerized Math Curriculum

Current School Schedule


A Alg 1 Alg 1A Alg 1

B Alg 1 Alg 1A Alg 1A

C Alg 1A Alg I

D Alg I Alg 1 Alg 1A


A Regular Regular LAB

B LAB LAB Regular

C Regular Regular

D Regular Regular LAB

Randomized Evaluation of a Computerized Math Curriculum

Current School Schedule

Below is a sample schedule that would be returned to each school after random assignment

Table 2a: Schools and Students in Study – District 1Relevant Schools

Schools in Study

Students in Study

Number of Students 29,603 8,148 973

% Grade 8 19.3 16.8 40.4

% Grade 9 18.0 18.3 47.2

% Grade 10 15.1 17.8 4.4

% Female 50.5 49.0 52.0

% African American 94.2 97.2 87.8

% Hispanic 1.0 0.8 0.8

% White 2.6 0.4 0.1

% Native American <0.1 <0.1 0.0

% Asian 2.2 1.6 1.8

% Missing demographic data

9.6

Table 2b: Schools and Students in Study – District 2Relevant Schools

Schools in Study

Students in Study


% Grade 8 2.3 0.0 0.0

% Grade 9 38.0 40.0 52.7

% Grade 10 22.0 23.2 31.8

% Female 48.4 48.2 46.7


% Hispanic 50.1 51.2 44.7

% White 5.5 5.9 6.6

% Native American 0.2 0.1 0.2

% Asian 0.7 0.8 0.5


0.2

Table 2c: Schools and Students in Study – District 3Relevant Schools

Schools in Study

Students in Study


% Grade 8 1.4 0.0 3.5

% Grade 9 35.6 40.0 91.5

% Grade 10 23.3 25.1 3.0

% Female 49.9 47.6 47.7


% Hispanic 15.2 1.2 0.5

% White 18.3 4.0 1.5

% Native American 1.1 0.4 0.0

% Asian 4.5 1.9 3.0


0.5

Numbers of Schools Classes, Teachers, and Randomization Pools

Analysis Sample

Combined District 1 District 2 District 3

Number of schools 17 10 4 3

Number of randomization pools

60 31 19 10

Number of classes 141 74 44 23

Number of teachers 57 36 14 7

Number of students 1,585 973 412 200

Outcomes

• Algebra and pre-algebra tests by Northwest Evaluation Association (NWEA) to be consistent with state and district standards;

• State-wide administered math tests;

• District benchmark tests in pre-algebra.

Table 3: Randomization of Treatment and Control Using Full Sample

Random Assignment

TraditionalInstruction CAI

p-value ofdiff.

Full Sample

Baseline algebra test score

24.7 24.7 0.494

Female 47.2 47.1 0.637

African American 80.0 83.2 0.561

Hispanic 15.9 13.5 0.195

Class Size 25.8 25.7 0.860

# of Observations 1,133 1,145

Table 3 (cont.): Randomization of Treatment and Control Using Analysis Sample

Random Assignment

TraditionalInstruction CAI

p-value ofdiff.

Analysis Sample

Baseline algebra test score

24.7 24.8 0.304

Female 51.1 48.9 0.148

African American 81.9 84.0 0.060

Hispanic 13.8 12.1 0.061

Class Size 25.8 26.2 0.549

# of Observations 785 800

Intent-to-Treat Estimates of the Effect of CAI on Algebra Achievement

(with and without Teacher Fixed Effects)

0.17

0.37

0.17

0.28

0

0.1

0.2

0.3

0.4

0.5

0.6

Without Teacher Fixed Effects With Teacher Fixed Effects

Est

imat

ed In

tent

-to-T

reat

Effe

ct S

ize

Advantages/Disadvantages of the Intent-to-Treat Effect

• Represents the gains a policy maker might reasonably expect to observe.

• Does not necessarily represent the effectiveness of the program.

Amount of Time in the Computer Lab by the Random Assignment of the Student’s Class

80.1

10.3 109.1

70.3

5.6

52.7

14.8

64.5

33

55.3

83.8

0

10

20

30

40

50

60

70

80

90

Number oflessons studentsare expected to

complete

% of stud.completing nolessons in CAI

% of stud.completing >10lessons in CAI

% of stud.completing >20lessons in CAI

Nr. of lessonscompleted in CAI

Nr. of CAI lessonscompleted as a %

of courseexpecations

Traditional Instruction CAI

Intent-to-Treat and Treatment on the Treated Estimates of the Effect of CAI

(with and without teacher fixed effects)

0.173

0.283

0.249

0.417

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Without Teacher Fixed Effects With Teacher Fixed Effects Inte

nt-to

-Tre

at a

nd T

reat

men

t-on-

the-

Trea

ted

Effe

ct S

izes

Intent-to-Treat Estimates in District 1 Using Different Tests

0.26

0.604

0.381

0.226

0

0.2

0.4

0.6

0.8

1

1.2

1.4

Algebra Scale Score 2nd Qtr Benchmark 3rd Qtr Benchmark State Math Test

Inte

nt-to

-Tre

at E

ffect

Siz

e

Intent-to-Treat Estimates in Districts 2 and 3 Using Different Tests

0.2

0.089

-0.124-0.062

-0.5

-0.4

-0.3

-0.2

-0.1

0

0.1

0.2

0.3

0.4

0.5

Algebra Scale Score State Math Test

Inte

nt-to

-Tre

at E

ffect

Siz

e

District 2 District 3

We might expect to see an advantage of CAI in…

• Classes where curriculum best suited to students;

• Larger classes;• Classes with more

disruptive students;• Classes with

heterogeneous students.

Differential Intent to Treat Effects of the Computerized Instruction on Pre-Algebra and

Algebra Achievement by Class Type

0.510.440.450.48

1.36

0.01 0.07 0.13

-0.31-0.23

-1.00

-0.50

0.00

0.50

1.00

1.50

2.00

2.50

3.00

All 3 districts Districts 1 & 2 District 1 District 2 District 3

Inte

nt-to

-Tre

at E

ffect

Siz

e

CAI Effect for Pre-Algebra CAI Effect for Algebra

Differential Intent to Treat Effects of CAI on Pre-Algebra and Algebra Achievement by

Baseline Test Score Quartile

0.2160.242

0.171 0.155

-0.1

0

0.1

0.2

0.3

0.4

0.5

Bottom baseline test scorequartile

2nd baseline test scorequartile

3rd baseline test scorequartile

Top baseline test scorequartile

Differential Intent to Treat Effects of CAI on Pre-Algebra and Algebra Achievement by

Individual Attendance Rates

0.439

-0.221 -0.051 -0.02

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

1

1.2

Bottom baselineattendance quartile

2nd baseline attendancequartile

3rd baseline attendancequartile

Top baseline attendancequartile

Differential Intent-to-Treat Effects by Class Characteristic: Attendance, Class Size, and Class

Baseline Test Score S.D.

0.06

0.350.21

0.01

-1.1

0.412

-1.2

-1

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

ClassAvg.

Attend.Rate

Attnd.Rate OneSD below

Mean

Class size=25

Class size=15

Smallclasses

Largeclasses

Est

imat

ed In

tent

-to-T

reat

Effe

ct

Overall we find:

• On average, students in classes taught using CAI scored higher on algebra tests than students in traditionally-taught classes.

• The effects appear larger for students in larger classes (especially large, heterogenous classes), those with worse attendance rates, and those in classes with lower average attendance rates.

Cost-Benefit Analysis

Cost of CAI

# ClassesTotal

studentsClass size # periods

CAI labs needed

Cost/ student

School A 22 730 33.2 8 3.0 $218

School B 12 321 26.8 8 1.5 $245

District 1 74 1736 23.5 8 9.3 $279

Cost of reducing class size to 13

# ClassesTotal

studentsClass size # periods

# new teachers

Cost/ student

School A 22 730 33.2 6 5.7 $329

School B 12 321 26.8 6 2.1 $278

District 1 74 1736 23.5 6 9.9 $241

Barrow Evidence Computerized Algebra Instruction December 2007

Documents

Transcript of Barrow Evidence Computerized Algebra Instruction December 2007