Measurement error, education production and data envelopment analysis

8/2/2019 Measurement error, education production and data envelopment analysis

1/7

Economics of Education Review 25 (2006) 327333

Measurement error, education production and

data envelopment analysis

John Ruggiero

Department of Economics and Finance, University of Dayton, Dayton, OH 45469-2251, USA

Received 30 September 2004; accepted 18 March 2005

Abstract

Data Envelopment Analysis has become a popular tool for evaluating the efficiency of decision making units. The

nonparametric approach has been widely applied to educational production. The approach is, however, deterministic

and leads to biased estimates of performance in the presence of measurement error. Numerous simulation studies

confirm the effect that measurement error has on cross-sectional deterministic models of efficiency. It is also known that

panel data models have the ability to smooth out measurement error, leading to more reliable efficiency estimates. In

this paper, we exploit known properties of educational production to show that aggregation can also have a smoothing

effect on production with measurement error, suggesting that efficiency analyses are more reliable than previously

believed.

r 2005 Published by Elsevier Ltd.

JEL Classification: I21

Keywords: Educational production; Data envelopment analysis

1. Introduction

Data Envelopment Analysis (DEA) is a nonpara-

metric mathematical programming approach to the

measurement of efficiency that was introduced in the

operations research literature by Charnes, Cooper, and

Rhodes (1978) and Banker, Charnes, and Cooper

(1984). Using linear programming, an observed decisionmaking unit (DMU) is evaluated relative to the

production frontier, which consists of combinations of

observed production possibilities using minimal assump-

tions. The primary advantage of the approach is the

ability to handle multiple inputs and multiple outputs,

particularly in the case when input prices are unavail-

able. The linear programming approach has withstood

the test of time and has become an acceptable approach

for efficiency evaluation.

One important application of DEA is to the analysis

of educational production. Many states have undergone

legal challenges because school districts are not provid-

ing educational services efficiently and outcomes are not

adequate. Reform has moved away from traditionalissues like equity to adequacy and efficiency. The

important policy implication is that school districts

need to spend their money more wisely and increase

their outcomes to acceptable levels. One popular

technique that has been used for measuring efficiency

in education is DEA. Research using DEA to measure

performance of educational production in the United

States include Bessent, Bessent, Kennington, and

Reagan (1982), Fa re, Grosskopf, and Weber (1989),

ARTICLE IN PRESS

www.elsevier.com/locate/econedurev

0272-7757/$- see front matterr 2005 Published by Elsevier Ltd.

doi:10.1016/j.econedurev.2005.03.003

Tel.: +1937 2292258; fax: +1937 2292477.

E-mail address: [email protected].
http://www.elsevier.com/locate/econedurevhttp://www.elsevier.com/locate/econedurev


2/7

Ray (1991), McCarty and Yaisawarng (1993), Ruggiero

(1996, 2001), and Duncombe, Miner, and Ruggiero

(1997). DEA studies analyzing performance in other

countries include Silva Portela and Thanassoulis (2001),

Farren (2002) and Muniz (2002).

The ability of DEA to measure efficiency is dependent

on the nature of observed production. Numerous studies

have shown that the performance of DEA deteriorates

in the presence of measurement error and other

statistical noise. Simulations in Banker, Gadh, and Gorr

(1993), Ruggiero (1999), and Bifulco and Bretschneider

(2001), for example, clearly show that deterministic

models including DEA are sensitive to measurement

error in cross-sectional analyses. Given an econometri-

cians view of the world, the use of DEA could be

problematic and potentially harmful given the important

policy implications. Gong and Sickles (1992) and

Ruggiero (2004) use simulation to show that the

problem of measurement error is alleviated whenefficiency is estimated using panel data. These models

assume that the production technology is the same

across time and use averaging to smooth production

with respect to measurement error.

In a separate but related literature, Hanushek (1979)

discusses other important issues with empirical educa-

tional production analyses. One of the important issues

involves the choice of aggregation. Conceptually,

education occurs at the student level, but most analyses

rely on aggregated data (primarily due to availability)

to analyze educational production. One potential

advantage, which is exploited in this paper, is the

smoothing of production with respect to measurement

error. Hence, aggregation, like panel data, helps

alleviate the problem of measurement error. In parti-

cular, we assume that educational production with

measurement error occurs at a disaggregated level and

consider efficiency measurement using both disaggregate

and aggregate data.

Simulation analysis is used to show that the use of

DEA on disaggregated data in the presence of measure-

ment error leads to biased efficiency estimation,

confirming the results of previous analyses. We also

apply DEA to aggregated data and show that aggrega-

tion can lead to unbiased efficiency estimates. Theseresults represent an important contribution to the DEA

literature: aggregation effectively controls for statistical

noise. This suggests that DEA can be used effectively to

identify inefficiency of school districts (schools), where

data are aggregated from the school (classroom) level.

We note that such aggregation can be applied to other

areas such as analyzing bank performance where

production occurs at the branch level. For purposes of

this paper, we consider district data as the aggregate of

school data. The rest of the paper is organized as

follows. The next section provides a model of educa-

tional production and efficiency using DEA. The third

section analyzes the performance of DEA using simula-

tion analysis and the last section concludes.

2. Educational production

We describe the educational production process as

follows. Assume that each of Ns schools uses a vector

X x1; :::; xM of M discretionary school inputs to

produce a vector Y y1; :::; yT of T outcomes. For

school j, inputs are given by Xj x1j; :::; xMj and

outputs by Yj y1j; :::; yTj. For purposes of this

paper, we ignore the role of non-discretionary socio-

economic variables to focus on measurement error and

aggregation. This further allows comparisons to the

approach used by Bifulco and Bretshneider (2001).

Ruggiero (1998) extended DEA to the case of multiple

non-discretionary inputs; allowing measurement error

with non-discretionary inputs is a trivial extension ofRuggieros model using the approach of this paper.

Assuming variable returns to scale, technical effi-

ciency can be estimated at the school level using the

Banker, Charnes, and Cooper (1984) input-oriented

model:

Minyol1 ;:::;lNS

s:t.

XNS

i1

liytiXyto 8 t 1; :::; T

XNS

i1

lixmipyoxmo 8 m 1; :::; M

XNS

i1

li 1

li ! 0 8 i 1; :::; NS. 1

In the absence of measurement error and with a

sufficiently large sample size, this model works well in

measuring relative performance with multiple inputs and

outputs. The returns to scale assumption can be

tightened to allow only constant returns by excluding

the convexity constraint; leading to the Charnes,Cooper, and Rhodes (1978) model. The analysis of a

given school seeks a maximum equi-proportional con-

traction of all inputs consistent with frontier production

with all outputs at least as high as the levels of the school

under analysis. Alternatively, we could have focused on

the output oriented models.

Model (1) leads to biased efficiency estimates in the

presence of measurement error. Numerous simulation

analyses confirm that even moderate levels of measure-

ment error can lead to severe bias. For this reason, the

usefulness of DEA to analyze educational production

must be questioned. Consider measurement error in one

ARTICLE IN PRESS

J. Ruggiero / Economics of Education Review 25 (2006) 327333328


3/7

output j, with ~yj yj j, where ~yji yji ji for school

i. We note that this extends Ruggiero (2004) by

considering additive error in the output. The BCC

model for measuring efficiency of a given school is then

given by

Min yol1 ;:::;lNS

s:t.

XNS

i1

liytiXyto 8 t 1; :::; T; taj

XNS

i1

li~yjiX ~yjo

XNS

i1

lixmipyoxmo 8 m 1; :::; M

XNS

i1

li 1

liX0 8 i 1; :::; NS. 2

As discussed in Ruggiero (2004), measurement error

biases efficiency estimation for two reasons. First, the

right-hand side of the contaminated output constraint

distorts the measurement by comparing frontier output

to a level of output not consistent with that school. This

results from measurement error for the school under

analysis. The second effect involves the location of the

frontier output; measurement error in other schools

will lead to a biased location of the true frontier. For a

further discussion as it relates to panel data, seeRuggiero (2004).

We consider the aggregation of the data to the school

district level. We assume that each of the NS schools can

be assigned to one and only one of the ND school

districts. Defining NSd as the number of schools in

district d for d 1; :::; ND, we construct an index that

identifies the school according to the district to which it

belongs. Then, for each district d, we can define a vector

for each input m and output t given by

Xmd xmd;1; :::; xmd;NSd

and Ytd ytd;1; :::;ytd;NSd

; respectively.

We note that ND PD

d1NSd prior to estimating

efficiency, we first average school level data to the

district level for each district d

xmd 1

NSd

XNSd

i1

xmd;i 8 m,

ytd 1

NSd

XNSd

i1

ytd;i 8 taj; and

yjd 1

NSd

XNSd

i1

~yjd;i:

With the additional assumption that the E 0, the

error term is averaged out with a sufficiently large

number of schools in the district. In the simulation

analysis performed in the next section, we vary the

number of schools within a district to be 5, 10 or 20 and

show that averaging data even with few schools can

effectively eliminate measurement error problems. One

concern raised by an anonymous reviewer was the

assumption of aggregating from the school level to the

district level. We note that state policy typically funds at

the district level and compares district performance on

state tests. And, in cases where this is not true,

aggregation could happen from the student or classroom

level to the school level. For space consideration, we

show only the aggregation from the school to the district

level; the results from a lower level to the school level

follow.

3. Simulation analysis

We assume that production can be characterized by

the conversion of discretionary inputs into output. As

mentioned above, non-discretionary inputs can be

effectively incorporated into the DEA model (see

Ruggiero, 1998). In this simulation, we focus exclusively

on the impact of measurement error. In particular, we

assume that three inputs x1, x2 and x3 are used by

schools to produce one output y according to the

following stochastic production function:

y u1vx0:41 x0:42 x0:23 ,

where u and v represent inefficiency and statistical noise,

respectively. Input data were generated uniformly on the

interval (5,10) for each input. Measurement error was

generated log-normal with standard deviation sv where

sv took on values 0.1, 0.2 and 0.3. Notably, sv 0:2

represented high measurement error in the Bifulco

and Bretschneider (2001) and hence, we consider an

even higher measurement error variance. The

inefficiency component ln u was generated half-normal

with standard deviation 0.2. By varying the standard

deviation of the measurement error, we effectively

allow varying ratios of measurement error toinefficiency variances. Finally, we consider various

combinations of schools and school districts; we vary

the number of schools within each district to take on

values of 5, 10 and 20 and we allow 50, 100 and 200

districts.

Initially, each school within a given district had the

same level of inefficiency. In the last simulation, we

allow school inefficiency to vary within a district. Three

measures of performance are considered in this study:

mean absolute deviation (MAD) between true and

measured efficiency, the correlation coefficient and the

rank correlation coefficient. Because the assumed

ARTICLE IN PRESS

J. Ruggiero / Economics of Education Review 25 (2006) 327333 329


4/7

production function is characterized by constant returns

to scale, we use the CCR model that excludes the

convexity constraint. This will allow us to focus

exclusively on measurement error in technical efficiency

estimation.

The performance results using school level data arereported in Table 1. Not surprisingly, these results

confirm the results known in the literature: measurement

error has a devastating effect on DEA performance.

Consider the absolute deviations reported in Table 1. In

general, we note that as the measurement error variance

increases, the MAD gets larger. Further, the standard

deviation increases as well. Even when sv was low (0.1),

the MAD was still unacceptably high in the 0.100.16

range. Holding constant the measurement error var-

iance, we also note the general result that the MAD gets

worse even as the number of schools and the number of

districts increase.

Turning to the correlation and rank correlation

results, we see a similar pattern. DEA performs better

when the measurement error variance is relatively low

while the correlation and rank correlation decrease as

the variance is increased. Even with a low measurement

error variance, DEA does not provide accurate estimatesof efficiency. The correlation and rank correlations are

all below 0.75 with sv 0:1. With sv 0:3, the highest

correlation is about 0.41. No general pattern for

improvement appears as the number of schools or

districts increases. Clearly, the use of DEA on disag-

gregated data in the presence of measurement error is

suspect.

Next, we estimate efficiency using the constant returns

to scale model using data averaged to the district level.

The results are reported in Table 2. The MAD results in

Table 2 reveal that DEA performs extremely well

relative to the performance using school level data.

ARTICLE IN PRESS

Table 1

Performance results using school level simulated data

sv Number of schools Number of districts Absolute deviation Correlation coefficients

Mean Standard deviation Pearson Rank

0.1 5 50 0.111 0.075 0.648 0.622100 0.115 0.074 0.716 0.680

200 0.128 0.071 0.726 0.694

10 50 0.128 0.072 0.683 0.666

100 0.157 0.074 0.545 0.532

200 0.163 0.073 0.709 0.680

20 50 0.134 0.072 0.680 0.665

100 0.140 0.073 0.740 0.723

200 0.123 0.071 0.714 0.682

0.2 5 50 0.260 0.122 0.463 0.454

100 0.287 0.123 0.500 0.473

200 0.314 0.114 0.462 0.450

10 50 0.302 0.116 0.447 0.432

100 0.299 0.119 0.510 0.498

200 0.336 0.118 0.443 0.435

20 50 0.272 0.120 0.423 0.406

100 0.336 0.119 0.481 0.461

200 0.326 0.115 0.481 0.460

0.3 5 50 0.383 0.152 0.318 0.296

100 0.388 0.146 0.389 0.395

200 0.404 0.148 0.332 0.339

10 50 0.381 0.147 0.344 0.325

100 0.419 0.146 0.319 0.319

200 0.607 0.114 0.288 0.301

20 50 0.418 0.131 0.348 0.337

100 0.452 0.125 0.406 0.409

200 0.472 0.132 0.332 0.329

All calculations by author. School efficiency was held constant within each district.



5/7

With low measurement error, the MAD is under 0.06

when the variance of the measurement error is low. As

the measurement error variance increases, holding the

number of districts and schools constant, the perfor-

mance declines but at levels far better than reported for

the school level data. All MADs are lower than 0.2 whenusing the aggregated data. We also note that the

performance of DEA improves as the number of schools

increases and remains about the same as the number of

districts increases.

Similar results are observed for the correlation and

rank correlations reported in Table 3. In general, all

correlations show marked improvement using the

aggregate data compared to the school level analysis.

In the case of low measurement error variance, the

correlation and rank correlation coefficients are above

0.85 and usually above 0.90. Similar to the MAD results,

the performance of DEA declines as the measurement

error variance increases. This is offset by improved

performance as the number of schools increases. This is

an important finding; as the number of units at the

disaggregated level increases, the effect of measurement

error on DEA performance at the aggregate level

becomes less pronounced. It is also important toremember that the units chosen for the disaggregated

level were schools. Extending the units to the classroom

or even the student level suggests that DEA at the

district level provides meaningful results even with high

measurement error. Of course, the standard assumption

that the expected value of the error term is zero was

employed.

One questionable assumption that was used in the

simulation was constant efficiency at the school level

within each district. For completeness, we now consider

random efficiency within a given school district. Holding

the number of schools at 20 and the number of districts

ARTICLE IN PRESS

Table 2

Performance results using simulated data aggregated to the district level

sv Number of schools Number of districts Absolute deviation Correlation coefficients

Mean Standard deviation Pearson Rank

0.1 5 50 0.045 0.031 0.869 0.866100 0.035 0.028 0.924 0.897

200 0.054 0.034 0.915 0.856

10 50 0.025 0.019 0.949 0.937

100 0.025 0.018 0.960 0.941

200 0.029 0.023 0.952 0.917

20 50 0.029 0.021 0.974 0.948

100 0.040 0.025 0.934 0.904

200 0.028 0.019 0.974 0.942

0.2 5 50 0.096 0.064 0.744 0.714

100 0.100 0.059 0.822 0.802

200 0.112 0.062 0.776 0.695

10 50 0.054 0.047 0.830 0.775

100 0.059 0.042 0.895 0.874

200 0.061 0.039 0.834 0.840

20 50 0.027 0.023 0.931 0.908

100 0.042 0.030 0.914 0.875

200 0.053 0.032 0.932 0.889

0.3 5 50 0.198 0.096 0.553 0.635

100 0.134 0.082 0.701 0.692

200 0.147 0.087 0.642 0.617

10 50 0.084 0.065 0.755 0.718

100 0.104 0.073 0.753 0.679

200 0.128 0.067 0.712 0.689

20 50 0.088 0.053 0.806 0.723

100 0.081 0.043 0.876 0.852

200 0.083 0.050 0.837 0.762

See comments in Table 1.



6/7

at 200, school efficiency was regenerated as ln us l-

n ud+ln e, where ln ud was generated as above and ln e

generated as N(0,s) with s taking on values of 0.02, 0.04

and 0.06. The value of us was set to one if it was

calculated to be greater than unity. As shown in Table 3,

this generating process led to large differences in school

efficiency within districts.

The results of this simulation are reported in Table 3.

We note that the first column of results repeats theresults from the earlier simulation where there was no

variation in efficiency among schools in the same

district. As s increases, we note that the variation

increases within a district. In the last column, the

average difference between maximum and minimum

efficiency is approximately 0.17. The performance of

DEA is measured by all three criteria: MAD, correlation

and rank correlation. District efficiency was measured

using the aggregated data and compared to the true

value, determined by the average true efficiency. We

note that the results are similar across columns for all

measures. This suggests that the assumption of constant

efficiency within a district does not effect performanceevaluation at the aggregate level. Of course, a trade-off

exists; with the aggregate data, it is not possible to

identify efficiency of the individual units. However, the

approach does work in identifying average efficiency of

the aggregate units.

4. Conclusions

The use of DEA to evaluate performance has been

called into question because of the potential bias that

exists when measurement error exists. Previous simula-

tion studies have shown that performance drops

significantly as the variance of measurement error

increases. In this paper, the problem of measurement

error was highlighted by two effects. First, measurement

error on a particular unit affects the placement of the

unit relative to the true frontier. Also, measurement

error on other units effects the evaluation of a given unit

because these other units might serve as reference unitsfor the unit under analysis. Unlike stochastic frontier

models that use panel data, DEA is deterministic.

However, as shown in this paper, aggregation of data

to higher units (for example, from school level to district

data or from classroom to school level) can smooth

measurement error. And, as a result, performance

evaluation using aggregate data can produce reliable

results, eve when measurement error is substantial.

As recommended by an anonymous referee, an

interesting extension of this paper would be a real-world

application. In particular, a useful application would be

to show how aggregation of individual student or

classroom data could help identify school and districtinefficiency. Data requirements prevented such an

application; however, I expect such data will be available

in the near future.

References

Banker, R. D., Charnes, A., & Cooper, W. W. (1984). Some

models for estimating technical and scale inefficiencies in

data envelopment analysis. Management Science, 30(9),

10781092.

ARTICLE IN PRESS

Table 3

Simulation results assuming varying school efficiency within districtsa (number of districts 200, number of schools 20, sv 0:2)

District descriptive statistics s

0b 0.02 0.04 0.06

Range usc

Mean 0 0.061 0.118 0.168

Standard deviation 0 0.014 0.028 0.041

Minimum 0 0.024 0.044 0.077

Maximum 0 0.095 0.200 0.305

Efficiency results

Correlation coefficient 0.932 0.924 0.926 0.926

Rank correlation coefficient 0.889 0.890 0.887 0.890

Absolute deviation

Mean 0.053 0.053 0.050 0.043

Standard deviation 0.032 0.032 0.031 0.027

aAverage district efficiency was randomly generated as: ln ud$jN0; 0:2j. School efficiency was calculated as ln us ln ud ln ,

where ln $N0; s and s is given in the table. If us was calculated to be greater than unity, it was set equal to one.bThe range is the maximum usminimum us, calculated for each district.cThis column is taken from previous tables and serves only as a reference.



7/7

Banker, R., Gadh, V., & Gorr, W. (1993). A monte carlo

comparison of two production frontier estimation methods:

Corrected ordinary least squares and data envelopment

analysis. European Journal of Operational Research, 67(3),

332343.

Bessent, A., Bessent, E. W., Kennington, J., & Reagan, B.

(1982). An application of mathematical programming toassess productivity in the Houston independent school

district. Management Science, 28(12), 13351366.

Bifulco, R., & Bretschneider, S. (2001). Estimating school

efficiency: A comparison of methods using simulated data.

Economics of Education Review, 20(5), 417429.

Charnes, A., Cooper, W. W., & Rhodes, E. (1978). Measuring

the efficiency of decision making units. European Journal of

Operational Research, 2(6), 429444.

Duncombe, W., Miner, J., & Ruggiero, J. (1997). Empirical

evaluation of bureaucratic models of inefficiency. Public

Choice, 93(1), 118.

Fare, R., Grosskopf, S., & Weber, W. (1989). Measuring school

district performance. Public Finance Quarterly, 17(4), 409428.

Farren, D. (2002). The technical efficiency of schools in Chile.Applied Economics, 34(12), 15331542.

Gong, B., & Sickles, R. (1992). Finite sample evidence on the

performance of stochastic frontiers and data envelopment

analysis using panel data. Journal of Econometrics, 51(1),

259284.

Hanushek, E. (1979). Conceptual and empirical issues in the

estimation of educational production functions. Journal of

Human Resources, 14(3), 351388.

McCarty, T., & Yaisawarng, S. (1993). Technical efficiency in

New Jersey school districts. In H. O. Fried, C. A. K. Lovell,

& S. S. Schmidt (Eds.), The measurement of productive

efficiency (pp. 271287). New York: Oxford University

Press.

Mun iz, M. (2002). Separating managerial inefficiency and

external conditions in data envelopment analysis. EuropeanJournal of Operational Research, 143(3), 625643.

Ray, S. (1991). Resource-use efficiency in public schools: A

study of connecticut data. Management Science, 37(12),

16201628.

Ruggiero, J. (1996). Measuring technical efficiency in the public

sector: An analysis of educational production. Review of

Economics and Statistics, 78(3), 499509.

Ruggiero, J. (1998). Non-discretionary inputs in data envelop-

ment analysis. European Journal of Operational Research,

111(3), 461468.

Ruggiero, J. (1999). Efficiency estimation and error decom-

position in the stochastic frontier model: A monte carlo

analysis. European Journal of Operational Research, 115(3),

555563.Ruggiero, J. (2001). Determining the base cost of education: An

analysis of Ohio school Districts. Contemporary Economic

Policy, 19(3), 268279.

Ruggiero, J. (2004). Data envelopment with stochastic data.

Journal of the OR Society, 55(9), 10081012.

Silva Portela, M. C., & Thanassoulis, E. (2001). Decomposing

school and school-type Efficiency. European Journal of

Operational Research, 132(2), 357373.

ARTICLE IN PRESS


Measurement error, education production and data envelopment analysis

Documents

Transcript of Measurement error, education production and data envelopment analysis