What can we learn from international comparisons? PISA ......Gundel Schümer, Max Planck Institute...

1

What can we learn from international comparisons?

PISA 2000 disentangled

Gundel Schümer, Max Planck Institute for Human Development, Berlin

I have been invited to this forum because I am familiar with some of the international school

achievement studies and, in particular, because I have worked on PISA 2000, the first of three

cycles of the OECD’s Programme for International Student Assessment. Researchers working

with the PISA data are assumed to be able to demonstrate how the study has kept the OECD’s

promise to

• afford “insights into the factors that contribute to the development of competencies

and into how these factors operate in different countries”,

• lead to a “better understanding of the causes and consequences of observed skill

shortages” and

• provide “directions for national policy, for schools’ curriculum and instructional

efforts and for students’ learning” (OECD 2001, p.3).

I shall do my best, but I shall not be able to identify causal factors explaining desirable or less

desirable findings, or contributing clearly to the justification of political decisions. I shall

come back to the limits of the international studies later. First, however, I want to comment on

some PISA results for the United Kingdom, compare these results with the corresponding

findings for other countries and ask how more desirable outcomes might be obtained.

I.

All things considered, 15-year-old pupils in the UK1 did rather well in PISA 2000. Only a few

of the 28 OECD countries participating in the study attained significantly better results2:

• Finland and Canada in reading,

• Japan and Korea in mathematics and

• Korea in science.

Some other OECD countries performed at a comparable level to the UK, but the majority

fared significantly worse.

Both parents and pupils seem to be satisfied with the educational system in the UK. The

percentage of 15-year-olds attending supplementary or remedial courses outside of school is

2

clearly below the OECD average. Moreover, UK pupils feel better supported by their teachers

than their contemporaries in all the other OECD countries, they often see teacher-student

relationships in a more favourable light, and they have a comparatively strong sense of be-

longing to their schools, even though they feel more pressure to do well at school and do more

homework than the pupils in most other OECD countries.

Since the PISA results were far more encouraging for the UK than the results of earlier

international studies, it seems fair to assume that the high performance of the PISA 2000

cohort may be attributed to the school reforms that have been instituted in the meantime. As

Pam Sammons and her co-workers pointed out in a report on schooling in England, the pupils

who participated in PISA 2000 were “amongst the first to experience both a national

curriculum and national assessment aligned to that curriculum. They attended schools subject

to regular inspection and whose national assessment and inspection results were published.

Schools received national and local guidance material and support from LEAs […]. In

addition, teachers were provided with considerable in-service education and professional

development opportunities, especially in the core curriculum subjects (English, mathematics

and science) and in relation to teacher assessment and feedback. […] The core curriculum

focus on English, mathematics and science, involving national tests, gave literacy and

numeracy a particularly high profile.” Finally, there was “greater awareness of the need to

focus on the performance of different pupil groups, with an emphasis on promoting equity and

raising standards for minority ethnic groups, boys’ (and in some subjects girls’) attainment

levels, and differences related to SES” (Sammons et al. 2003, p. 136).

However, it appears that the UK has been less successful at promoting equity for various

groups of disadvantaged pupils. The high average score of the UK pupils masks the fact that

there are wide differences in performance within the country, especially in reading. The range

of reading skills in the UK is wider than the OECD average. According to the English PISA

report, which relates to more than 82 per cent of the pupil population in the UK, “proficiency

in reading literacy is most closely associated with the following characteristics, in descending

order of importance:

• having parents with high socio-economic status occupations;

• attending an independent school;

• attending a single-sex school;

• speaking English at home;

3

• having parents with qualifications at upper secondary level or higher;

• being an only child or having only one sibling,

• being a girl,

• living with both natural parents; and

• being in Year 11” (Gill et al. 2002, p. xv).

The gender gap is moderate compared to other OECD countries and the gap between pupils in

Year 10 and Year 11 indicates only that an additional year of schooling indeed has positive

effects on reading performance. What is surprising is that this additional year of schooling is

the least influential of the nine factors mentioned (Gill et al. 2002). And disturbingly, with the

exception of ‘sex’ and ‘year’, all of these characteristics are either social factors or related to

social factors. Even ‘living with both natural parents’ and ‘being an only child or having only

one sibling’ are factors which are associated with the socio-economic status of the pupil’s

family.

The association between pupils’ socio-economic background and reading performance is

alarming for all those concerned with the equality of educational opportunities. In the UK, the

differences between pupils belonging to the highest and lowest SES groups are among the

largest of all participating countries. Only Belgium and Switzerland show greater differences

than the UK in all three test domains, Germany and Hungary in two domains, and Luxemburg

and the United States in one domain (Deutsches PISA-Konsortium 2001).

II.

The relationship between social background and school performance can be demonstrated by

social gradients. In the following diagram (Graph 13), each pupil participating in PISA 2000 is

represented by a dot. The higher the dot is situated in the cloud, the better is the pupil’s

reading performance; and the further the dot is to the right, the higher the social status of the

pupil’s family. The PISA index of pupils’ social background is based on data relating to

parental occupation, wealth, cultural resources, parental education, family structure and immi-

grant status. The white line represents the social gradient, that is, the regression line indicating

how pupils’ reading performance relates to the social structure of the pupil population – or

how inequality in reading literacy is attributable to socio-economic factors.

4

The next diagram (Graph 23) shows the social gradients for Finland, the UK and Germany.

The different levels of the three lines illustrate the differences in the pupils’ reading literacy

across the three countries. The fact that the three lines are of roughly the same length indicates

similar ranges of scores on the index of social background in these three pupil populations

compared to the other OECD countries. The slopes of the lines demonstrate the impact of the

pupils’ social background on their reading performance.

As you can see, the slope of the Finnish gradient is relatively gentle, i.e. the Finnish pupils’

reading performance is less dependent on their social background than that of the British or

the German pupils. Furthermore, the British and the Finnish gradients converge at higher

levels of economic, social and cultural status, indicating that British pupils who have grown

up in privileged families perform as well as Finnish pupils belonging to the highest SES

groups. Yet there is a substantial performance gap between Finnish and British pupils from

the lowest SES groups. In other words, if the UK were aiming to increase mean reading per-

formance, it would have to provide additional support for low achieving pupils. Conversely, if

the dependency of reading performance on the social status of pupils’ families were reduced,

the mean reading performance of UK pupils would increase.

5

More detailed comparisons of the social gradients of the PISA countries may seem to suggest

that the slopes are dependent on the proportion of ethnic minority pupils in the population. As

shown in the following graph (Graph 3), the participating countries can be arranged according

to their pupils’ mean reading performance and the slope of the social gradient (Deutsches

PISA-Konsortium 2001, p. 392). Low-achieving countries, including Germany (the country

with the steepest social gradient), are shown on the left-hand side. High achieving countries,

upon which we shall concentrate in the following, are shown on the right-hand side. The

countries in the upper part of the graph, including the UK, have steeper social gradients; those

in the lower part have more gentle gradients. The fact that Finland, Korea, Japan and Iceland

belong to this group of exemplary countries seems to suggest that the gentle slopes are related

to the cultural homogeneity of a country. Indeed, the UK, Belgium and Austria, the United

States, New Zealand and Australia have both steeper gradients and higher proportions of

ethnic minority pupils.

6

However, further analyses of the data show that this assumption cannot be sustained, at least

not for the UK. In the following graph (Graph 4), I arranged the high achieving countries and

Germany according to their proportions of foreign-born pupils. In addition, the proportions of

two other groups of pupils are shown; firstly, the first generation of immigrants, i.e. native-

born pupils with two foreign-born parents, and secondly, native-born pupils with one foreign-

born parent. Going by the relatively low percentages of foreign-born and first generation

pupils in the UK and Austria, we would expect gentle social gradients in these countries, but

steep gradients in Canada or Sweden.

Based on the proportion of pupils who do not speak the test language at home (Graph 5), we

would again expect a gentle social gradient in the UK. Consequently, reasons other than the

proportion of ethnic minority pupils must be responsible for the strong impact of social

factors on reading performance in the UK.

8

III.

Our knowledge of the German school system suggests another explanation – that the social

composition of UK schools might be responsible for the influence of social factors on reading

performance and explain the large performance gap between pupils from higher and lower

SES groups. In general, selective school systems lead to a high degree of social segregation of

the pupil population. A concentration of low-achieving pupils growing up in unfavourable

social conditions results in lower achievement than would be predicted on the basis of the

pupils’ individual abilities and social background. Conversely, a concentration of high

achievers growing up in favourable social conditions results in higher learning outcomes than

would be predicted on the basis of the pupils’ individual and family characteristics.

The hypothesis of composition effects in the UK schools was tested using a two-level

hierarchical linear regression model4. The dependent variable is the pupils’ reading perform-

ance; the independent variables on the pupil level are

• the pupils’ sex,

• the family type, i.e. whether or not the pupils live with both their natural parents,

• their parents’ highest socio-economic status (International Socio-Economic Index

(ISEI); Ganzeboom & Treiman 1996),

• the language spoken at home,

• the parents’ highest educational qualifications (International Standard Classification of

Education (ISCED); OECD 1999) and

• family communication about cultural topics.

The independent variables on the school level are

• the percentage of pupils whose parents belong to the lowest quartile of the social

distribution,

• the percentage of pupils whose parents did not complete upper secondary or tertiary

education,

• whether or not the school is attended by pupils who do not speak English at home

most of the time,

• the type of school, i.e. whether or not it is independent, and finally

• whether or not the school is a single sex one.

9

The first model (Table 1) takes only the pupils’ individual learning conditions into con-

sideration. The adjusted mean (i.e. the intercept) of 528 points is the performance of an

average pupil who is male and lives with both of his natural parents; his parents’ socio-

economic status is equal to the mean status of all parents of the pupil population; he speaks

English at home; his parents have no upper secondary or tertiary qualifications; the level of

cultural communication in his family is average. The values given for the variables ‘family

type’, ‘language at home’ and ‘parents’ educational level’ show the mean decrease in average

performance that would be expected if the pupil did not live with both his natural parents, if

he belonged to an ethnic minority and did not speak English at home, or if none of his parents

had completed secondary education. The other values indicate the mean increase in

performance that would be expected if the pupil was female, if the socio-economic status of

the family was higher than average, if at least one of his parents had completed upper

secondary or tertiary education and if family communication on cultural topics exceeded the

mean. More specifically, the values given for the continuous variables ‘family SES’ and

‘family communication’ indicate the average increase in the pupil’s mean reading score when

SES or cultural communication in the family is one standard deviation above – or the average

decrease in the pupil’s mean reading score when these variables are one standard deviation

below the mean. If you are surprised by the small effect of parents’ tertiary education, note

that this might be an artefact of the International Standard Classification of Education:

according to the ISCED, about 52 per cent of the parents of the PISA 2000 cohort completed

tertiary education.

The second model takes the social composition of individual schools into consideration. We

can neglect the effects of the variables on the individual level since they do not differ much

between the various models. Instead, let us concentrate on the school level and consider the

percentage of pupils whose parents belong to the lowest quartile of the social distribution.

When this percentage is one standard deviation higher than in the average school, there is a

mean decrease of 22 points in the individual pupils’ performance. If the proportion of pupils

from low SES families were one standard deviation below the mean, there would be a

performance increase of 22 points. Bearing in mind that the proportion of pupils from low

SES families may be up to three standard deviations above or below the mean, it is clear that

the schools’ impact on pupil performance can be very strong, over and above the effects of the

pupils’ individual learning conditions.

10

Table 1: Individual and school effects on reading performance in the UK (I)

Model 1 Model 2 Model 3 Model 4

Adjusted mean 527.9 (3.0) 527.8 (3.0) 527.3 (2.9) 525.5 (3.0)

Individual level

Sex: female 20.6 (2.6) 20.8 (2.6) 20.8 (2.6) 20.8 (2.6)

Family type: not two-parent family -18.2 (2.6) -18.7 (2.6) -18.8 (2.9) -18.8 (2.6)

Family SES (continuous variable) 20.0 (1.5) 18.6 (1.5) 18.6 (1.5) 18.6 (1.5)

Language at home: not English -26.5 (7.1) -24.0 (7.0) -22.3 (7.1) -23.4 (7.1)

Parents‘ education level (reference class: secondary education)

no secondary education -16.6 (4.3) -15.3 (4.3) -14.3 (4.4) -14.3 (4.4)

upper secondary education 20.4 (4.4) 19.5 (4.4) 19.4 (4.4) 19.4 (4.4)

tertiary education 2.9 (2.9) 2.4 (2.9) 2.3 (2.9) 2.3 (2.9)

Cultural communication (cont. variable) 15.1 (1.3) 14.9 (1.3) 15.0 (1.3) 14.9 (1.3)

School level

Percentage of parents: with low SES -22.2 (2.2) -18.1 (2.9) -17.7 (2.9)

with low ed. qual. -6.9 (2.5) -7.6 (2.6)

Pupils not speaking English at home 5.7 (4.4)

R2: Total variance explained 19% 22% 23% 23%

Note: Results in bold are statistically significant at the .05 level.

The third model includes the parents’ educational qualifications. Here again, the effect is not

as strong as expected, and mitigates the effect of the socio-economic composition of the pupil

population, since qualifications and socio-economic status are closely connected. While

model 1 explains only 19 per cent of the variation of the pupils’ performance, model 3 and the

following model 4 explain 23 per cent. Whether or not the school is attended by pupils from

ethnic minorities (model 4), has no significant effects on pupils’ reading performance. Some

practitioners may have doubts about this finding. They should bear in mind that we controlled

for the pupils’ socio-economic and cultural background, and that pupils from ethnic minorities

who do not speak English at home amount to less than 5 per cent of the 15-year-olds in the

UK.

11

IV.

Now the question arises as to whether these effects of the social composition of schools are

moderate or strong. To address the question, I conducted the same analyses for two other

countries, namely Finland, the country with the highest reading performance in PISA 2000,

and Germany, one of the relatively low-achieving countries. This comparative approach gives

clues as to how the results for other countries can be interpreted. I restrict my analysis to West

German schools, because I do not want to complicate the research design unnecessarily. The

division of Germany led to many differences between the two parts of the country which have

not yet ceased to exist, e.g. differences in the socio-economic distribution of the population,

the proportion of migrants, the average levels of parents’ education, the structure and size of

families and, not least, the structure of the state school systems. Politically, it is not correct to

carry out separate analyses for the two parts of Germany, but this approach helps to prevent

misinterpretations of our findings.

As the following table shows (Table 2), the type of family is less relevant for pupils’ reading

performance in West Germany than in the UK. The same holds for socio-economic status or

cultural communication in the pupil’s family. Conversely, it seems more important in Ger-

many whether the pupils speak German at home – or Turkish, or the languages of the former

Soviet republics or the former Yugoslavia, to name only the three largest immigrant groups.

The results concerning the parents’ educational levels might indicate that the International

Standard Classification of Education is not an adequate representation of the German

education system. More importantly, however, the social composition of the schools has a far

stronger impact on reading performance in West Germany than in the UK. It is well esta-

blished that the big social differences between West German schools derive mainly from

pupils being assigned to the more or less demanding schools in the three-tier secondary

school system, i.e. that the social differences are unwanted side-effects of sorting the pupils

according to their abilities and academic achievements.

12

Table 2: Individual and school effects on reading performance in Finland, the UK and West Germany

Finland UK West Germany

Adjusted mean 530.5 (3.2) 525.5 (3.0) 474.4 (3.8)

Individual level

Sex: female 49.8 (2.3) 20.8 (2.6) 16.5 (1.2)

Family type: not two-parent family -16.4 (2.8) -18.8 (2.6) -3.3 (1.4)

Family SES (continuous variable) 12.2 (1.5) 18.6 (1.5) 6.6 (0.7)

Language at home: not test language -55.2 (9.7) -23.4 (7.1) -35.7 (2.0)


no secondary education -15.9 (3.3) -14.3 (4.4) -21.7 (2.2)

upper secondary education 13.1 (3.6) 19.4 (4.4) -7.4 (1.9)

tertiary education 10.3 (3.5) 2.3 (2.9) 1.6 (1.5)

Cultural communication (continuous variable) 14.2 (1.2) 14.9 (1.3) 7.5 (0.6)

School level

Percentage of parents: with low SES -0.3 (2.6) -17.7 (2.9) -35.7 (1.9)

with low ed. qualifications -1.1 (2.2) -7.6 (2.6) -24.9 (2.1)

Pupils not speaking test language at home 0.0 (4.7) 5.7 (4.4) 3.9 (3.6)

R2: Total variance explained 21% 23% 47%


The school effects in the UK are not nearly as pronounced and they explain only an additional

4 per cent of the total variance of the pupils’ reading performance. Nevertheless, the fact that

some schools in the UK are independent and that various types of more or less selective

schools are controlled by the state and by Local Education Authorities may account for the

between-school differences in the social composition of the student body and for the schools’

stable and significant effects on pupil performance. Further differences between schools in

terms of pupil intake may be caused by residential segregation, especially socio-economic

segregation within big cities.

The situation in Finland is totally different. The effects of some variables on the individual

level are stronger than in the UK. The gender gap, for instance, is more than twice as big in

Finland as in the UK, and so is the gap between pupils speaking the test language at home and

13

those speaking another language. On the school level, however, the various aspects of the

social composition of schools do not have the slightest influence on pupils’ reading perfor-

mance. Evidently, there are only minor differences in the intakes of Finnish schools, and there

are two reasons for this:

• Firstly, compared to the UK or West Germany, the Finnish society is comparatively

homogeneous – economically and socially as well as culturally. Moreover, the catch-

ment areas of the schools bear much more resemblance to each other, since there is

little residential segregation brought about by socio-economic segregation.

• Secondly, the Finnish comprehensive school system is accepted by the population and,

in contrast to the UK system, not interspersed with selective private schools or various

types of more or less selective state schools with special programmes. As a rule, all

children attend the nearest comprehensive school, although their parents have been

granted a free choice of school.

I think these facts speak for themselves. Social segregation of secondary schools in the UK

may be one of the reasons for the relatively high impact of social factors on British pupils’

reading performance. The limited PISA data on the existence of different types of schools in

the UK confirm this assumption (Table 3). Pupils attending independent schools or single-sex

schools perform much better than would be expected on the basis of their individual and

family characteristics (models 5 and 6). Since independent schools are often single sex

schools, their effects are smaller when both variables are included in the model (model 7) and

the probability of error rises to p = .07. It seems reasonable to assume that city technology

colleges, grant maintained schools, beacon schools, specialist schools, city academies and so

on have similar positive effects on pupil performance, and that the mere existence of such

selective schools contributes to the development of parallel schools for the less privileged

proportion of the pupil population and of schools with a high concentration of socially

disadvantaged, low-achieving pupils.

14

Table 3: Individual and school effects on reading performance in the UK (II)


Adjusted mean 525.5 (3.0) 525.0 (3.0) 522.4 (2.9) 522.4 (2.9)

Individual level

Sex: female 20.8 (2.6) 21.0 (2.6) 20.2 (2.5) 20.4 (2.5)



Language at home: not English -23.4 (7.1) -23.4 (7.1) -23.0 (7.2) -23.0 (7.2)



upper secondary education 19.4 (4.4) 19.2 (4.4) 19.3 (4.4) 19.2 (4.4)

tertiary education 2.3 (2.9) 2.1 (2.9) 2.3 (2.8) 2.2 (2.9)

Cultural communication (continuous variable) 14.9 (1.3) 14.8 (1.3) 14.8 (1.3) 14.7 (1.3)

School level

Percentage of parents: with low SES -17.7 (2.9) -15.0 (2.7) -14.8 (2.8) -13.4 (2.8)

with low educ. qualifications -7.6 (2.6) -7.4 (2.5) -9.4 (2.7) -9.1 (2.6)

School with pupils not speaking English at home 5.7 (4.4) 1.0 (4.0) 0.2 (3.9) -2.2 (4.0)

School type: independent school 29.3 (10.6) 17.7 (9.8)

School type: single-sex school 36.2 (5.8) 33.2 (5.9)



The situation in West Germany may be a warning (Table 4). You have already seen the first

model. When we include the various types of schools (model 2), nearly all effects of the

social composition of the pupil populations disappear, i.e. the social segregation of the

schools is a side-effect of processes of selection based on pupils’ abilities and achievements.

These prerequisites for successful learning are used to justify the allocation of pupils to

different types of schools. Accordingly, I included pupils’ cognitive abilities into the third

model. (This variable is not included in the international data.) The results show that the

genuine effects of the school types are much smaller than before but that they do not dis-

appear. Given that the proportions of the age cohort attending the different types of secondary

schools vary across the eleven West German states (Länder), I finally included the pupils’

mean cognitive abilities in the model (model 4). It emerges that the school types have effects

15

on pupil performance which could not be predicted by the pupils’ cognitive abilities or family

background. Comparing the effect sizes of the different school types shows what can happen

when so-called comprehensive schools have to compete with selective schools. The West

German comprehensive schools were established in addition to the schools of the three-tier

system but failed to attain the same prestige as either the academic-track schools (Gymnasien)

or the intermediate-track schools (Realschulen). In fact, they soon became the better alter-

native to the basic-track schools (Hauptschulen) and, with a few exceptions, their intake is not

at all representative of the age cohort.

Table 4: Individual and school effects on reading performance in West Germany


Adjusted mean 478.0 (2.5) 486.5 (2.7) 486.2 (1.8) 486.1 (2.1)

Individual level

Cognitive Abilities (continuous variable) 51.0 (0.5) 50.3 (0.8)

Sex: female 16.2 (1.2) 16.0 (1.2) 15.8 (0.7) 15.8 (1.0)



Language at home: not German -36.1 (2.1) -36.2 (2.1) -22.9 (1.2) -23.1 (1.8)

Parents’ education level (reference class: secondary education)


upper secondary or tertiary education -1.0 (1.4) -1.7 (1.4) -2.6 (0.9) -2.6 (1.2)

Cultural communication (continuous variable) 7.6 (0.6) 7.4 (0.6) 5.0 (0.4) 5.0 (0.5)

School level

Percentage of parents: with low SES -36.3 (2.0) -1.5 (1.6) -0.3 (1.2) -0.3 (1.2)

Percentage of unqualified parents -27.9 (2.5) -12.6 (1.7) -9.8 (1.3) -8.5 (1.2)

Percentage of pupils not speaking German at home 5.2 (2.5) -3.2 (1.7) -0.4 (1.1) 1.1 (1.2)

Mean cognitive abilities 12.2 (1.9)

School type (reference class: medium track: i.e. Realschule)

basic track (Hauptschule) -78.0 (3.3) -42.9 (2.3) -31.5 (3.0)

academic track (Gymnasium) 60.9 (2.4) 28.6 (2.1) 17.9 (2.6)

comprehensive track (Gesamtschule) -22.9 (3.1) -14.1 (2.3) -11.3 (2.2)



16

V.

Based on our knowledge of West German schooling, I feel confident in stating that the

performance gap between high and low SES pupils is not independent of the structure of the

school system. I can imagine that the situation in the UK is not so very different from that in

West Germany. But I would never go so far as to assert that comprehensive schooling in

Finland is the one and only reason for the gentle social gradient and high average score of the

Finnish pupils. The PISA data do not tell us the whole story about learning in the various

countries because the OECD fails to consider cultural influences on learning and schooling.

I would like to illustrate my point by telling you something about the development of reading

literacy in Finland, drawn from a report on schooling in Finland published by Linnakylä

(2003). In his country, reading literacy became widespread after the Reformation, since “the

Protestant doctrine required that everyone should be able to read the Holy Bible by them-

selves.” “From the 16th century, literacy was a prerequisite for receiving the sacraments and

contracting a Christian marriage” and “the church took responsibility for the teaching of

literacy.” The parents and, later, the parish clerk instructed the children, whose “reading

literacy was assessed at annual parish meetings” and “entered in the parish register”. In 1866,

public schools were instituted. “One of the conditions for acceptance to public school was,

however, the ability to read fluently, thus, the initial teaching of reading still remained as the

task of the church and the home […].” After Finland had become independent, compulsory

education for the whole population was established in 1921. “Nevertheless, the traditional

view prevailed; i.e. that it was the parents’ task to teach the basics of reading literacy. It was

actually considered a misuse of schools for parents to send illiterate children to school”

(Linnakylä 2003, p. 150 – 151).

Coming back to my conclusions and summing them up, I would like to cite Pam Sammons

and her co-workers again, who pointed out that the mere existence of comprehensive

schooling “is unlikely to account for the improvement of educational standards […] nor for

the better performance of English pupils in the international PISA 2000 comparisons […].

Rather the sustained emphasis on standards of performance and policy priority accorded to

education is likely to have had a greater impact. In particular the PISA 2000 cohort’s

performance is likely to have been significantly influenced by the introduction of a national

curriculum, national assessments, and emphasis on core subjects” (Sammons et al. 2003,

17

p. 138). “The findings […] suggest that there has been a steady increase in attainment levels

across the board but indicate that the system-wide changes of the 1988 – 1996 era (involving

the application of market forces to education) have not reduced the strong social gradient in

attainment” (l.c., p. 139).

The authors seem to take comfort from the finding that “in England low SES pupils still attain

relatively well in comparison with similar SES pupils in many other countries,” in other

words, that “the higher SES group is largely responsible for the extent of the SES equity gap

in attainment in England” (l.c., p. 139). According to my own analyses, the gap should be

considered more alarming. I agree with Sammons and her co-authors that comprehensive

schooling is not a sufficient condition for high pupil performance. Nevertheless, I can imagine

that avoiding the establishment of selective schools could improve the learning conditions of

low SES pupils, in other words, that they would benefit from the comprehensive school

system being maintained more consistently. Corresponding efforts would help increase the

equality of educational opportunities, an aspiration which contemporary democracies can

hardly renounce.

Epilogue

Perhaps some of you expected to hear more policy-related ideas from a PISA expert.

Therefore I would like to add some remarks on the design of the study and on the limited

nature of the insights it provides into the factors influencing the variation of achievement

across the participating countries.

As PISA is a cross-sectional study, we cannot draw any conclusions about causal

relationships between learning contexts and learning outcomes in the various countries. The

correlations identified in PISA cannot be interpreted as causal connections unless we have a

profound theory explaining the reasons for the international variation in pupil achievement.

No such theory has yet been proposed and developing one will not be easy. Although

hundreds of thousands of pupils have been assessed in comparative international studies, these

studies are very small considering that countries are the units of investigation. The Third

International Mathematics and Science Study (TIMSS) was conducted in 42 countries, PISA

2000 in 32 countries, the Progress in International Reading Literacy Study (PIRLS) in 35

countries. Higher participation rates would simply imply higher numbers of developing

18

countries, and result in nothing but confirmation of the oft-replicated finding that pupils in

industrial countries outperform their contemporaries in developing countries.

If we want to gain insights into the significance of various educational practices for the

development of high educational achievement, we are more or less confined to studying the

30 or so industrial states belonging to the OECD. Such a small number of research units is not

sufficient for conducting multivariate analyses and bivariate analyses are unsatisfactory.

Dependent on the social and cultural contexts, comparable features of education systems may

have different effects on pupil achievement, while different features may fulfil equivalent

functions. Jointly, the results of bivariate analyses demonstrate only “that there is no single

factor that explains why […] some countries have better results than others. Successful per-

formance is attributable to a constellation of factors” as the OECD states in the first PISA

report (OECD 2001, p. 212). Thirty countries are certainly not enough to test the effects of the

systematically varied constellations of factors characterising their school systems. Conse-

quently, we are restricted to the persuasiveness of interpretations.

The information yielded by PISA 2000 is stimulating but not sufficient for understanding the

international variation of the PISA results. That is why swarms of German politicians and

journalists travelled to Scandinavia to find out for themselves the secret of the Finnish and

Swedish pupils’ success. The educational tourists’ travelling impressions were rather

disappointing for us. The same applies to the first country reports, which appeared after the

publication of the PISA results. Many insights into different education systems were

provided, but no convincing reasons for their learning outcomes. Consequently, colleagues of

mine from the German Institute for International Educational Research (DIPF) commissioned

expert reports on the education systems of six successful countries in order to gain better

insights into the reasons for the sub-standard outcomes of the German schools (Döbert et al.

2003). This survey of the socio-cultural contexts of education in the selected countries, their

school systems, and the functioning of individual schools concluded that many differences

exist between the countries, but that there is no common pattern of factors contributing to

their success (Bundesministerium 2003).

Notes: 1. The OECD’s comparative analyses refer to the United Kingdom although Wales did not

participate in PISA 2000.

19

2. All results are taken from the OECD’s PISA 2000 report (OECD 2001) unless another source is named.

3. Graph 1 and graph 2 were adopted from the OECD’s powerpoint presentation of the first

PISA 2000 results. 4. The analyses are based on the international PISA 2000 data for the United Kingdom

(without Wales) and Finland, and on the West German data of the extended German PISA 2000 data set. Pupils with incomplete data sets and schools with less than 10 participants were excluded from the analyses. The final data sets consist of 7810 pupils from 338 schools in the United Kingdom, 4528 pupils from 153 schools in Finland, and 20987 pupils from 909 schools in West Germany. - The two-level analyses were conducted on the assumption that the variables on the school level have fixed effects while the variables on the individual level have random effects. The independent continuous variables were z-standardised. Both z-values and categorical data were added uncentered into the equations.

References: Bundesministerium für Bildung und Forschung (Ed.): Vertiefender Vergleich der Schul-systeme ausgewählter PISA-Staaten. Bonn 2003. www.bmbf.de Deutsches PISA-Konsortium (Ed.): PISA 2000. Basiskompetenzen von Schülerinnen und Schülern im internationalen Vergleich. Opladen 2001. Döbert, H., Klieme, E. & Sroka, W. (Eds.): Conditions of School Performance in Seven Countries. A Quest for Understanding the International Variation of PISA Results. Münster et al. 2003. Ganzeboom, H. B. G. & Treiman, D. J. : International Comparable Measures of Occupational Status for the 1988 International Standard Classification of Occupations. In: Social Science Research 1996 (201-239). Gill, B., Dunn, M. & Goddard, E.: Student achievement in England. London 2002. www.statistics.gov.uk Linnakylä, P.: Finland. In: Döbert, H., Klieme, E. & Sroka, W. (Eds.): Conditions of School Performance in Seven Countries. A Quest for Understanding the International Variation of PISA Results. Münster et al. 2003 (150-218). OECD: Classifying Educational Programmes. Manual for ISCED-97 Implementation in OECD Countries. Paris 1999. OECD: Knowledge and Skills for Life. First Results from PISA 2000. Paris 2001. www.oecd.org Sammons, P., Elliot, K., Welcomme, W., Taggart, B. & Levacic, R.: England. In: Döbert, H., Klieme, E. & Sroka, W. (Eds.): Conditions of School Performance in Seven Countries. A Quest for Understanding the International Variation of PISA Results. Münster et al. 2003 (65-149).

What can we learn from international comparisons? PISA ......Gundel Schümer, Max Planck Institute...

Documents

Transcript of What can we learn from international comparisons? PISA ......Gundel Schümer, Max Planck Institute...