Comparing

Indonesian Performance Compared to other Countries 1

INDONESIAN PERFORMANCE COMPARED TO OTHER COUNTRIES

Introduction

The results of the Third International Mathematics and Science Studies (TIMSS)

undertaken by the IEA indicate that among the participating countries, Singapore

outperforms all other countries in both Mathematics and Science, followed by

Korea, Japan, Hong Kong, and Belgium (Fl) in Mathematics and Czech

Republic, Japan, Korea and Bulgaria in Science. At the other end of the scale,

South African Junior Secondary School students are the lowest achievers in both

Mathematics and Science (Lokan et al., 1996).

The purpose of this article is to report the results of the analysis of the

Indonesian Population 2 data in order to get some information on the

performance in the Indonesian Junior Secondary School students in Mathematics

and Science compared to other countries participating in the TIMSS. The data

used for this comparison is the Grade 8 student responses to the mathematics and

science achievement tests. For this purpose, it is necessary to put the Indonesian

data on the same scale as that of the TIMSS in order to get comparable scores.

The procedure of placing two or more tests onto the same scale is known as

equating (Jaeger, 1981; Petersen et al., 1989; Cook and Eignor, 1991).

It has been stated previously that Indonesia was not included in the main

reports of the TIMSS because of problems that were encountered in the data

collection. Eng (1996) has reported that the exclusion of Indonesia and four other

countries (Argentina, Italy, Mexico, and the Philippines) in the report were

because Indonesia and the other four countries did not “complete the steps

necessary for their data to appear in the report (p. 4). However, in 1996,

Indonesia readministered the Mathematics and Science tests employing the same

instruments which had been used earlier so that data are available for analysis.

The TIMSS reports and the reports of its predecessors have been

discussed quite extensively by many researchers. Some researchers extend their

Ramon Mohandas


support to such studies while some others question the significance of the studies.

Stedman (1997) has raised questions about how large the achievement

differences were among the countries participating in TIMSS. He argued that

when the scores of students in different countries were ranked, the position of a

country in the rank assigned might differ greatly in the rank compared with the

rank assigned to a particular country even though in fact their scores might be

relatively close together. For this reason, he suggested that the use of ranking

might give misleading impression on the performance of a particular country.

Nevertheless, Baker (1997, p. 16) claimed that the ranking of participating

countries in an international study such as the one conducted by TIMSS was

tempting and might be “useful in a limited way”. However, he cautioned that this

ranking “should not be the main focus, particularly among researchers”.

Method of the Equating

In order to place the Indonesian data on the same scale as that of the TIMSS

reports, and hence to compare the achievement of the Indonesian Junior

Secondary School students with other students at the same level in the

participating countries, the Indonesian data have to be equated with the data of

some other participating countries. Fortunately, the data of one of the

participating countries, namely Australia, were accessible for this purpose.

Several studies have shown that concurrent equating for horizontal

equating is better than the anchor item equating (Mohandas, 1996). For this

reason, concurrent equating in this study is employed to locate Indonesian and

Australian data on the same scale. In the concurrent equating, the data for the

student responses from the different forms to be equated are combined together

to form a large data file, provided that there are common items available in each

form. In this study, there were eight forms of both the mathematics and science

tests administered to the Junior Secondary School students at Grade 8 in

Indonesia and the same tests with the same number of forms were also

administered to Year 8 students in Australia.

Ramon Mohandas


The report of the analysis in this chapter is divided into three parts. The

first part involves comparing the mathematics achievement of the Grade 8

students in Indonesia with other countries participating in the TIMSS. The

second part involves comparing the science achievement of the Grade 8 students

in Indonesia with other countries participating in TIMSS, and the third part

provides the conclusion.

The concurrent equating of the Indonesian data and the Australian data

was undertaken by first combining the two data files to become a single large

data file. In the Indonesian data file, there were 3929 Grade 8 students, while in

the Australian data file, there were 7926 Year 8 students. The mathematics and

science tests each consisted of eight different forms, with some items being

common across the different forms (the details of the test structure was presented

in Chapter 4). Since the Indonesian data contained only student’s responses from

the multiple-choice items, the concurrent equating procedure reported in this

study analyses only the student’s responses to the multiple-choice questions in

the Indonesian data as well as in the Australian data.

It has been described earlier that the mathematics and science tests each

consisted of eight different forms, with several common items across the

different forms. The structures of the mathematics and science tests were also

presented in that chapter. The mathematics and science tests used in Indonesia

were basically the same as those used in Australia, although there was the

necessary difference in the language used in the tests. Consequently, the correct

response codes given to each item were the same for the Indonesian and the

Australian data. However, for some items in the Indonesia data in this study, the

number of alternatives for certain multiple-choice items was changed from their

original five options to four options. For this reason, these items had to be treated

as different items in the Indonesian data file and the Australian data file. There

were 15 items which were treated as different items because of this problem,

consisting of 12 mathematics items (a02, b10, e01, e05, g06, h08, j14, m02, p17,

q01, q09, and r09) and three science items (l01, n01, and q14). While these items

Ramon Mohandas


were excluded from calibration and equating, they were not excluded in the

scoring.

As has been described earlier, there were also items in Indonesia which

had to be deleted from the analysis because of translation and test construction

problems. There were 12 items with this these problems, consisting of eight

mathematics items (a07, d10, f10, j11, j15, k08, l14, and p15) and four science

items (a12, f06, j02, and p04). In addition to these 12 items, two items had to be

deleted from further analyses because the preliminary analysis of the Indonesian

data showed that these two items had very poor item statistics. These items were

mathematics items, namely item f08 with a point biserial value of 0.07 and item

o04 with a point biserial value of -0.01. Consequently, there were 14 items

deleted for the analyses of the Indonesian data. In the combined data file,

consequently, the Indonesian student’s responses for these items were treated as

blank so that they could be ignored in the analysis.

Comparing Mathematics Achievement

In this section, the comparison of the Indonesian student achievement in

mathematics with that in other countries participating in the TIMSS is reported in

three parts. The first part discusses the results of the concurrent equating

procedure, the second part discusses the procedures employed to locate the

results of the equating onto the same scales as the ones presented in the TIMSS

reports issued by IEA, and the third part presents the comparison of the

Indonesian student achievement in mathematics with other countries

participating in TIMSS.

The results of the concurrent equating in the first part are required to

obtain the case estimates of the Australian data by State using the Rasch scale.

The second part, presenting the data of the Australian Year 8 student

achievement in mathematics by states as reported by Lokan et al. (1996), is

required to put the scales of the resulting equating procedure onto the same scale

as the scales used in the TIMSS report. By obtaining the Australian data by State

Ramon Mohandas


in the form of the Rasch scale from the equating procedure and the scale used in

the TIMSS report, the correlation coefficient between these two scales could be

obtained, together with their corresponding standard deviation values.

Subsequently, the regression equation could be used to calculate the equivalent

value of the Indonesian performance similar to that used in the TIMSS report,

which records the performances of other countries that participated in the study.

Results of the Concurrent Equating Procedure for Mathematics

The concurrent equating of the Indonesian and the Australian data was

undertaken using the QUEST program (Adams and Khoo, 1993). The results of

the analysis indicate that five mathematics items had to be deleted from the

analysis because their infit mean square values were greater than the acceptable

values, which ranged from 0.77 to 1.30 as described by Adams and Khoo (1993).

The results of the final run of the mathematics item estimates are presented in

Appendix 12.2.

The QUEST output that is required to obtain the scores of each individual

student in Indonesia as well as in Australia is the output from the show case

command line. However, the output of this command line does not provide the

scores for students who answered all questions correctly (perfect scores) or who

answered all questions incorrectly (zero scores). Consequently, these perfect and

zero scores have to be calculated manually before further analysis takes place.

For this purpose, individual students in the Indonesian and Australian data had to

be identified and grouped according to the test booklet they worked on so that

each group would consist of students who attempted exactly the same questions.

The calculation of the perfect scores for each group of students who took the

same test booklets is carried out by taking into consideration the first three

highest scores for the groups of students who made one, two, and three errors

respectively. Similarly, the calculation of the zero scores is carried out by taking

into consideration the last three lowest scores for the corresponding group of

students who obtained one, two, and three items correct respectively. For

Ramon Mohandas


example, if the first three highest scores in the particular group of students are

3.42, 2.68, and 2.22, the perfect score is:

3.42 + (3.42 - 2.68) + {(3.42 - 2.68) - (2.68 - 2.22)} = 4.44.

If the last three lowest scores in the particular group of students are -2.56, -2.21,

and -1.92, the zero score is:

-2.56 + (-2.56 - -2.21) + {(-2.56 - -2.21) - (-2.21 - -1.92)} = -2.97.

The scores for all individual students in the Indonesian as well as in the

Australian data, after calculating the perfect and the zero scores, are then

transferred into the SPSS program to calculate the mean score for Indonesia and

the mean scores for each State in Australia after taking into account the weights

assigned to the Indonesian and the Australian data. The procedure used in the

calculation of the weights for the Indonesian and the Australian data is the same

as the procedure discussed in Chapter 11.

Table 12.1 presents the mean scores of the Australian states in Rasch

scores (logit) based on the results obtained from the SPSS calculations. The

figures in the last column are the mean scores of the Australian States using the

scale presented in the TIMSS reports, as given by Lokan et al. (1996). The mean

scores reported here are taken for Year 8 students in the States rather than by

using upper grade and lower grade categories employed in the report. In other

words, for Australian Capital Territory, New South Wales, Victoria, and

Tasmania, the mean scores taken from Table A.6: Mathematics Achievement by

State, Upper Grade in Lokan et al. (1996), while for Western Australia, South

Australia, Queensland, and Northern Territory, the mean scores are taken from

Table A.7: Mathematics Achievement by State, Lower Grade. This decision was

made because of the lack of equivalence in the upper and lower grades in the

school systems in the States concerned. Furthermore, the Australian data

analysed

Table 12.1: Mathematics Mean Scores for Australian States in Logit and TIMSS

Ramon Mohandas


State Logit TIMSSACT 0.800 548NSW 0.775 522VIC 0.503 511QLD 0.602 512SA 0.624 516WA 0.861 532TAS 0.445 501NT 0.352 478Mean 0.620 515Standard Deviation 0.182 21Correlation 0.910

in this study consisted of Year 8 students, so that by making this decision, the

year level of students in the data would match the year level of students in the

table. In this report, the scale units of the scores derived from this equating

procedures are referred to as logits, while the scores obtained from the TIMSS

report are referred to as TIMSS units. The Logit scale is a scale with meaningful

units, while the units of the scale employed in the TIMSS report are not clearly

specified in a way that would facilitate equating, in either the Australian national

or the international reports.

From these two sets of data (Logit and TIMSS), the correlation coefficient

between the two sets can be calculated, together with their Means and Standard

Deviations to permit the use of the regression equation.

Based on these figures, the next section discuss the use of the regression

equation to calculate the estimates for the Indonesian data using the TIMSS

scale.

Locating the Indonesian Mathematics Scores to the TIMSS Scale

In this section, a regression equation is used to predict the Indonesian scores on

the TIMSS scale so that they can be compared to the results presented in the

reports on mathematics achievement in the TIMSS. Guilford and Fruchter (1981,

Ramon Mohandas


p. 346) have argued that the main use of a regression equation is “to predict the

most likely measurement in one variable from the known measurement in

another”. In this procedure, the accuracy of the prediction is determined by the

correlation coefficient between the two known measures. As Guilford and

Fruchter (1981, p. 346) put it, “the higher the correlation, the greater the accuracy

of prediction and the smaller the errors of prediction”.

Hinkle et al. (1988, p. 128) suggest that the procedure of prediction

involves two steps, namely (1) determining the regression line as a mathematical

equation, and (2) using the mathematical equation to predict scores. The

mathematical equation to determine the regression line is: $Y = bX + a Equation 12.1

where $Y = the predicted score

b = the slope for the line

a = the intercept.

Hinkle et al. (1988) further show that the slope for the line (b) is also

called the regression coefficient, and is calculated using the following formula:

b = (r)SySx

Equation 12.2

where

r = the correlation between variables X and Y

Sy = standard deviation of the Y scores

Sx = standard deviation of the X scores.

The intercept of the regression line (a), is also called the regression constant, and

is calculated using the following formula:

a = Y - b X Equation 12.3

where

Y = the mean score of Y

Ramon Mohandas


b = the slope of the regression coefficient

X = the mean score of X.

By using the three formulae presented above, it is possible to calculate the

predicted value of the new scores for each state in Australia, and at the same

time, using the same formulae, to calculate the predicted value of the new score

for Indonesia. This can be done because as a result of the concurrent equating,

the scores of the Indonesian students are already in the same logit scale as the

scores of the Australian students. Hence, the new predicted scores of the

Indonesian students on the TIMSS scale are also on the same scale as the

Australian scores, which also means they are located on the same scale as the

other countries participating in the TIMSS.

Table 12.2 presents the calculation of the predicted scores for the

Australian data based on the information provided in Table 12.1 and using the

three formulae described above. Since there are two groups of scores given in

Table 12.1 (Logit and TIMSS), the correlation coefficient (r) of the two groups

and the standard deviation of each group (Sx and Sy) can be calculated.

Consequently, the slope (b) and the intercept (a) of the regression line can also be

calculated.

The results in Table 12.2 show that the correlation coefficient (r) between

the Logit scores (X) and the TIMSS scores (Y) is 0.91, which indicates a strong

correlation. The mean of the resulting predicted scores is the same as the mean of

the TIMSS score, which is 515. Based on these results, it can be argued that the

calculation of the Indonesian score can be undertaken with relatively high

accuracy.

The correlation between the two sets of scores for the Australian States is

not perfect (ie equal to unity as might be expected) for the following reasons.

1. Some items were excluded from consideration in the calculation of the logit

scale, although this should not lead to scale differences.

Ramon Mohandas


Table 12.2: Calculation of Predicted Scores for Australia in Mathematics Achievement

STATES Logit

(X) TIMSS

(Y) PREDICTED

SCORES ACT 0.800 548 $Y bX a= + = 534 NSW 0.775 522 $Y bX a= + = 531 VIC 0.503 511 $Y bX a= + = 503 QLD 0.602 512 $Y bX a= + = 513 SA 0.624 516 $Y bX a= + = 515 WA 0.861 532 $Y bX a= + = 540 TAS 0.445 501 $Y bX a= + = 497 NT 0.352 478 $Y bX a= + = 487

Mean 0.620 515 515

r 0.910 Sy 20.764 Sx 0.182 b 103.964 a 450.516 Indonesia -0.510 $Y bX a= + = 397

2. The weights assigned in the calculation of the logit scale scores did not

employ the procedures used in the TIMSS which were considered to have

involved errors.

3. The numbers of cases on the two data files obtained for the Australian data did

not correspond, nor did they correspond with the numbers of cases reported

for the Australian TIMSS (Lokan et al., 1997).

Nevertheless, these shortcomings in the Australian analyses are not likely to have

led to gross errors in the estimation of the Indonesian scores, and are probably no

greater than the unknown bias introduced into the Australian data by the

relatively low level of response. Likewise while items were excluded from the

calibration, equating, and scoring of the Indonesian data, the Rasch scaling

procedure makes allowance for these exclusions to produce equivalence to the

Ramon Mohandas


scores obtained for the Australian States through the concurrent equating

procedure employed.

From the results of calculation of the mean score for Indonesia undertaken

in the same way as the calculation of the Australian data, the mean score for

Indonesia in mathematics is -0.51. By applying the mathematical equation used

for the Australian data to calculate the predicted score, the predicted score for

Indonesia is shown at the foot of Table 12.2, which is 397. This figure is taken as

the score of the Indonesian students at Grade 8 in mathematics, which is on the

same scale as the scores presented in the TIMSS reports.

The next section discusses the mathematics achievement of the Indonesian

students at Grade 8 by comparing the scores obtained from the above

calculations to the scores of other countries participating in TIMSS.

Comparing Indonesian Student Achievement in Mathematics with Other Countries Participating in TIMSS

The comparison of the Indonesian students’ achievement in mathematics with

other countries participating in TIMSS is undertaken by using the predicted score

as shown in Table 12.2, which is 397.

Table 12.3 shows the position of Indonesia among other countries

participating in the TIMSS. It is seen in the table that Indonesia is ranked number

39 out of 42 countries in the table. The three countries below Indonesia are

Kuwait, Colombia, and South Africa with scores of 392, 385, and 354

respectively. The four top positions are all held by Asian countries, namely

Singapore, Korea, Japan, and Hong Kong with the scores of 643, 607, 605, and

588 respectively. Australia, a country used as a base to put Indonesia onto the

same scale as the one reported in TIMSS, is ranked number 16 with a score of

539.

Another Asian country, Thailand, is ranked number 20 with a score of

522. However, Beaton et al. (1996b) report that the difference in the mathematics

achievement of students in Australia and Thailand is not significant, in spite of

Ramon Mohandas


Table 12.3: Comparison of Mathematics Achievement at Year 8 among Countries Participating in the TIMSSa

No. Country Mean Standard Error1 Singapore 643 4.92 Korea 607 2.43 Japan 605 1.94 Hong Kong 588 6.55 Belgium (Fl) 565 5.76 Czech Republic 564 4.97 Slovak Republic 547 3.38 Switzerland 545 2.89 Netherlands 541 6.710 Slovenia 541 3.111 Bulgaria 540 6.312 Austria 539 3.013 France 538 2.914 Hungary 537 3.215 Russian Federation 535 5.316 Australia 530 4.017 Ireland 527 5.118 Canada 527 2.419 Belgium (Fr) 526 3.420 Thailand 522 5.721 Israel 522 6.222 Sweden 519 3.023 Germany 509 4.524 New Zealand 508 4.525 England 506 2.626 Norway 503 2.227 Denmark 502 2.828 United States 500 4.629 Scotland 498 5.530 Latvia (LSS) 493 3.131 Spain 487 2.032 Iceland 487 4.533 Greece 484 3.134 Romania 482 4.035 Lithuania 477 3.536 Cyprus 474 1.937 Portugal 454 2.538 Iran 428 2.239 Indonesia 397 b40 Kuwait 392 2.541 Colombia 385 3.442 South Africa 354 4.4 a Adapted from: Beaton, A.E., et al., 1996b. Mathematics Achievement in the Middle School Years, p. 22 b

No estimate could be made for the standard error of the score for Indonesia.

Ramon Mohandas


the gap in the ranking position (16 and 20). The mean achievement of the top

performer, Singapore, is significantly higher than the mean achievement of any

other countries listed in the table. Furthermore, the mean achievement of the

country at the bottom, South Africa, is significantly lower than the mean

achievement of any other countries listed in the table.

Lokan et al. (1996) report that the scale used in the TIMSS reports is

based on item response theory (IRT) and the scale has a mean of 500 and a

standard deviation of 100. It should be noted here that while the scale mean is

presumably the average difficulty level of the mathematics test items, the

reference sample to obtain the standard deviation is not reported, nor is a reason

given for not using the natural unit of the Rasch scale, the logit. Several sub-

studies in the TIMSS project indicate that 30 scale points of mathematics

achievement reported in the TIMSS is equivalent to a year’s growth in

mathematics achievement at the middle secondary school stage (Stedman, 1997;

Beaton et al., 1996b). If this is the case, then, based on the figures in Table 12.3,

Indonesia is lagging four years behind Thailand and Australia, and eight years

behind Singapore, which is clearly several years of mathematics learning ahead

of Australia.

This results raise concerns about the performance of the Indonesian

students in mathematics at the Junior Secondary School level.

Comparing Science Achievement

In this section, the comparison of Indonesian student achievement in science with

other countries participating in the TIMSS, as in the section for mathematics, is

reported in three parts. The first part discusses the results of the concurrent

equating procedure, the second part discusses the procedures employed to place

the results of the equating onto the same scales as the ones presented in the

TIMSS reports issued by IEA, and the third part presents the comparison of the

Ramon Mohandas


Indonesian student achievement in science with other countries participating in

TIMSS.

The results of the concurrent equating in the first part are required to

obtain the case estimates of the Australian data in science by State using the

Rasch scale. The second part, presenting the data of the Australian Year 8 student

achievement in science by states as reported by Lokan et al. (1996), is required to

locate the scales of the resulting equating procedure onto the same scale as the

scales used in the TIMSS report. By obtaining the Australian data by State in the

form of the Rasch scale from the equating procedure and the scale used in the

TIMSS report, the correlation coefficient between these two scales could be

obtained, together with their corresponding standard deviation values.

Subsequently, the regression equation could be used to calculate the equivalent

value of the Indonesian performance on the scale that is similar to that used in the

TIMSS report.

Results of the Concurrent Equating Procedure for Science

The concurrent equating of the Indonesian and the Australian data for science

was undertaken using the QUEST program (Adams and Khoo, 1993). The results

of the analysis indicate that for the science items, there were no items that had to

be deleted from the analysis because the infit mean square values for all items

were within the acceptable range, from 0.77 to 1.30, as described by Adams and

Khoo (1993). The results of the final run of science item estimates are presented

in Appendix 12.3.

The QUEST output that is required to obtain the science scores of each

individual student in Indonesia as well as in Australia is the output from the show

case command line. However, the output of this command line does not provide

the scores for students who answered all questions correctly (perfect scores) or

for those who answered all questions incorrectly (zero scores). Consequently,

these perfect and zero scores have to be calculated manually before further

analysis takes place. For this purpose, individual students in the Indonesian and

Ramon Mohandas


Australian data had to be identified and grouped according to the test booklet

they took so that each group would consist of students who attempted exactly the

same questions. The procedures employed in the calculation of the perfect scores

and the zero scores are the same as those undertaken for mathematics.

The science scores for all individual students in the Indonesian as well as

in the Australian data, after calculating the perfect and the zero scores, are then

transferred to the SPSS program to calculate the mean score for Indonesia and

the mean scores for each State in Australia by taking into account the weights

given for the Indonesian and the Australian data as presented in mathematics.

Table 12.4 gives the science mean scores of the Australian States in Rasch

score (logit) units based on the results from the SPSS calculation. The figures on

the last column are the mean scores of the Australian States using the scale

presented in the TIMSS report, as given by Lokan et al. (1996). The mean scores

reported here are taken for Year 8 students in the States rather than by using the

upper grade and lower grade categories employed in the TIMSS reports. In other

words, for Australian Capital Territory, New South Wales, Victoria, and

Tasmania, the mean scores are taken from Table A.8: Science Achievement by

State, Upper Grade in Lokan et al. (1996), while for Western Australia, South

Australia, Queensland, and Northern Territory, the mean scores are taken from

Table A.9: Science Achievement by State, Lower Grade. This decision is made

because of the differences in the school systems in the States concerned.

Furthermore, the Australian data analysed in this study consisted of Year 8

students, so that by making this decision, the year level of students in the data

would match the year level of students in the table. In this report, the scores

derived from the equating procedures are referred to as Logit, while the scores

obtained from the TIMSS report are referred to as TIMSS.

From these two sets of data (Logit and TIMSS), the correlation coefficient

between the two types of figures is calculated, together with their standard

deviation.

The next section discusses the use of the regression equation to calculate

the estimates for the Indonesian data using the TIMSS scale.

Ramon Mohandas


Table 12.4: Science Mean Scores for Australian States in Logit and TIMSS

State Logit TIMSSACT 0.793 564NSW 0.662 538VIC 0.448 521QLD 0.503 519SA 0.611 531WA 0.724 545TAS 0.528 525NT 0.510 504Mean 0.597 531Standard Deviation 0.121 18Correlation 0.910

Locating Indonesian Science Scores to the TIMSS Scale

In this section, the regression equation is used to predict the Indonesian scores in

the TIMSS scale so that they can be compared to the results in the reports on

science achievement in the TIMSS. The mathematical equations used to

determine the regression line are the same as the ones presented for mathematics,

namely Equations 12.1, 12.2, and 12.3.

By using the three formulae given earlier, it is possible to calculate the

predicted value of the new science scores for each State in Australia, and at the

same time, using the same formulae, to calculate the predicted value of the new

science score for Indonesia. This can be done because as a result of the

concurrent equating, the science scores of the Indonesian students are already on

the same scale as the science scores for the Australian students. Hence, the new

predicted science scores for the Indonesian students on the TIMSS scale are also

on the same scale as the Australian scores, as well as that for other countries

participating in the TIMSS.

Ramon Mohandas


Table 12.5 presents the calculation of the predicted science scores for the

Australian data based on the information provided in Table 12.4 and using the

three formulae given earlier. Since there are two groups of scores given in Table

12.4 (Logit and TIMSS), the correlation coefficient (r) of the two groups and the

standard deviation of each group (Sx and Sy) can be calculated. Consequently,

the slope (b) and the intercept (a) of the regression line can also be calculated.

The results in Table 12.5 show that the correlation coefficient (r) between

the Logit scores (X) and the TIMSS scores (Y) is 0.91, which indicates a strong

correlation. The mean of the resulting predicted scores is the same as the mean of

the TIMSS score. From these results, it can be argued that the calculation of the

estimated Indonesian mean score can be undertaken with a relatively high degree

of accuracy.

From the results of calculation of the mean score for Indonesia undertaken

in the same way as the calculation of the Australian data, the mean score for

Indonesia in science is -0.18. By applying the regression equation used with the

Australian data to calculate the predicted score, the predicted score for Indonesia

is shown at the foot of Table 12.5, which is 424. This figure is taken as the mean

score for the Indonesian student at Grade 8 in science, which is on the same scale

as presented in the TIMSS reports.

The next section discusses the mean science achievement of the

Indonesian students at Grade 8 by comparing the score obtained from the above

calculations to the scores of other countries participating in TIMSS.

Comparing Indonesian Student Achievement in Science with Other Countries Participating in TIMSS

The comparison of the mean Indonesian student achievement in science with

other countries participating in TIMSS is undertaken by using the predicted score

as shown in Table 12.5, which is 424.

Table 12.6 shows the position of Indonesia among other countries

participating in the TIMSS. It is seen in the table that Indonesia is ranked number

Ramon Mohandas


Table 12.5: Calculation of Predicted Scores for Australia in

Science Achievement STATES Logit TIMSS PREDICTED ACT 0.793 564 $Y bX a= + = 558 NSW 0.662 538 $Y bX a= + = 540 VIC 0.448 521 $Y bX a= + = 510 QLD 0.503 519 $Y bX a= + = 518 SA 0.611 531 $Y bX a= + = 533 WA 0.724 545 $Y bX a= + = 548 TAS 0.528 525 $Y bX a= + = 521 NT 0.510 504 $Y bX a= + = 519

Mean 0.597 531 531

r 0.910 Sy 18.295 Sx 0.121 b 137.615 a 448.667 INA -0.180 $Y bX a= + = 424

40 out of 42 countries. The two countries below Indonesia are Colombia and

South Africa with scores of 411 and 326 respectively. The four top positions are

held by Singapore, Czech Republic, Japan, and Korea with the scores of 607,

574, 571, and 565 respectively. Hong Kong, which is ranked number 4 in

mathematics, is performing relatively poorly in science compared to its level of

achievement in mathematics, and is ranked number 24 in science with a score of

522. On the other hand, the Czech Republic, which is ranked number 6 in

mathematics, is performing very well in science, and is ranked number 2.

Australia, a country used as the base to locate Indonesia onto the same scale as

that reported in TIMSS, is ranked number 12 with a score of 545. Another Asian

country, Thailand, is ranked number 22 with a score of 525. However, Beaton et

Ramon Mohandas


Table 12.6: Comparison of Science Achievement at Year 8 among Countries Participating in the TIMSSa

No. Country Mean Standard Error1 Singapore 607 5.52 Czech Republic 574 4.33 Japan 571 1.64 Korea 565 1.95 Bulgaria 565 5.36 Netherlands 560 5.07 Slovenia 560 2.58 Austria 558 3.79 Hungary 554 2.810 England 552 3.311 Belgium (Fl) 550 4.212 Australia 545 3.913 Slovak Republic 544 3.214 Russian Federation 538 4.015 Ireland 538 4.516 Sweden 535 3.017 United States 534 4.718 Canada 531 2.619 Germany 531 4.820 Norway 527 1.921 New Zealand 525 4.422 Thailand 525 3.723 Israel 524 5.724 Hong Kong 522 4.725 Switzerland 522 2.526 Spain 517 1.727 Scotland 517 5.128 France 498 2.529 Greece 497 2.230 Iceland 494 4.031 Romania 486 4.732 Latvia (LSS) 485 2.733 Portugal 480 2.334 Denmark 478 3.135 Lithuania 476 3.436 Belgium (Fr) 471 2.837 Iran 470 2.438 Cyprus 463 1.939 Kuwait 430 3.740 Indonesia 424 b41 Colombia 411 4.142 South Africa 326 6.6 a Adopted from: Beaton, A.E., et al., 1996a. Science Achievement in the Middle School Years, p. 22 b No estimate could be made for the standard error of the score for Indonesia.

Ramon Mohandas


al. (1996a) report that the difference in science achievement of students in

Australia and Thailand is significant, with the achievement in science for

students from Thailand being significantly lower than the achievement of the

Australian students. The mean achievement of the top performer, Singapore, is

significantly higher than the mean achievement of any other country listed in the

table. On the contrary, the mean achievement of the country at the bottom, South

Africa, is significantly lower than the mean achievement of any other country

listed in the table.

Lokan et al. (1996) report that the scale used in the TIMSS reports is

based on the item response theory (IRT) and the scale has a mean of 500 and a

standard deviation of 100. However, the scale mean and standard deviation relate

to an undefined population, and these statistics were presumably obtained by

some ill-defined pooling of a sample of cases drawn from the countries that

participated in the TIMSS testing program. Several analyses of the TIMSS data

report that 37 scale points of science achievement reported in the TIMSS would

be equivalent to a year’s growth in science achievement (Stedman, 1997; Beaton

et al., 1996a). If this is the case, then as in mathematics and based on the figures

in the table, Indonesia is lagging in science by almost three years behind

Thailand, and more than three years behind Australia, and almost five years

behind Singapore.

This result also raises concerns about the performance of the Indonesian

students in science at the Junior Secondary School level.

Conclusion

The results of the analysis indicate that in comparison with the other countries

participated in the TIMSS, Indonesia is performing very poorly both in

mathematics and science. In mathematics, Indonesia is ranked number 39 out 42

countries, while in science, Indonesia is ranked number 40 out of 42 countries. In

contrast, Singapore is the top performer for both mathematics and science.

Ramon Mohandas

Indonesian Performance Compared to other Countries

Ramon Mohandas

21

The apparently low level of performance of the Indonesian Junior

Secondary School students in mathematics and science found in this study raises

the question of why this could happen. What are the reasons for the low level of

performance?

The results of the analyses presented earlier indicate that at the student as

well as at the school levels, factors that consistently have significant influences

on achievement in mathematics and science are extra lessons and absenteeism.

This might suggest that students who spend more time engaging in mathematics

and science are more likely to achieve at a higher level in mathematics and

science than the students who spend less time on mathematics and science. If this

is the case, then should it mean that students might be encouraged to take extra

lessons after schools hours? The idea of encouraging students to take extra

lessons might not be wise because it would give extra burden to the parents

financially. One possible solution might be to decrease the seemingly

overcrowded curriculum and to put extra hours into the teaching of mathematics

and science.

Keeves (1995) reported that in the previous studies conducted by the IEA,

the achievement level of students from the developing countries is considerably

lower than the achievement level of students from the developed countries.

Several studies conducted in Indonesia also indicate that the achievement level of

the Indonesian students in mathematics and science is very low. The results of

comparison in this study confirm these findings.

Nevertheless, the reasons for the lower levels of achievement of the

Indonesian Junior Secondary School students in mathematics and science needs

to be investigated further in order to find ways of improving the quality of

mathematics and science teaching in Indonesia.

Comparing

Documents

Transcript of Comparing