Reading literacy of Austrian school leavers: between PISA and " Matura”

IATEFL/TEASIG, Innsbruck 2011

READING LITERACY OF AUSTRIAN SCHOOL LEAVERS: BETWEEN PISA AND

"MATURA”

Irene Thelen-Schaefer, BIFIE Wien


Presentation overview The study Theoretical background The research questions The instruments Operationalisation Results Findings Limitations Future research


Background to the study

An interest in researching the predictive potential in PISA reading tests


Cross-cultural assessment and translationPISA = Programme for international student assessment

In order to make a comparison of competence possible

Adaptation of tests across borders and culturesSet the specific psychological construct into the right context

Exclude cultural bias


The Study

Background test: Reading in German (L1)

• German translation of PISA reading literacy tasks 2009

Reading in English (L2)

• English version PISA reading literacy tasks 2009

Austrian standardised school leaving exam

• (“Matura”)

Research focus

• the comparability of items

• the predictability of L2 reading proficiency


Theoretical background 1: The skill reading

L1 reading and L2 readingAre we testing the same thing?Same basic cognitive comprehension process across L1

and L2 contexts

BUT

distinctive features in L2 reading:Language proficiency (“language threshold“)L2 processing skillsBackground knowledge


Theoretical background 2: The learnerLearning to read: L1/L2 differences

L1 LearnerYoung learnerOral skills developed before starting readingLearning to read and write at the same timeLarge knowledge of linguistic structures and vocabulary

L2 LearnerCognitively matureThey are L1 literate, but: they have not developed L2 oral comprehension yet different starting point in L2 readingL2 reading processes occur in a dual-language system


Theoretical background 3: The processes

Lower level processes

Automatisation is essentialAutomatisation of word recognitionSpeed of fixationWorking memoryRauding (Reading + auding; Carver, 1984)

Higher level processes Motivation / purpose of

reading Two levels of understanding

“a text model of comprehension““a situation model of reader

interpretation“; (Grabe, 2009)

Strategies Background knowledge

(Schema theory)


The Studyꞌs Research Questions

1. Can L1 reading results predict L2 reading results?

2. Can reading results from PISA English predict results of the “Matura” English L2?


The Instruments

Tests PISA German PISA English "Matura"

No of tasks 3 3 4

Time 45 min 45 min 50 min

Test methods MCQ; Text answer +

justif.

MCQ; Text answer +

justif.

MCQ; MM; T/F/NG; NF

No of items 11 18 25

Student feedback questionnaire


Operationalisation

Participating schools

Allgemeinbildende Höhere Schulen• (AHS; grammar schools)

Urban and rural areas • (Vienna, Linz, Admont, Hallein, Reutte, Villach…)

Public and private schools

Different types of AHS • BORG, BG, BRG, WIKU

• Different branches (foreign languages, science, arts, technology…)


Operationalisation

Test takers (TT)18-year-old students50 females, 51 males101 TT completed all three tests96 questionnaires returned

Teachers as test administratorstrained test administratorsprecise instructions given


(Provisional) Results

Tests compared

PISA German

PISA English

“Matura“

Mean 8.54 7.85 17.43

Mean % 77.36% 71.36% 69.72%

Std. Deviation

1.45 1.65 3.91

Mode 9 8 20


(Provisional) Results

All three tests show that

The test takers did well on the tests (FV, skew...)

The sample is relatively homogeneous (SD)

Normal distribution on the PISA English test and the

“Matura”


(Provisional) Results - Facility values

FV % 1 2 3 4 5 6 7 8 9 10 11

PISA G 79.2 63.4 90.1 90.1 35.6 97.0 93.1 83.2 58.4 74.3 90.1

PISA E 80.2 58.4 87.1 69.3 16.8 94.1 86.1 73.3 58.4 78.3 83.2

Findings

FV equal or slightly better in PISA German than in PISA English

Exceptions: items 4 and 5

The PISA tasks


(Provisional) Results – Item analysisItem 4

FV Corrected Item-total Correlation

Cronbach‘s Alpha if Item

Deleted

PISA German 90.1% .058 .265

PISA English 69.3% .279 .281

Possible reasons: technical language Frequent reasoning for test difficulty given in

questionnaires by TT: (lack of) vocabulary


(Provisional) Results - Item analysis

Item 5

Possible reasons:Test methodAspect targeted (“access and retrieve”) – answer to item 5 is after answer to item 6

FV Corrected Item-total Correlation

Cronbach‘s Alpha if Item

Deleted

PISA German 35.3% .050 .278

PISA English 16.7% .108 .358


(Provisional) Results – Item analysisItem 10

Corrected Item-total Correlation

Cronbach‘s Alpha if Item Deleted

PISA German .307 .126

PISA English .054 .379

Possible reasons: Partial credit in English only Possible answers for partial credit are included in the

“incorrect”-answer key in GermanItem 10 is the same in both languages re translation, but: it is the best item in PISA German and the worst in PISA English


(Provisional) Results - Reliability and discrimination

PISA tasks Low coefficients on both reliability and discriminationGerman: eight items discriminate positivelyEnglish: all items discriminate positively; but only item 4 is above .25 (at .279)

These plus the number of items might be the reasons for the low Cronbach’s alpha coefficients on the PISA tasks

PISA German PISA English “Matura”

Cronbach‘s Alpha .270 .371 .723


(Provisional) Findings - Item analysis

The “Matura”

Problematic items mainly in one test format: T/F/NG

Possible reasons:

Difficult for TT to distinguish between F and NG

This test format has now been replaced by T/F

justification.


(Provisional) Results - CorrelationsPISA German PISA English

PISA English .454**

“Matura“ .268** .383**

Findings

Empirical data show that research question 1 on a low possible overlap between reading in L1 and L2 can be stated.Surprisingly, TT did nearly as well on the PISA English tasks than on the PISA German tasks.

**. Correlation is significant at the 0.01 level (2-tailed).


Possible impact on the results

Motivation Unknown test methods in the PISA tasks Order of items Partial credit in PISA English (item 10) Tests target different purposes or aspects Instructions Examples


Further steps to be conducted

Further item analysis, incl. factor analysis PISA

English – “Matura”

Correlations of test results and questionnaire items,

e.g. Do avid readers have higher scores?


Limitations

Only 11 items in the PISA tasks

Items low reliability values

Possible memory effect PISA English – PISA German

Due to time pressure no counter-balanced design

possible


Ideas for further research

Longitudinal study as a diagnostic instrument:

Can reading results from the PISA reading literacy test taken by 16-year-olds in their L2 predict the ”Matura“ reading results?

Diagnostic purpose

Advice for students and parents


List of referencesAlderson, J.C. (1984). Reading in a foreign language: a reading problem or

a language problem? In J. C. Alderson and A. H. Urquhart (eds.). Reading in a foreign language. London: Longman.

Alderson, J.C. (2000). Assessing reading. Cambridge, UK: Cambridge University Press.

Bernhardt, E. (2005). Progress and procrastination in second language reading. Annual Review of Applied Linguistics 25, 133-150.

Carver, R. (1984). Rauding theory predictions of amount comprehended under different purposes and speed reading conditions. Reading Research Quarterly 19 (2), 205-218.

Grabe, W. (2009). Reading in a second language. Cambridge, UK: Cambridge University Press.

Koda, K. (2005). Insights into second language reading. Cambridge, UK: Cambridge University Press.

Stanovich, K. (1980). Toward an interactive-compensatory model of individual differences in the development of reading fluency. Reading Research Quarterly 16 (1), 32-71.

Walter, C. (2008). Phonology in second language reading: not an optional extra. TESOL QUARTERLY 42 (3), 455-474.


THANK YOU FOR YOUR ATTENTION!

CONTACT ADDRESS

[email protected]

Reading literacy of Austrian school leavers: between PISA and " Matura”

Documents

Transcript of Reading literacy of Austrian school leavers: between PISA and " Matura”