Project Goals
-
Upload
dalton-woods -
Category
Documents
-
view
34 -
download
0
description
Transcript of Project Goals
![Page 1: Project Goals](https://reader035.fdocuments.us/reader035/viewer/2022062717/56812a68550346895d8deafd/html5/thumbnails/1.jpg)
© Stichting CITO Instituut voor Toetsontwikkeling 1
Mapping the Dutch Foreign Language State Examinations onto the Common
European Framework of Reference
Report of a Cito research project commissioned by
the Dutch Ministry of Education, Culture and Science
José Noijons & Henk Kuijper
![Page 2: Project Goals](https://reader035.fdocuments.us/reader035/viewer/2022062717/56812a68550346895d8deafd/html5/thumbnails/2.jpg)
© Stichting CITO Instituut voor Toetsontwikkeling 2
Project Goals
• To establish links between the existing examinations in French, German and English and the CEFR, following the steps as outlined in the Manual published by the Council of Europe.
• To study the possibilities of developing more comprehensive CEFR-related examinations in the foreign languages.
![Page 3: Project Goals](https://reader035.fdocuments.us/reader035/viewer/2022062717/56812a68550346895d8deafd/html5/thumbnails/3.jpg)
© Stichting CITO Instituut voor Toetsontwikkeling 3
Four Project Phases
Phase 1: Familiarisation
Phase 2: Specification
Phase 3: Standardisation
Phase 4: Empirical validation
![Page 4: Project Goals](https://reader035.fdocuments.us/reader035/viewer/2022062717/56812a68550346895d8deafd/html5/thumbnails/4.jpg)
© Stichting CITO Instituut voor Toetsontwikkeling 4
Phase 2: Specification
Specification of texts
• Related to the reading scales for communicative language activities (CEFR chapter 4)
• Related to text type, source, topic, domain (Dutch Grid)
• Related to the scales for communicative language competence (CEFR chapter 5 & Dutch Grid)
![Page 5: Project Goals](https://reader035.fdocuments.us/reader035/viewer/2022062717/56812a68550346895d8deafd/html5/thumbnails/5.jpg)
© Stichting CITO Instituut voor Toetsontwikkeling 5
Phase 2: Specification
Specification of items• Question types
• Task dimensions 1 (recognising, inferences, evaluation)
• Task dimensions 2 (explicit vs. implicit)
• Task dimensions 3 (content of operations)
![Page 6: Project Goals](https://reader035.fdocuments.us/reader035/viewer/2022062717/56812a68550346895d8deafd/html5/thumbnails/6.jpg)
© Stichting CITO Instituut voor Toetsontwikkeling 6
Specification texts: reading scales communicative activities German
5
21
173
3
0
50
100
150
200
readingcorrespondence
reading for orientation reading for informationand argument
reading instructions
![Page 7: Project Goals](https://reader035.fdocuments.us/reader035/viewer/2022062717/56812a68550346895d8deafd/html5/thumbnails/7.jpg)
© Stichting CITO Instituut voor Toetsontwikkeling 7
Specification texts: text type, source, topic, domain
• Great variation in text source and communicative themes (topics)
• Increase of proportion of expository and argumentative texts from lower to higher educational levels
• Domain: mainly personal
![Page 8: Project Goals](https://reader035.fdocuments.us/reader035/viewer/2022062717/56812a68550346895d8deafd/html5/thumbnails/8.jpg)
© Stichting CITO Instituut voor Toetsontwikkeling 8
Specification texts: communicative language competences
• Level of abstraction• Grammatical complexity• Vocabulary• Text length
Dutch grid
![Page 9: Project Goals](https://reader035.fdocuments.us/reader035/viewer/2022062717/56812a68550346895d8deafd/html5/thumbnails/9.jpg)
© Stichting CITO Instituut voor Toetsontwikkeling 9
Specification texts: communicative language competences, level of abstraction
0
0,5
1
1,5
2
2,5
3
3,5
4
vmbo-bb vmbo-kb vmbo-gl/tl havo vwo
English
German
Only concrete 1
Mostly concrete 2
Fairly extensive abstract 3
Mainly abstract 4
![Page 10: Project Goals](https://reader035.fdocuments.us/reader035/viewer/2022062717/56812a68550346895d8deafd/html5/thumbnails/10.jpg)
© Stichting CITO Instituut voor Toetsontwikkeling 10
Specification texts: communicative language competences, grammar
0
0,5
1
1,5
2
2,5
3
3,5
4
vmbo-bb vmbo-kb vmbo-gl/tl havo vwo
English
German
Only simple structures 1
Mainly simple structures 2
Limited range of complex structures 3
Wide range of complex structures 4
![Page 11: Project Goals](https://reader035.fdocuments.us/reader035/viewer/2022062717/56812a68550346895d8deafd/html5/thumbnails/11.jpg)
© Stichting CITO Instituut voor Toetsontwikkeling 11
Specification items
From lower to higher educational level:• more inferences and evaluation, less
recognising• Relatively more implicit information• Greater variation of operations
![Page 12: Project Goals](https://reader035.fdocuments.us/reader035/viewer/2022062717/56812a68550346895d8deafd/html5/thumbnails/12.jpg)
© Stichting CITO Instituut voor Toetsontwikkeling 12
Specification: general conclusions
• Increase of linguistic and cognitive complexity of texts
• Increase of variation of operations demanded in the items
![Page 13: Project Goals](https://reader035.fdocuments.us/reader035/viewer/2022062717/56812a68550346895d8deafd/html5/thumbnails/13.jpg)
© Stichting CITO Instituut voor Toetsontwikkeling 13
Claims through Specification and Standardisation
If a claim of a link to the CEFR is based on specification only, we do not know what score a candidate needs to claim that the candidate’s ability is at the CEFR-level the test claims to be at.
Claims can be further substantiated through standardisation of judgements: item-difficulties are judged in relation to CEFR levels.
![Page 14: Project Goals](https://reader035.fdocuments.us/reader035/viewer/2022062717/56812a68550346895d8deafd/html5/thumbnails/14.jpg)
© Stichting CITO Instituut voor Toetsontwikkeling 14
Standardisation
Standard-setting
Basket procedure: judges are to assign items (texts + tasks) to CEFR-levels.
What minimum CEFR level does one need to master this item?
![Page 15: Project Goals](https://reader035.fdocuments.us/reader035/viewer/2022062717/56812a68550346895d8deafd/html5/thumbnails/15.jpg)
© Stichting CITO Instituut voor Toetsontwikkeling 15
Results for Exams in English
Required minimum CEFR level for Exams in English
![Page 16: Project Goals](https://reader035.fdocuments.us/reader035/viewer/2022062717/56812a68550346895d8deafd/html5/thumbnails/16.jpg)
© Stichting CITO Instituut voor Toetsontwikkeling 16
Results for Exams in French
Required minimum CEFR level for Exams in French
![Page 17: Project Goals](https://reader035.fdocuments.us/reader035/viewer/2022062717/56812a68550346895d8deafd/html5/thumbnails/17.jpg)
© Stichting CITO Instituut voor Toetsontwikkeling 17
Results for Exams in German
Required minimum CEFR level for Exams in German
![Page 18: Project Goals](https://reader035.fdocuments.us/reader035/viewer/2022062717/56812a68550346895d8deafd/html5/thumbnails/18.jpg)
© Stichting CITO Instituut voor Toetsontwikkeling 18
Scores & Standards: English
Distribution of scores & cut score vwo-exam English
VWO English
Score distributionScore distribution
![Page 19: Project Goals](https://reader035.fdocuments.us/reader035/viewer/2022062717/56812a68550346895d8deafd/html5/thumbnails/19.jpg)
© Stichting CITO Instituut voor Toetsontwikkeling 19
Scores & Standards: French
Distribution of scores & cut score vwo-exam French
VWO French
Score distribution
VWO French
![Page 20: Project Goals](https://reader035.fdocuments.us/reader035/viewer/2022062717/56812a68550346895d8deafd/html5/thumbnails/20.jpg)
© Stichting CITO Instituut voor Toetsontwikkeling 20
Scores & Standards: German
Distribution of scores & cut score vwo-exam German
VWO English
Score distributionScore distribution
VWO German
![Page 21: Project Goals](https://reader035.fdocuments.us/reader035/viewer/2022062717/56812a68550346895d8deafd/html5/thumbnails/21.jpg)
© Stichting CITO Instituut voor Toetsontwikkeling 21
Conclusions standard-setting
• In the Netherlands a candidate can “pass” an exam without attaining the relevant CEFR level for that exam.
• Only scores at, or higher than the CEFR cut score indicate that the candidate is at or above the CEFR level the exam has been estimated to be at.
• Empirical validation may help to show that a CEFR level (a score) attained on one exam is equivalent to a CEFR level on another exam.
![Page 22: Project Goals](https://reader035.fdocuments.us/reader035/viewer/2022062717/56812a68550346895d8deafd/html5/thumbnails/22.jpg)
© Stichting CITO Instituut voor Toetsontwikkeling 22
General Conclusions
• A test that is linked or validated through specification only, cannot provide sufficient information on how candidates need to perform on the test to claim they have reached relevant attainment targets.
• It is necessary through standard setting to compute minimum scores that are needed for candidates to claim they have reached relevant attainment targets.
• External validation is needed to verify claims of links to the CEFR.