Conceptualising and Testing Generic Skills in an International Assessment
Of Higher Education Learning Outcomes
Doug McCurry
Australian Council for Educational Research
OECD’s Assessment of Higher Education Learning Outcomes (AHELO) Feasibility Study
17 countries
248 higher education institutions
4 900 faculty
23 000 final-year engineering and economics students
The Feasibility Study aimed to
•test what is or could be feasible in international testing of learning in higher education
•determine what instruments would work well in such testing
•provide lessons on how learning outcomes might be effectively assessed in the future
The Feasibility Study looked at outcomes in
•the discipline-specific knowledge and skills of economics and engineering courses
•the GS common to all students (such as critical thinking, analytical reasoning, problem-solving, and written communication)
Why test generic skills?
The attention to GS reflects
• a recognition that competence means more than technical knowledge and skills
• a demand for an increasingly skilled workforce
• dissatisfaction with work readiness of some exit students
• increasing internationalisation and greater international competition
The generic skills debate
The rejecters
Cognitive performance is knowledge dependent, everything is learned in a context, transfer of learning across contexts is uncertain, and competence and expertise are knowledge rich
GS should be tested within domains
The accepters
GS are the fluid capacities that are used to learn new things, and that are developed by learning things
GS assessment should be focussed on the ability to think about something new and unfamiliar rather than focussed on what has been learned in the past
To the accepters the notion of ‘domain-specific generic skills’ is meaningless
Testing GS in domains is testing the domain specific not the generic
What is meant by the term generic skills?
These GS are distinguished from• the specific knowledge and skills needed for a particular
performance• what the expert knows• competencies or an area in which you have to be competent• a class of tasks
GS are the fluid capacities used to learn new things as distinct from what has been learned in the past
The assessment of general abilities is focused on aptitude to learn rather than achievement or competence
What is a generic skills test?
• aims to minimise the testing of prior knowledge• is stimulus based, and uses generally accessible,
non-specialist public domain material as stimulus• aims to maximise testing of the ability to
understand unfamiliar material and undertake novel thinking tasks
• involves a broad and balanced set of tasks• involves a range of tasks, but is consistent and
coherent enough to be summarised in a single score
Some other international generic skills studies
Prepared for life?: how to measure cross-curricular competencies 1997
OECD Program of International Student Assessment (PISA)
The OECD Definition and Selection of Competencies project (DeSeCo)
IEA Composition Study 1988
The Collegiate Learning Assessment
Council for Aid to Education
The CLA is a stimulus-based performance assessment of
• analytic reasoning and evaluation• problem solving• writing effectiveness• writing mechanics
The GS MCQ
The CC GS skills tested by ACER
• are seen as developed rather than innate abilities• do not involve specialized knowledge or skills• involve integrated and holistic thinking• aim at higher-order skills rather than basic skills or abstract
and de-contextualised information processing• involve thinking about the material world and socio-cultural
issues• aim to be generally accessible and test a broad and balanced
range of cognitive tasks
The ACER Tradition of CC TestingThe art of item writing
Psychometric methods but not psychometric constructs
Based on rich stimulus – specific, real-world, socio-cultural
Minimising the important of prior knowledge
Meaningful questions about real problems
Not abstract exercises or meaningless games
Unfamiliar questions and higher-order thinking
The common central core of cognitive abilities is tested by
•minimising the testing of prior knowledge
•maximising the ability to understand unfamiliar material and undertake novel thinking tasks
•basing testing on generally accessible, non-specialist, public domain stimulus material
•using a set of tasks that is broad enough to involve a complex of abilities, but coherent enough to be summarised in a single score
Categories and characteristics for the AHELO MCQ classification
International reactions to proposed material
Too logico-deductive and quantitative
Too much emphasis on concrete and material issues and information processing
Not conceptual enough
Not girl friendly enough
Some problems with language and cultural references
Some socio-political problems
Development of the proposed GS MCQ
Removed a unit of harsh and lenient prisons, but retained in similar set on a political issue
Added passages by Tolstoy and Ibsen about socio-cultural issues
Added part of a speech by Ghandi with historical and political significance
The 55 items used were broad, rich and intelligible from all sorts of perspectives
Adapting material to work in other languages
The risk of changing material that worked in English (in Australia) for easier transition to other languages
51 of 55 items had satisfactory to good difficulty and discrimination
Only 4 items misfitted in general and were removed from the analysis
A small number of items were removed for the analysis in individual countries
In some respects the GS MCQ were more successful than the domain-specific material
Towards an optimal generic skills test for international use
Testing GS across languages and cultures through unfamiliar but generally accessible material was successful
The feasibility study showed that • GS MCQ test did work well across languages
and cultures• CRI tests could work within languages and
cultures
A optimal test of GS for international use would
• have a number of different tasks rather than one theme to give breadth and balance
• integrate CRI and MCQ• be focussed on thinking rather than literacy• have some specific and some broad and open
tasks• have item specific scoring for some SAQ and
global scoring for some broad and open tasks
The AHELO feasibility study warrants an international GS test that is specific in some respects and broad and open in other respects
Such a test can deal with cultural and socio-political issues
A very encouraging and satisfying result
Top Related