OECD Review on Evaluation and Assessment Frameworks for Improving School Outcomes - Deborah Nusche,...

OECD REVIEW ON EVALUATION AND

ASSESSMENT FRAMEWORKS FOR

IMPROVING SCHOOL OUTCOMES

Deborah Nusche, Analyst

OECD Directorate for Education

Bundesministerium für Unterricht, Kunst und Kultur

Vienna, 4 December 2012

Key topics for this presentation

• Why do evaluation and assessment matter?

• The OECD project

• Key findings from the Review

Why do evaluation and assessment matter?

• Highest rated education policy priority among OECD countries

• Analysing strengths and weaknesses of the education system is key to defining strategies for further development

• Need to provide information about education quality to parents and society at large

• Decentralisation and school autonomy are creating a greater need for evaluation and assessment

– To ensure that schools get the information and feedback they need to improve their work

– To allow the government to monitor if national goals for quality and equity in education are achieved

OECD Review on Evaluation and Assessment

Frameworks for Improving School Outcomes

• Purpose: To explore how systems of evaluation and assessment can be used to improve the quality, equity and efficiency of school education.

• Focus: A Review of national approaches to evaluation and assessment in school education (primary and secondary schools)

• Comprehensive approach: The Review looks at the various components of assessment and evaluation such as:

– Student assessment;

– Teacher appraisal;

– School evaluation;

– Education system evaluation.


Frameworks: Overview

System

School

Classroom

Student assessment

Teacher appraisal

School evaluation

System evaluation

Improvement of teaching and learning outcomes

Evaluation and assessment framework

Analytical strand – Reviewing the literature and evidence on the impact of evaluation and assessment

procedures

– Gathering data on countries’ policies and practices

Country Review strand – Country Reviews provide specific advice to individual countries.

– OECD-led Review Team

– The scope and focus is determined by the country in consultation with the Secretariat

Synthesis report – Comparative report to analyse policy options and highlight good practices across

countries.

Spotlight reports – Short reports providing a customised summary of the synthesis report putting the

spotlight on a particular country


Frameworks: Methodology

• 28 systems are providing detailed data on their evaluation and assessment practices

• 26 systems are preparing a comprehensive Country Background Report

• 14 systems (in green) opted for a Country Review

• 6 systems (in orange) opted for a Spotlight Report

Australia, Austria, Belgium (Flemish Community), Belgium (French Community), Canada, Chile, Czech Republic, Denmark, Finland, France, Hungary, Iceland, Ireland, Italy, Korea, Luxembourg, Mexico, Netherlands, New Zealand, Norway, Poland, Portugal, Slovak Republic, Slovenia, Sweden and United Kingdom (Northern Ireland).


Frameworks: Participation

• Background papers on issues such as formative assessment, use of test results, teacher appraisal, school evaluation etc. (11)

• Country Background Reports (26)

• OECD Publication series: OECD Reviews of Evaluation and Assessment in Education (14 Country Review Reports)

• Final Synthesis Report (early 2013)

All outputs available at:

www.oecd.org/edu/evaluationpolicy


Frameworks: Outputs

1. Governance: Striking the right balance between central efforts and local initiative and for improvement

2. Procedures: Designing the right instruments to ensure evaluation and assessment contribute to improvement of teaching and learning

3. Capacity: Developing competencies for evaluation and for using feedback at all levels of the education system

4. Use of results: Organising evaluative information in such a way that it facilitates effective use by stakeholders; avoiding ‘misuse’ of E&A results


Frameworks: Key topics

Achieving the right balance between central and local initiatives

• Decentralisation and school autonomy are widespread across OECD countries

• In about two-thirds of OECD countries, most schooling decisions in lower secondary education are taken at the local or school level

• At the same time: Central authorities remain responsible for ensuring quality and equity of education nationally

• Strong accountability mechanisms are often seen as a counterbalance to school autonomy


Frameworks: Governance

0

10

20

30

40

50

60

70

80

90

100

Neth

erla

nds

Engla

nd

Esto

nia

Belg

ium

(F

l.)

Czech R

epublic

Hungary

Icela

nd

Slo

vak R

epublic

Indonesia

Slo

venia

Irela

nd

Au

str

alia

Scotla

nd

Pola

nd

Sw

eden

Chile

Denm

ark

OE

CD

avera

ge

Kore

a

Italy

Isra

el

Fra

nce

Austr

ia

Belg

ium

(F

r.)

Sw

itzerla

nd

Spain

Germ

any

Port

ugal

United S

tate

s

Japan

Canada

Tu

rke

y

Mexic

o

Norw

ay

Luxem

bourg

Gre

ece

Chart D6.1 Percentage of decisions taken at each level of government in public lower secondary education (2011)

School Local

Countries are ranked in descending order of the percentage of decisions taken at the school level. Source: OECD, Education at a Glance 2012. Table D6.1 See Annex 3 for notes (www.oecd.org/edu/eag2012).



Advantages of a decentralised approach:

• Evaluation and assessment will only lead to improvement if stakeholders are fully engaged and use results / feedback to adjust their practices

• Ownership and responsibility for evaluation and assessment can generate commitment, professionalism and dynamism.

• Diversity of approaches allows for innovation.

Challenges of a decentralised approach

• Not all sub-national authorities and schools are systematic in their evaluation activities Concerns about those regions & schools that have less capacity or commitment for implementing quality management approaches.

• School practices in student assessment, teacher appraisal and school self-evaluation are variable and often depend on the leadership of individual principals and teachers.

Concerns about equity and equivalence of education across the country

• The potential for system learning is not fully exploited.



Some suggestions for policy development

• Develop central reference points for common orientation – Student assessment: Standards, learning progressions, assessment criteria and exemplars to

support fair and reliable assessment of students across schools

– Teacher appraisal: A professional profile or standards for the teaching profession as a basis for initial teacher education, teacher appraisal and professional development

– School evaluation: Shared quality indicators to guide both school self-evaluation and external evaluations

• Map out existing tools to support student assessment, teacher appraisal and school evaluation

• Clarify responsibilities for different aspects of evaluation and assessment

• Follow up with competency descriptions

• Develop adequate professional development offers



Using the right procedures for evaluation and assessment

• Research shows that evaluation and assessment can have a positive or negative impact on teaching and learning, depending on how approaches are designed and implemented

• Key question: How to design evaluation and assessment in a way that optimises potential positive effects and minimise potential negative effects?

Some challenges

• Aspects assessed: How to avoid signalling narrow expectations?

• Instruments used: How to find the right mix of instruments to give attention to all valued aspects of performance ? How to optimise use of ICT?

• Adaptations to context: How to ensure equity and fairness in E&A?


Frameworks: Procedures

Multiple choice

Closed-format

short answer

questions

Open-ended writing

tasks/calculations

Performing

a task

Oral questions and

answers Oral presentation

Austria Yes Yes Yes No Yes Yes

Australia Yes Yes Yes No No No

Belgium (Fr.) Yes Yes Yes Yes No No

Belgium (Fl.) Yes Possible Possible Possible No No

Czech Republic Yes Yes No No No No

Denmark Yes Yes No No No No

Finland Yes Yes No No No No

France Yes Yes No No No No

Hungary Yes No No Vocational only Vocational only No

Iceland Yes No ISCED 2 only ISCED 1 only ISCED 2 only No

Ireland Yes Yes No No No No

Israel Yes Yes Yes No No No

Italy Yes Yes No No No No

Mexico (sample) Yes Yes Yes No No No

Mexico (full) Yes No No No No No

Netherlands Yes No Yes No No No

New Zealand Yes Yes Yes Yes Yes No

Norway Yes Yes No No No No

Poland Yes No Yes No No No

Slovenia Yes No Yes No No No

Sweden Yes Yes Yes ISCED 1 only Yes ISCED 3 only



Testing formats used in national assessments, 2012


• Ensure that all important aspects of teaching and learning are given some attention in evaluation and assessment (beyond basic knowledge and skills)

• Draw on a comprehensive set of evaluation and assessment methodologies (observation, performance-tasks, dialogue, portfolios)

• Use multiple evaluators and instruments, both internal and external to the school

• Explore the potential of ICT to facilitate innovative evaluation and assessment methods

• Ensure that the context is considered in implementing evaluation and assessment procedures



How to ensure that stakeholders have the right competencies to evaluate and use evaluation results?

• Increasing focus on the role of the learner in assessment – Need to build student competencies to become active participants in their own assessment

– Teachers themselves need to develop skills for learner-centred teaching and assessment

• Countries report gaps in teacher skills regarding assessment – Attention to assessment varies across initial teacher education institutions

– Approaches to professional development in assessment are often fragmented

• School leaders may not be well prepared for leading evaluation and assessment in their schools – In many countries they have traditionally more administrative roles

– Lack of training regarding analysis of assessment data, teacher appraisal and school self-evaluation


Frameworks: Capacity

Percentage of teachers in lower secondary education who have a ‘high level of need’ of professional development in student assessment practices

05

101520253035404550

Ma

laysi

a

Lit

hu

an

ia

Ita

ly

Slo

ve

nia

No

rwa

y

Ko

rea

Bra

zil

Bu

lga

ria

TA

LIS

Ave

rag

e

Be

lgiu

m (

Fl.

)

Me

xic

o

Ice

lan

d

De

nm

ark

Po

lan

d

Au

stri

a

Est

on

ia

Tu

rke

y

Slo

va

k R

ep

ub

lic

Ire

lan

d

Au

stra

lia

Ma

lta

Po

rtu

ga

l

Hu

ng

ary

Sp

ain



Source: OECD TALIS Database

Some suggestions for policy development • Build student capacity for self-assessment

– Students will only develop their assessment capacity if teachers themselves have such capacity and are adequately supported

• Build teacher capacity for student assessment – Priority areas may include: assessment of key competencies; reliable grading; effective formative

assessment, inclusive and fair assessment of diverse students; use of national test results

– Adopt a strategic approach to teacher learning in assessment throughout initial teacher education and professional development

• Build the capacity of school leaders for teacher appraisal and school self-evaluation – In particular the capacity to evaluate, coach and guide staff; use of data for improvement.

• Build the capacity of sub-national authorities for school and local system evaluation – In particular the capacity to understand and make decisions based on data; to provide constructive

feedback to schools

• Build central expertise for system evaluation



• Evaluation and assessment are typically used for both improvement and accountability purposes

• But countries vary in the degree to which they emphasise one or the other – Countries with a strong focus on improvement typically emphasise: formative, low-

stakes assessment of students; teacher appraisal that is linked to decisions regarding teacher professional development and learning opportunities; school self-evaluation and external support for organisational learning.

– Countries with a strong focus on accountability typically emphasise: high-stakes standardised assessment of students; teacher appraisal that is linked to decisions regarding career advancement, salary, promotion and dismissal; external reviews or inspections of school quality; publication of school evaluation results and / or public comparisons of school performance.

• Where the accountability function overshadows the improvement function, this may lead to strategic responses (focussing only on the areas that are assessed)


Frameworks: Use of results

0

10

20

30

40

50

60

70

80

90

Un

ite

d S

tate

s

Un

ited

Kin

gdo

m

Ne

w Z

ea

lan

d

Ne

the

rla

nd

s

Slov

ak R

epub

lic

Swe

de

n

Nor

way

Ca

na

da

Po

lan

d

Turk

ey

Au

stra

lia

Den

mar

k

Luxe

mb

ou

rg

OEC

D a

vera

ge

Slo

ven

ia

Ch

ile

Mex

ico

Hu

ng

ary

Kore

a

Est

on

ia

Gre

ece

Cze

ch R

epu

blic

Ita

ly

Po

rtu

gal

Isra

el

Icel

and

Irel

and

Ge

rma

ny

Spai

n

Aus

tria

Jap

an

Swit

zerl

and

Fin

lan

d

Bel

giu

m

%

Percentage of 15-year-old students in schools where the principal reported that student

achievement data was posted publicly (2009)



Source: OECD PISA Database .


• Ensure the results of evaluation and assessment are used to make further improvements to teaching and learning – this should be the key purpose of all evaluation and assessment activities

• Link evaluation and assessment activities to professional learning and school development

• Use multiple sources of evidence when using results for summative / accountability purposes

• Consider the potential unintended effects when establishing high stakes accountability systems

• Ensure that results are used in line with the original purpose of each instruments. Avoid misuse of results.



VIELEN DANK! MEHR INFORMATIONEN WWW.OECD.ORG/EDU/EVALUATIONPOLICY

OECD Review on Evaluation and Assessment Frameworks for Improving School Outcomes - Deborah Nusche,...

Education

Transcript of OECD Review on Evaluation and Assessment Frameworks for Improving School Outcomes - Deborah Nusche,...