OECD Reviews of Evaluation and Assessment in Education: Czech Republic

OECD Reviews of Evaluation and Assessment in Education: Czech RepublicBy Paulo Santiago, Alison Gilmore, Deborah Nusche and Pamela Sammons

www.oecd.org/edu/evaluationpolicy

Conference, PragueThe main conclusions and policy

recommendationsPresentation by

Paulo SantiagoDirectorate for Education, OECD

15 March 2012

http://www.oecd.org/edu/evaluationpolicy

Outline of presentation

Part 1

The OECD Review on Evaluation and Assessment Frameworks for Improving School Outcomes and the Review of Evaluation and Assessment in the Czech Republic

Part 2

Conclusions and Recommendations• The Evaluation and Assessment Framework

• Student assessment

• Teacher appraisal

• School evaluation

• Education system evaluation

PART 1

(1) The OECD Review on Evaluation and Assessment Frameworks for Improving School Outcomes and;

(2) the Review of Evaluation and Assessment in the Czech Republic

OECD REVIEW - OBJECTIVES

●Purpose: To explore how systems of evaluation and assessment can be used to improve the quality, equity and efficiency of school education.

A Review of national approaches to evaluation and assessment in school education

●The Review:

Synthesises research-based evidence on the impact of evaluation and assessment strategies and disseminate this knowledge among countries.

Identifies innovative and successful policy initiatives and practices.

Facilitates exchanges of lessons and dialogue.

Identifies policy options for governments to consider.

OECD REVIEW - SCOPE

• The Review looks at the various components of assessment and evaluation frameworks. These include:

• Student assessment;

• Teacher appraisal;

• School evaluation;

• Education system evaluation;

• Other types of evaluation (programme evaluation, evaluation of school leadership etc)

• Comprehensive approach: Investigation of each component individually, as well as the coherence of the framework as a whole (including the links between the different components).

OECD REVIEW - KEY ISSUES

Key policy issues for analysis

1.Governance: Designing a systemic framework for evaluation and assessment2.Procedures: Ensuring the effectiveness of evaluation and assessment procedures3.Competencies: Developing competencies for evaluation and for using feedback4.Use of results: Making the best use of evaluation results5.Implementation: Implementing evaluation and assessment policies

OECD REVIEW - BROAD FEATURES

●Oversight by the Group of National Experts (GNE) on Evaluation and Assessmenthttps://community.oecd.org/community/evaluationpolicyinschools

●Countries working collaboratively with the SecretariatCountries do part of the work of the Review, a national co-ordinator is nominated, dialogue within

countries is encouraged

●Countries exchanging lessons and experiences, in particular through GNE meetings

●Collection of views and perspectives from a range of stakeholdersThrough their participation in GNE meetings and their interaction with Country Review expert

teams

●A diverse set of external experts involved in the Country ReviewsInvolvement of top academics and policy makers to bring expertise into the Review

●Ensuring a range of principles are respectedIndependence of analysis; inclusiveness in terms of the views and perspectives which are collected;

extensive reviews of the literature; evidence-based policy advice; site visits to interact with school realities; contextualisation of policy advice.

https://community.oecd.org/community/evaluationpolicyinschools

OECD REVIEW - METHODOLOGY

Analytical strand• Identifying the key questions for analysis and the background information needed

from countries• Reviewing the literature and evidence on the impact of evaluation and

assessment procedures• Gathering data on countries’ policies and practices (Country Background Reports)

Country Review strand• Country Reviews provide specific advice to individual countries. • OECD-led Review Team with external experts• The scope and focus is determined by the country in consultation with the

Secretariat Synthesis report

• Comparative report to analyse policy options and highlight good practices across countries.

Collaboration with other OECD activities and international agencies• INES (in particular NESLI), PISA, TALIS, CERI’s ILE , Longitudinal Information Systems and

Governing Complex Education Systems project, CELE’s on evaluating quality in educational facilities, GOV’s work on Public Sector Evaluation.

• EC, Eurydice, the World Bank, Standing International Conference of Inspectorates(SICI), UNESCO, BIAC, TUAC.

OECD REVIEW - PARTICIPATION

●Twenty four systems are preparing a Country Background Report:

Australia, Austria, Belgium (Flemish Community), Belgium (French Community), Canada, Chile, Czech Republic, Denmark, Finland, France, Hungary, Iceland, Ireland, Korea, Luxembourg, Mexico, Netherlands, New Zealand, Norway, Poland, Portugal, Slovak Republic, Slovenia and Sweden.

●Twelve countries also opted for a Country Review:Australia, Belgium (Flemish Community), Chile, Czech Republic, Denmark, Luxembourg, Mexico,

New Zealand, Norway, Portugal, Slovak Republic and Sweden.

●All OECD Member countries and Observers to the Education Policy Committee are members of the Group of National Experts on Evaluation and Assessment

REVIEW of EVALUATION and ASSESSMENT in the CZECH REPUBLIC

●Objectives: 1. To provide insights and suggestions for policy development in the Czech

Republic;2. To inform the wider international community about (a) the main features of the

Czech Republic’s evaluation and assessment policies; and (b) innovative and effective approaches to evaluation and assessment in the Czech Republic; and

3. To inform the final synthesis report from the project.

●Structure of visit (29 March – 5 April, 2011):• Review team with 4 members: 2 OECD Secretariat members; and researchers

from New Zealand and the United Kingdom;• Interacted with about 200 individuals; had about 45 meetings amounting to

about 50 hours of discussions; visited 6 schools in Prague, Ostrava and Liberec• Discussions were held with a wide range of national, regional and local

authorities; education officials; the Schools Inspectorate; parents’ organisations; representatives of schools and school directors; teacher educators; and researchers with an interest in evaluation and assessment issues.

REVIEW of EVALUATION and ASSESSMENT in the CZECH REPUBLIC

●Relevant features: 1. External perspective: looking from a distance;2. Independent perspective: no review team member has a vested interest in

the system;3. Not an examination: qualitative analysis that seeks to provide an input of a

specific nature to the internal debate;4. Bring together research evidence, data and information available in a broad

comprehensive comparative framework.

●Country Review report released on 30 January, 2012 and available at: www.oecd.org/edu/evaluationpolicy


PART 2

Conclusions and policy recommendations


The evaluation and assessment framework

The Evaluation and Assessment Framework

Strengths

There are common references at the national level to provide the basis for evaluation and assessment

4-year Long-term policy objectives with indicators; Framework Education Programmes

There are good conditions for adapting learning to local needsSchool Education Programmes: curriculum innovation; collaborative workRegional 4-year Long-term policy objectives

Responsibilities across the evaluation and assessment framework are well articulated

MEYS, CSI, Regions and municipalities, schools


Strengths (continued)

There is a range of initiatives to strengthen evaluation and assessment in the school system

National standardised tests, common part of the school-leaving examination, external school evaluation, school self-evaluation mandatory, national indicators on education

There is an “open door” climate among teachers

There is a good principle of supporting policy work with specific expertise

Institutes with specialised expertise, data collections, OECD projects

There are some reporting requirements:

Education database, inspection reports


Challenges The evaluation and assessment (E&A) framework needs to be completed

and made coherent

o There is no integrated E&A framework

o The E&A framework is incomplete (formative assessment of students, moderation of marks, systematic teacher appraisal, school self-evaluation incipient, no framework for school leadership appraisal)

o Some articulations within the E&A framework are not sufficiently developed

There is little attention to equity and inclusion in the evaluation and assessment framework

The Framework Education Programmes are not perceived as specific enough to guide teaching and assessment

Little shared understanding about what constitutes adequate, good and excellent performance

It is unclear that the students are at the centre of the evaluation and assessment framework


Challenges (continued) There is a narrow understanding of the purposes of evaluation and

assessmentPerceived as instrument to hold stakeholders accountable, to “control” and assess

compliance with regulations There is a need to strengthen competencies for evaluation and assessment

across the system

The articulation between levels of government and the support from the centre are limited

o Concerns about the lack of systematic application of national directions, inconsistency of practices and little capacity or commitment to developing quality frameworks.

o Weak articulations between the different decision-making levels.o Limited provision from the centre of tools and guidelines to assist evaluation and

assessment activities There are challenges in the implementation of some evaluation and

assessment initiatives


Policy recommendations Better articulate learning goals

o Establish clear goals for education and make equity and inclusion more prominent

o Clarify reference points and criteria for quality in evaluation and assessment

Integrate the evaluation and assessment framework

Develop a strategic plan or framework document that conceptualises a complete evaluation and assessment framework and articulates ways to achieve the coherence between its different components

Strengthen some of the components of the evaluation and assessment framework

Teacher appraisal, appraisal of school leaders, formative student assessment, school self-evaluation

Further develop some articulations within the evaluation and assessment framework


Policy recommendations (continued)

Build on some key principles to effectively implement evaluation and assessment

o Place the students at the centre of the evaluation and assessment frameworko Ensure a good emphasis on the improvement function of evaluation and

assessmento Communicate the rationale for evaluation and assessmento Recognise the importance of school leadershipo Establish an implementation strategy

Develop evaluation and assessment capacity across the school system

Improve the articulation between levels of government and assure support from the centre


Student assessment

Student Assessment

Strengths

Assessment is seen as part of the professional role of teachers

o Teachers play the major role in assessing and reportingo Schools decide and publish assessment criteriao Variety of approaches

An external dimension to assessment was introduced

o National component of school-leaving examinationo Common assignments for the apprenticeship certificateo Addresses the need for checks and balances to ensure reliability in the

application of standards There is an increased focus on student outcomes

o Including a move to identify expected minimum standards of achievement for students at key points in their education

Student Assessment

Challenges Approaches to learning and assessment remain markedly traditional

o Traditional approach to the organisation of classroomso Little emphasis in assessment practices on providing student feedback

Assessment for learning is not systematically used in Czech schools

o Feedback often understood as ‘summative assessment done more often’ Summative assessment of students raises some concerns

o Assessment is often norm-referencedo Achievement mixed with effort and motivation

There is limited consistency of student assessment across schools and classes

The national-level support for teacher-based student assessment is limited

o Lack of guidance to assess against FEPs (e.g. in the form of exemplars), insufficient assessment tools for teachers, limited professional development.

Student Assessment

Challenges (continued)

The national standardised tests entail a range of limitations and risks

o Development of the standards being rushedo May be more appropriately regarded as specifications for the national tests,

rather than standardso Potential negative consequences of high-stakes use: (i) ‘teaching to the test’;

(ii) cross-curricular competencies ignored; (iii) classroom time spent on preparation of the test; (iv) schools selecting the students taking the test.

The assessment of students leads to little interaction among teachers

o Low levels of teacher moderation of student assessments Multiple purposes to school-leaving examinations raise some concerns

o To provide a certificate of achievement to studentso To compare performance of schools: need other sources of information to

achieve this and value-added techniques

Student Assessment

Policy recommendations

Develop educational standards covering the breadth of student learning objectives prior to developing national standardised tests

o There is a need for clear external reference points in terms of expected levels of student performance at different levels of education

oNational tests not to be the vehicle to develop standards Limit the undesired effects of national standardised tests

Reflect further on the purposes of the national tests; should be trialled first; minimise negative impacts

Develop a broad strategy for student assessment and strengthen the role of formative assessment

Assessment for learning; provide tools supporting the assessment of students; broad range of assessment approaches; formative ass.

Build teachers’ capability for student assessment

Student Assessment

Policy recommendations (continued)

Develop a range of tools at the central level to support teacher-based student assessment

e.g. marking rubrics; exemplars illustrating student performance, assessment tools

Put in place moderation processes to ensure the consistency of student summative assessment

Student assessment should be criterion-based rather than norm-referenced

Ensure student assessment is inclusive

e.g. students with special needs, avoid biases by socio-economic background and minority status (e.g. Roma students)

Build capacity of markers of external tests and examinations


Teacher appraisal

Teacher Appraisal

Strengths

The principle that teachers should be evaluated is widely accepted

o Teacher appraisal takes place in all schools and is an important and normal part of school activities

Teacher appraisal is focused on evaluating classroom teaching

The importance of teacher professional development is recognised in the legislation

Some structures for co-operation and exchange among teachers are in place

o E.g. Subject commissions bringing together all teachers teaching a particular subject; lesson preparation; exchange of pedagogical approaches

Teacher Appraisal


There are plans to develop teaching standards and a new career system for teachers

The link between teacher appraisal and pay increments has potential to incentivise high performance

o Teachers may be awarded additional pay increments and bonuses that are determined by the school leadership team

o However, there are important questions regarding the transparency of how salary rewards are implemented

Teacher Appraisal

Challenges

There is currently no shared understanding of what constitutes high quality teaching

o Lack of a national framework defining standards for the teaching profession

o There are no uniform performance criteria or reference frameworks against which teachers could be appraised

Teacher appraisal is not systematically implemented for all teachers

The quality and extent of teacher appraisal approaches in individual schools depend on the capacity and leadership style of the school principals.

There is little tradition for educational leadership in schools

Many school leaders have not been sufficiently prepared for their wide range of tasks, in particular leading teaching and learning processes in the school.

Teacher Appraisal

Challenges (continued) There are tensions between the accountability and improvement functions of

teacher appraisal

o Teacher appraisal traditionally been conceived as a summative and accountability-oriented process

o There are risks involved in trying to achieve both the accountability function and the improvement function of teacher appraisal in one single process

The link between teacher appraisal and rewards is not transparent

Links between teacher appraisal and professional development could be enhanced

o The provision of prof. development appears fragmented and not systematically linked to teacher appraisal

o There is scope to better link teacher professional development to school development and improvement

Teacher Appraisal

Policy recommendations

Develop a professional profile or standards for the teaching profession

A clear and concise statement of what teachers are expected to know and be able to do is a key element in any teacher appraisal system as it provides a credible reference to make judgements about teacher competence

Strengthen teacher appraisal for improvement purposes (developmental appraisal)

o Strengthening regular school-based formative appraisal with a professional development focus which is separate from the more summative appraisal processes.

o To be validated externally by Schools Inspectorate

Further enhance the role of educational leadership

Teacher Appraisal

Policy recommendations (continued) Consider establishing a system of teacher certification to determine career

progression

Access to each of the key stages of a career could be associated with formal processes of summative appraisal that complement the regular formative appraisal

The different career stages should match the different levels of expertise reflected in teaching standards

Need a stronger component external to the school to validate the process and ensure that practices are consistent across the country

Three core instruments: classroom observation, self appraisal and documentation of practices in a simplified portfolio.

Ensure links between developmental appraisal and appraisal for certification

Ensure appropriate articulation between teacher appraisal and school evaluation


School evaluation

School Evaluation

Strengths External school evaluation is established

o Clear commitment to external accountability based around school evaluation with a regular cycle of external school evaluations

o External school evaluation valued by stakeholders

The external evaluation model embodies a number of features of best practice

o Well structured and systematico Set of publicly-available criteria for external inspection is drawn up every year o Publication of inspection reportso A range of sources of informationo Thematic reports

Schools facing greater challenges benefit from some follow-up

The CSI undertakes a follow-up inspection to assess whether improvements were undertaken to address the challenges previously identified.

School Evaluation


Classroom observation is part of school evaluation processes

Placing learning and teaching at the heart of the evaluation process sends clear signals about what matters.

There is a new emphasis on schools’ self-evaluation

Has the potential to encourage schools and principals to place a greater emphasis on school improvement and development planning.

School leadership is promoted in school evaluation

There is an explicit recognition that the processes of self-evaluation and external evaluation are hugely dependent on a principal’s capacity to stimulate engagement, to mobilise resources and to ensure appropriate training and support.

School Evaluation

Challenges External school evaluation seems to have limited emphasis on school

improvement

o The accountability function tends to emphasise compliance with legislation rather than the promotion of school improvement.

o Advice is only given to “weaker” schools which are identified as those that do not meet the minimum standards as set by law.

o There is not enough guidance about what will lead to school improvement and little attention is paid to identifying and disseminating best practice

There are a number of limitations in external school evaluation

Difficult to take account of the socio-economic context of the school; not enough emphasis on pedagogical aspects, Criteria used in the CSI external evaluations are not stable enough.

There is little emphasis on student results/progress

Limited ability to assess quality of learning and student progress.

School Evaluation

Challenges (continued) School self-evaluation needs to be strengthened

Its penetration across the school system remains at an early stage of development

The use of data for school development is limited

The evaluation by organising bodies has a limited scope and impact

There is no full recognition of the role of school leaders and their appraisal is limited

oWhile the school principal has a key role in the system and considerable responsibilities, this has not as yet been translated into a dedicated career structure

o Limited preparation for the role of school principalo Little recognition and financial rewardo The evaluation of school principals, conducted by organising bodies, is largely

absent except in terms of the financial aspects of budget management.

School Evaluation

Policy recommendations Strengthen external school evaluation

o Strengthen focus on school improvement and move away from the current “compliance” driven model.

o Provide advice for improvement to all schools evaluated.

o The school evaluation framework, the criteria and questions governing judgements and the methods employed should all focus much more directly on the quality of learning and teaching and their relationship to student outcomes.

Improve the alignment between external and self-evaluation and raise the profile of self-evaluation

Criteria used in both processes sufficiently similar as to create a common language

Give stronger emphasis to the follow-up to external evaluation

Improve the capacity of schools to engage in school evaluation

School Evaluation

Policy recommendations (continued) Improve the instructional leadership skills of school principals

Plan to use data on student results effectively

Any publication of results of school performance in students’ school-leaving examinations and/or national tests should be presented in ways that take account of intake differences including, for example, the socio-economic background of students.

Strengthen the evaluation of school principals

Improve the scope and role of organising bodies in school quality improvement

o The regional and municipal authorities should strengthen their role in supporting school improvement.

o Collaboration and networking amongst schools could be encouraged to help develop and spread good practice and enhance teachers’ professional skills.


Education system evaluation

Education System Evaluation

Strengths An Education Indicators Framework is established

It involves well-established procedures for data collection in close articulation with schools.

There is a concern to assess the progress of the education system towards pre established objectives

Principle of establishing educational objectives and the subsequent monitoring of the progress towards achieving them

The qualitative analysis of thematic reports provides valuable information for system monitoring

Richness of contextualised qualitative analysis

The participation in international surveys is instrumental for system evaluation


Challenges There is little emphasis on the evaluation of the education system

o Limited policy attention thus far and there is no comprehensive strategic approach to it

o The current narrow approach to system evaluation does not allow a broad enough assessment of the extent to which student learning objectives are being achieved

The absence of student performance data is a major gap in system monitoring

Presently there is no mechanism for the Czech Republic to monitor at a national level the achievement of its students against learning objectives specified in the Framework Education Programmes

There are key information gaps at the system level

There are no measures on students’ socio-economic background; There are gaps in the data collected from schools; Little emphasis on investigating inequities in the system; Limited information on the teaching and learning environment


Challenges (continued) It is not possible to monitor student outcomes over time and across schools

o Mostly due to the absence of national data on student performanceo School-level results of national tests might be disclosed with no account for

schools’ particular contexts. This can considerably distort considerations about the effectiveness of each school

Monitoring at the region and municipality levels is faced with considerable challenges

o Regions have a limited intervention in quality assurance with their main tool being the evaluation of school principals

oNo overview of the different quality assurance systems in the regions, including strategies for school improvement

o Limited articulation between Regions and municipalities System-level information is not fully exploited

Little analysis to inform educational planning; Limited use to inform school management; No comprehensive information system; Systematic sharing of data between schools is limited


Policy recommendations Raise the profile of system evaluation within the evaluation and assessment

framework

The challenge for system-level evaluation is to ensure that the measures of system performance are broad enough to capture the whole range of student learning objectives – policy making at the system level needs to be informed by high quality data and evidence, but not driven by the availability of such information.

Develop national student performance data for system monitoring

o Design national standardised tests for national monitoring and as a pedagogical tool [closely aligned with student learning objectives; can measure only a limited range of student learning objectives; publish test results at the school level is premature]

o Develop strategies to monitor a wider range of curricular areas and broader outcomes [develop sample-based surveys]

o Ensure that national monitoring covers broader outcomes [e.g. higher-order thinking skills and cross-curricular competencies]


Policy recommendations (continued) Prioritise efforts to meet information needs for national monitoring

o Develop measures of the socio-economic background of students o The data collection from schools needs to be improvedo Give more prominence to the analysis of inequities in the systemo Improve the information on the teaching and learning environment

Explore ways to more reliably track educational outcomes over time and across schools

Make meaningful comparisons across schools if test results are published at the school level

Strengthen the role of regions and municipalities in quality improvement Optimise the reporting and use of system-level data

o Strengthen the analysis for educational planning and policy developmento Improve feedback for local monitoringo Integrate available data and facilitate access by key agencieso Facilitate the sharing of student information across schools

Thank you([email protected])

(www.oecd.org/edu/evaluationpolicy)

mailto:[email protected]


OECD Reviews of Evaluation and Assessment in Education: Czech Republic

Education

Transcript of OECD Reviews of Evaluation and Assessment in Education: Czech Republic