The role of assessment data in Austria
-
Upload
oecd-education -
Category
Education
-
view
424 -
download
4
Transcript of The role of assessment data in Austria
© Schreiner, February 2015
Education Governance: The Role of Data in Austria
Education Governance:
The Role of Assessment Data
in Austria
Governing Complex Education Systems
Tallinn,12-13 February 2015
Dr. Claudia Schreiner
Director
BIFIE – Federal Institute for Education Research,
Innovation & Development of the Austrian School System
Salzburg, Austria
© Schreiner, February 2015
Education Governance: The Role of Data in Austria
The Austrian Education System
© Schreiner, February 2015
Education Governance: The Role of Data in Austria
What data are collected
and for what purpose?
System monitoring: international
assessments
OECD-PISA
IEA-PIRLS(IEA-TIMSS)
© Schreiner, February 2015
Education Governance: The Role of Data in Austria
What data are collected
and for what purpose?
System monitoring: international
assessments
Quality development in schools:
national standards assessments;
census with focus on the school
level, but includes also system level
data for monitoring purposes
National Standards Assessments
© Schreiner, February 2015
Education Governance: The Role of Data in Austria
What data are collected
and for what purpose?
System monitoring: international
assessments
Quality development in schools:
national standards assessments;
census with focus on the school
level, but includes also system level
data for monitoring purposes
Self evaluation tools: centrally
provided test sets for self
evaluation; voluntary for teacher
use with no central data collection
7
6
5
3
9
© Schreiner, February 2015
Education Governance: The Role of Data in Austria
National Standards AssessmentsMain Features
© Schreiner, February 2015
Education Governance: The Role of Data in Austria
Starting point: Educational Standards
Intended outcomes defined through mandatory
Performance Standards in 2009
In German and Mathematics (grade 4) and German,
Mathematics and English (grade 8)
Baseline tests 2009 (grade 8) und 2010 (grade 4)
– samples of about 10.000 students per grade
National Standards Assessments as a census since
2012 with feedback to all schools and teachers as a basis
for quality development in schools
One assessment per year – one full cycle takes 5 years
© Schreiner, February 2015
Education Governance: The Role of Data in Austria
• National Report (1)
• Reports for Provinces (9)
• feedback to the regional school supervision
• school reports (1400 – 3100; online)
• feedback to teachers (~ 4900 per grade; online)
• individual student feedback (~75000 per grade; online)
system
monitoring
Target Groups for Feedback
quality
development
in schools
© Schreiner, February 2015
Education Governance: The Role of Data in Austria
• Census: every school and every teacher gets individual
results to raise local responsibility to work with the data
• Focus on instruction outcomes: students cannot be
identified in the teacher and school reports (are included
as individual data points, but without reference to names or
IDs)
• Quality assured external feedback through central data
collection, coding, processing and analysis
Main Purpose:
Quality Development in Schools
© Schreiner, February 2015
Education Governance: The Role of Data in Austria
Possibilities of comparison
in school/teacher feedback
Performance of school/teaching
group in comparison to …
direct comparisons … all Austrian students/schools
fair comparisons … schools/teaching groups working
under similar conditions
criterion-referenced
comparisons
… a centrally defined criterion
(proficiency levels)
= educational standards in the
form of achievement outcomes
© Schreiner, February 2015
Education Governance: The Role of Data in Austria
FEEDBACKfrom national standards
assessments
Competency-oriented teaching through evidence
based, regional school development
• professional self concept
• attitude towards evaluation
• expertise
• readiness to cooperate
school principal
reflecting on
the results
acting on the
resultsabsorbing the
feedback
“new” school inspection with coaching
– agreement of objectives and
development plans
feedback
facilitators
systems of development external support
teacher training and development
school characteristics• climate of evaluation und cooperation
• engagement / involvement of parents
• culture of open feedback
• innovative orientation
Wiesner & Schreiner, 2014 based on Helmke & Hosenfeld, 2006
© Schreiner, February 2015
Education Governance: The Role of Data in Austria
Our experiences so far …
© Schreiner, February 2015
Education Governance: The Role of Data in Austria
Evaluation of the feedback system and
experiences
positive feedback on content, structure, scope and comprehensibility of the reports
mostly open communication in schools
many schools managed to derive measures within the first 6 months
most difficult phase is the transition from feedback to action.
© Schreiner, February 2015
Education Governance: The Role of Data in Austria
Main challenges and success factors
One main challenge for teachers is accepting the results as an outcome of their teaching and use them in order to reflect upon their teaching.
This data system will only make a real change if the results are used by schools and are handled in a reflective way.
How the data is handled by other players, is crucial for the acceptance by schools and teachers. Trust is a decisive factor!
There are different players involved in the process, who need to collaborate in a productive way.
© Schreiner, February 2015
Education Governance: The Role of Data in Austria
Résumé
A comprehensive assessment and feedback system has been introduced in a complex system which lacks a strong tradition in evaluation.
The first round of assessments in central subjects as a census in grades 4 and 8 has started 2012.
The feedback system generally gets positive feedback concerning the organisation of the assessment, the feedback reports and the willingness of school staff to engage in working on the basis of the results.
Translating the information of the feedback reports into specific measures is being found difficult and needs special support – and requires a long learning process.
© Schreiner, February 2015
Education Governance: The Role of Data in Austria
Thank you!
Dr. Claudia Schreiner
BIFIE – Federal Institute for
Education Research,
Innovation & Development
Salzburg, Austria
www.bifie.at
© Schreiner, February 2015
Education Governance: The Role of Data in Austria
Aus: Schulbericht
(Rückmeldung an
die Schulleitung
und Schulpartner)
MathematikStreuung an der
Schule
© Schreiner, February 2015
Education Governance: The Role of Data in Austria
Aus: Schulbericht
(Rückmeldung an
die Schulleitung
und Schulpartner)
MathematikErgebnis nach
Geschlecht
© Schreiner, February 2015
Education Governance: The Role of Data in Austria
Mathematikkompetenz im fairen
VergleichÄhnliche Rahmenbedingungen in
Bezug auf:
Merkmale des Schulstandorts
(Gemeindegröße, Urbanisierungs-
grad, Entfernung zur nächsten
AHS)
Merkmale der Schule
(Schulgröße, Schulart, Privatschule,
Anteil ausgenommener
Schüler/innen)
Merkmale der Schülerschaft
(Anteil an Mädchen/Burschen,
Anteil an Kindern mit nichtdeutscher
Erstsprache, Anteil an Kindern mit
Migrationshintergrund, mittlerer
Sozialstatus)
© Schreiner, February 2015
Education Governance: The Role of Data in Austria
Mathematikkompetenz im fairen
Vergleich
© Schreiner, February 2015
Education Governance: The Role of Data in Austria
Mathematikkompetenz nach Kompetenzstufen
© Schreiner, February 2015
Education Governance: The Role of Data in Austria
Aus: Schulbericht
(Rückmeldung an
die Schulleitung
und Schulpartner)
Mathematik:Verteilung auf die
Kompetenzstufen
© Schreiner, February 2015
Education Governance: The Role of Data in Austria
Rückmeldungaus der Überprüfung der
Bildungsstandards
Kompetenzorientiertes Unterrichten durch
datengestützte, regionale Schulentwicklung durch flächendeckende Erhebung und Rückmeldung
• Professionelles Selbstverständnis
• Akzeptanz von Evaluation
• Vorwissen/Expertise
• Kooperationsbereitschaft
Schulleitung
• Suche nach
Erklärungen
• Analyse der
Ergebnisse
Reflexion
• Verbesserung
von Unterricht
• Schul-
entwicklung
AktionRezeption
• Verständnis
• Erkennen von
Stärken und
Schwächen
Schulqualität Allgemeinbildung (SQA) -
Zielvereinbarung und Entwicklungspläne
Rückmelde-
moderatoren
Entwicklungsberatung in Schulen (u.a. EBIS)
Lehreraus- und -weiterbildung
SchulischeRahmenbedingungen• Evaluations- und Kooperationsklima
• gelebte Schulpartnerschaft
• Feedbackkultur
• Innovative Orientierung
Modell Wiesner/Schreiner © 2014
© Schreiner, February 2015
Education Governance: The Role of Data in Austria
Rückmeldemoderation
• In allen Bundesländern wurde ein Pool an RMMs
ausgebildet.
• Ausbildung in Kooperation zwischen BIFIE und Päd.
Hochschulen
• Grundausbildung: 2 * 2 Präsenztage
• Einsatztraining (jährlich): 1 Tag je Fach/Schulstufe
• RMMs sollen den Prozess der Rezeption unterstützen
• Umfang: max. 2 Kontakte
• Gespräch mit Schulleitung (fixer Bestandteil)
• ggf. zweiter Besuch für Arbeit mit gesamtem Kollegium, Fachteam
oder den betroffenen Lehrer/innen
© Schreiner, February 2015
Education Governance: The Role of Data in Austria
Rückmeldungaus der Überprüfung der
Bildungsstandards
Kompetenzorientiertes Unterrichten durch
datengestützte, regionale Schulentwicklung durch flächendeckende Erhebung und Rückmeldung
• Professionelles Selbstverständnis
• Akzeptanz von Evaluation
• Vorwissen/Expertise
• Kooperationsbereitschaft
Schulleitung
• Suche nach
Erklärungen
• Analyse der
Ergebnisse
Reflexion
• Verbesserung
von Unterricht
• Schul-
entwicklung
AktionRezeption
• Verständnis
• Erkennen von
Stärken und
Schwächen
Schulqualität Allgemeinbildung (SQA) -
Zielvereinbarung und Entwicklungspläne
Rückmelde-
moderatoren
Entwicklungsberatung in Schulen (u.a. EBIS)
Lehreraus- und -weiterbildung
SchulischeRahmenbedingungen• Evaluations- und Kooperationsklima
• gelebte Schulpartnerschaft
• Feedbackkultur
• Innovative Orientierung
Modell Wiesner/Schreiner © 2014
© Schreiner, February 2015
Education Governance: The Role of Data in Austria
What data are collected and for what purpose?
System monitoring: international assessments
© Schreiner, February 2015
Education Governance: The Role of Data in Austria
The Austrian School System –
international Assessments
OECD-PISAin M – R – S3-year cycle
IEA-PIRLS (& TIMSS)in R – M – S
4/5-year cycle
© Schreiner, February 2015
Education Governance: The Role of Data in Austria
What data are collected and for what purpose?
System monitoring: international assessments
Quality development in schools: national standards
assessments; census with focus on the school level, but
includes also system level data for monitoring purposes
© Schreiner, February 2015
Education Governance: The Role of Data in Austria
The Austrian School System –
data for quality development in schools
National Standards Assessment in German – Mathematics
National Standards Assessmentin German – Mathematics – English
© Schreiner, February 2015
Education Governance: The Role of Data in Austria
What data are collected and for what purpose?
System monitoring: international assessments
Quality development in schools: national standards
assessments; census with focus on the school level, but
includes also system level data for monitoring purposes
Self evaluation tools: centrally provided test sets for self
evaluation; voluntary for teacher use with no central data
collection
© Schreiner, February 2015
Education Governance: The Role of Data in Austria
The Austrian School System –
self evaluation tools
Informal competency assessment 3in German – Mathematics
Informal competency assessment 6In German – Mathematics – English
Informal competency assessment 7in German – Mathematics – English – Science
competency check 9in Mathematics, English (German will follow)
competency check 5Mathematics in trial (German will follow)
© Schreiner, February 2015
Education Governance: The Role of Data in Austria
Domain-orientated Test Design
The cycle is now being changed into a 5-year cycle with only one assessment per year.
© Schreiner, February 2015
Education Governance: The Role of Data in Austria
School report: overview of content
Information about school results:
compared to AUT
compared to the defined goals (in the regulation)
compared to other schools with similar basicconditions („fair comparison“)
Areas of conflict:
Educational Standards apply to all students – and
therefore to all schools
Conditions of schools vary widely
© Schreiner, February 2015
Education Governance: The Role of Data in Austria
Evaluation of the feedback reports and
procedures for schools and teachers (1)
positive feedback on content, structure, scope and comprehensibility of the reports
communication:
in most schools open communication amongst teachers
with the whole teaching body, the subject matter groups or bilaterally with individual teachers
but low interest by parent representatives
~ 40 % of headmasters asked for feedback facilitators, in most cases in order to validate their own interpretation of the school report
© Schreiner, February 2015
Education Governance: The Role of Data in Austria
Evaluation of the feedback reports and
procedure for schools and teachers (2)
The use of feedback reports:
Many schools initiated measures in response to feedback reports within the first 6 months.
Action was mostly taken with the purpose of quality development (in contrast to quality assurance).
Most measures derived concerned the development of subject matter teaching and didactics.
Headmasters described their teachers mostly as ranging from positive to playing an active role in taking action.
© Schreiner, February 2015
Education Governance: The Role of Data in Austria
Educational Standards –
What is special about the Austrian Model?
School-specific development of instruction
Mandatory educational standards
Survey of all students and schools
Feedback to all students, teachers, schools
Feedback facilitators („Rückmeldemoderation“)
Stimulation of quality development
= Austrian Model
Strong linkage of monitoring and practical school andinstructional development
Strengthening of schools, school management and regional quality development
© Schreiner, February 2015
Education Governance: The Role of Data in Austria
Different players and roles
Government & education ministry: determination of goals (educational standards) and recipient of nation-wide results
Schools and teachers: recipients of external feedback and responsible for quality development
School authorities: support school development work and check on it (possible role conflict)
BIFIE: responsible for assessment, data collection, processing and feedback
© Schreiner, February 2015
Education Governance: The Role of Data in Austria
Herzlichen Dank
für Ihre
Aufmerksamkeit!
www.bifie.at
© Schreiner, February 2015
Education Governance: The Role of Data in Austria
OECD-PISA
IEA-PIRLS(IEA-TIMSS)
International Assessments: system monitoring
National Standards Assessments
National Standards Assessments: quality development in schools
7
X Self evaluation tools and competency checks in grades X
6
5
3
9
What data are collected,
at what points, and for what purpose?