PISA 2012 Evaluating systems to improve education
36
OECD EMPLOYER BRAND Playbook 1 PISA 2012 Evaluating systems to improve education The yardstick for success is no longer improvement by national standards alone but the best performing education systems Bett 2014, School Leaders Summit 23 January 2014 Michael Davidson Head of Early Childhood and Schools Division, OECD Directorate for Education and Skills
description
The yardstick for success is no longer improvement by national standards alone but the best performing education systems
Transcript of PISA 2012 Evaluating systems to improve education
OECD EMPLOYER
BRAND
Playbook
1
PISA 2012 Evaluating systems
to improve education
The yardstick for success is no longer
improvement by national standards
alone but the best performing
education systems
Bett 2014, School Leaders Summit
23 January 2014
Michael Davidson Head of Early Childhood and
Schools Division, OECD Directorate
for Education and Skills
2 PISA in brief
• Over half a million students… – representing 28 million 15-year-olds in 65 countries/economies
… took an internationally agreed 2-hour test… – Goes beyond testing whether students can
reproduce what they were taught…
… to assess students’ capacity to extrapolate from what they know and creatively apply their knowledge in novel situations
– Mathematics, reading, science, problem-solving, financial literacy
– Total of 390 minutes of assessment material
… and responded to questions on… – their personal background, their schools
and their engagement with learning and school
• Parents, principals and system leaders provided data on… – school policies, practices, resources and institutional factors that help
explain performance differences .
3
Helen the Cyclist
Helen has just got a new bike. It has a speedometer which
sits on the handlebar. The speedometer can tell Helen the
distance she travels and her average speed for a trip.
Helen rode 6 km to her aunt’s house. Her speedometer
showed that she had averaged 18 km/h for the whole trip.
Which one of the following statements is correct?
A. It took Helen 20 minutes to get to her aunt’s house.
B. It took Helen 30 minutes to get to her aunt’s house.
C. It took Helen 3 hours to get to her aunt’s house.
D. It is not possible to tell how long it took Helen
to get to her aunt’s house.
PISA 2012 Sample Question 2
4
Correct Answer: A. It took Helen 20 minutes to get to her aunt’s house.
This item belongs to the change and relationships category. This involves understanding
fundamental types of change and recognising when they occur in order to use suitable
mathematical models to describe and predict change.
SCORING:
Description: Calculate time travelled given average speed and distance
travelled
Mathematical
content area:
Change and relationships
Context: Personal
Process: Employ
Helen the Cyclist
PISA 2012 Sample Question 2
5
Percent of 15-year-olds who scored Level 3 or Above S
hang
hai-C
hina
S
inga
pore
H
ong
Kon
g-C
hina
K
orea
C
hine
se T
aipe
i M
acao
-Chi
na
Japa
n Li
echt
enst
ein
Sw
itzer
land
E
ston
ia
Net
herla
nds
Fin
land
C
anad
a P
olan
d V
ietn
am
Ger
man
y B
elgi
um
Aus
tria
Ir
elan
d D
enm
ark
Aus
tral
ia
Cze
ch R
epub
lic
Slo
veni
a N
ew Z
eala
nd
Fra
nce
Uni
ted
Kin
gdom
Ic
elan
d O
EC
D a
vera
ge
Latv
ia
Nor
way
Lu
xem
bour
g P
ortu
gal
Spa
in
Italy
R
ussi
an F
eder
atio
n S
lova
k R
epub
lic
Sw
eden
Li
thua
nia
Uni
ted
Sta
tes
Hun
gary
Is
rael
C
roat
ia
Gre
ece
Ser
bia
Tur
key
Bul
garia
R
oman
ia
Uni
ted
Ara
b E
mira
tes
Kaz
akhs
tan
Chi
le
Tha
iland
M
alay
sia
Uru
guay
M
onte
negr
o M
exic
o A
lban
ia
Qat
ar
Cos
ta R
ica
Bra
zil
Arg
entin
a T
unis
ia
Jord
an
Per
u C
olom
bia
Indo
nesi
a
0
10
20
30
40
50
60
70
80
90
100
PISA 2012 Sample Question 2
What do 15-year-olds know… …and what can they do with what they know?
Mathematics (2012)
6
Each year OECD countries spend 200bn$ on maths education in school
Singapore
Hong Kong-China Chinese Taipei
Korea
Macao-China Japan Liechtenstein Switzerland
Netherlands Estonia Finland Canada
Poland Belgium Germany Viet Nam
Austria Australia Ireland Slovenia
Denmark New Zealand Czech Republic France
United Kingdom Iceland
Latvia Luxembourg Norway Portugal Italy Spain
Russian Fed. Slovak Republic United States Lithuania Sweden Hungary
Croatia Israel
Greece Serbia Turkey
Romania
Bulgaria U.A.E. Kazakhstan Thailand
Chile Malaysia
Mexico 410
420
430
440
450
460
470
480
490
500
510
520
530
540
550
560
570
580
Mean score
High mathematics performance
Low mathematics performance
… Shanghai-China performs above this line (613)
… 12 countries perform below this line
Average performance
of 15-year-olds in
Mathematics Fig I.2.13
Socially equitable
distribution of learning
opportunities
High mathematics performance
Low mathematics performance
Average performance
of 15-year-olds in
mathematics
Strong socio-economic
impact on student
performance
Singapore
Hong Kong-China Chinese Taipei
Korea
Macao-China Japan Liechtenstein Switzerland
Netherlands Estonia Finland Canada
Poland Belgium Germany Viet Nam
Austria Australia Ireland Slovenia
Denmark New Zealand Czech Republic France
United Kingdom Iceland
Latvia Luxembourg Norway Portugal Italy Spain
Russian Fed. Slovak Republic United States Lithuania Sweden Hungary
Croatia Israel
Greece Serbia Turkey
Romania
Bulgaria U.A.E. Kazakhstan Thailand
Chile Malaysia
Mexico
Australia Austria
Belgium Canada
Chile
Czech Rep.
Denmark
Estonia
Finland
France
Germany
Greece
Hungary
Iceland Ireland
Israel
Italy
Japan
Korea
Luxembourg
Mexico
Netherlands
New Zealand
Norway
Poland
Portugal
Slovak Rep.
Slovenia
Spain Sweden
Switzerland
Turkey
UK
US
Singapore
Hong Kong-China Chinese Taipei
Macao-China
Liechtenstein
Viet Nam
Latvia
Russian Fed. Lithuania
Croatia
Serbia Romania
Bulgaria United Arab Emirates
Kazakhstan
Thailand
Malaysia
02468101214161820222426
2012 Shanghai-China
Socially equitable
distribution of learning
opportunities
Strong
socio-economic
impact on student
performance
200
494
-3 -2 -1 0 1 2 3
School performance and socio-economic background:
United Kingdom 10
Advantage PISA Index of socio-economic background Disadvantage
Student performance and students’ socio-economic background
School performance and schools’ socio-economic background
Student performance and students’ socio-economic background within schools
Stu
dent
perf
orm
ance
700
Private school
Public school in rural area
Public school in urban area
0
2
4
6
8
10
12
14
16
18
20S
han
ghai-
Chin
a
Hon
g K
ong-…
Macao-C
hin
a
Sin
gapore
Kore
a
Chin
ese T
aip
ei
Japan
Neth
erland
s
Pola
nd
Port
ugal
Germ
any
Italy
Spain
United K
ingd
om
Fra
nce
Norw
ay
Icela
nd
New
Zea
lan
d
Russia
n F
ed
.
United S
tate
s
Den
mark
Sw
ed
en
Hun
gary
Slo
va
k R
epublic
Mexic
o
Gre
ece
Isra
el
Indo
nesia
Bra
zil
Chile
Arg
entina
%
Percentage of resilient students
More than 10
% resilient
Between 5%-10% of resilient
students Less than 5%
Fig II.2.4 11
Australia Austria
Belgium Canada
Chile
Czech Rep.
Denmark
Estonia
Finland
France
Germany
Greece
Hungary
Iceland Ireland
Israel
Italy
Japan
Korea
Luxembourg
Mexico
Netherlands
New Zealand
Norway
Poland
Portugal
Slovak Rep.
Slovenia
Spain Sweden
Switzerland
Turkey
UK
US
Singapore
Hong Kong-China Chinese Taipei
Macao-China
Liechtenstein
Viet Nam
Latvia
Russian Fed. Lithuania
Croatia
Serbia Romania
Bulgaria United Arab Emirates
Kazakhstan
Thailand
Malaysia
02468101214161820222426
2012 Shanghai-China
Socially equitable
distribution of learning
opportunities
Strong socio-economic
impact on student
performance
Australia Austria
Belgium Canada
Chile
Czech Rep.
Denmark
Estonia
Finland
France
Germany
Greece
Hungary
Iceland Ireland
Israel
Italy
Japan
Korea
Luxembourg
Mexico
Netherlands
New Zealand
Norway
Poland
Portugal
Slovak Rep.
Slovenia
Spain Sweden
Switzerland
Turkey
UK
US
Australia
Austria
Belgium
Canada
Chile
Czech Rep.
Denmark
Estonia
Finland
France
Germany
Greece
Hungary
Iceland
Ireland
Israel
Italy
Japan
Korea
Luxembourg
Mexico
Netherlands
New Zealand
Norway
Poland
Portugal
Slovak Rep.
Slovenia
Spain
Sweden
Switzerland
Turkey
UK
US
Singapore
2003 - 2012 Germany, Turkey and Mexico improved
both their mathematics performance
and equity levels
Brazil, Italy, Macao-China, Poland,
Portugal, Russian Federation,
Thailand and Tunisia improved
their mathematics performance
(no change in equity)
Liechtenstein, Norway, the United
States and Switzerland improved
their equity levels (no change in
performance)
Australia Austria
Belgium Canada
Chile
Czech Rep.
Denmark
Estonia
Finland
France
Germany
Greece
Hungary
Iceland Ireland
Israel
Italy
Japan
Korea
Luxembourg
Mexico
Netherlands
New Zealand
Norway
Poland
Portugal
Slovak Rep.
Slovenia
Spain Sweden
Switzerland
Turkey
UK
US
Australia
Austria
Belgium
Canada
Chile
Czech Rep.
Denmark
Estonia
Finland
France
Germany
Greece
Hungary
Iceland
Ireland
Israel
Italy
Japan
Korea
Luxembourg
Mexico
Netherlands
New Zealand
Norway
Poland
Portugal
Slovak Rep.
Slovenia
Spain
Sweden
Switzerland
Turkey
UK
US
Singapore
2003 - 2012 Germany, Turkey and Mexico improved
both their mathematics performance
and equity levels
Brazil, Italy, Macao-China, Poland,
Portugal, Russian Federation,
Thailand and Tunisia improved
their mathematics performance
(no change in equity)
Liechtenstein, Norway, the United
States and Switzerland improved
their equity levels (no change in
performance)
Brazil, Italy, Macao-China, Poland, Portugal,
Russian Federation, Thailand and Tunisia
saw significant improvements in math performance between
2003 and 2012 (adding countries with more recent trends results in 25 countries with
improvements in math)
Australia Austria
Belgium Canada
Chile
Czech Rep.
Denmark
Estonia
Finland
France
Germany
Greece
Hungary
Iceland Ireland
Israel
Italy
Japan
Korea
Luxembourg
Mexico
Netherlands
New Zealand
Norway
Poland
Portugal
Slovak Rep.
Slovenia
Spain Sweden
Switzerland
Turkey
UK
US
Australia
Austria
Belgium
Canada
Chile
Czech Rep.
Denmark
Estonia
Finland
France
Germany
Greece
Hungary
Iceland
Ireland
Israel
Italy
Japan
Korea
Luxembourg
Mexico
Netherlands
New Zealand
Norway
Poland
Portugal
Slovak Rep.
Slovenia
Spain
Sweden
Switzerland
Turkey
UK
US
Singapore
2003 - 2012 Germany, Turkey and Mexico improved
both their mathematics performance
and equity levels
Brazil, Italy, Macao-China, Poland,
Portugal, Russian Federation,
Thailand and Tunisia improved
their mathematics performance
(no change in equity)
Liechtenstein, Norway, the United
States and Switzerland improved
their equity levels (no change in
performance)
Norway, the United States and
Switzerland improved equity
between 2003 and 2012
Of the 65 countries… …40 improved at least in one subject
16
17 Change in performance between PISA 2003 and 2012
Indonesia
Thailand
Russian Fed.
United States
Latvia
Spain
Norway Luxembourg
Ireland
Austria
Switzerland Japan
Liechtenstein
Korea
Brazil
Tunisia
Mexico
Uruguay
Turkey
Greece
Italy
Portugal
Hungary
Poland
Slovak Republic
OECD average
Germany
Sweden
France
Denmark
Iceland
Czech Republic
New Zealand Australia
Macao-China
Belgium
Canada
Netherlands
Finland
Hong Kong-China
-4
-3
-2
-1
0
1
2
3
4
5
350 400 450 500 550 600
Ave
rag
e a
nn
ual m
ath
em
ati
cs
sc
ore
ch
an
ge
Average mathematics performance in PISA 2003
Imp
rovin
g p
erfo
rma
nce
D
ete
riora
ting
perfo
rma
nce
PISA 2003 performance below the OECD average PISA 2003 performance
above the OECD average
Fig I.2.18
But underachievement persists
…
18
0
10
20
30
40
50
60
70
80
90
100S
ha
ngh
ai-
Ch
ina
Sin
ga
po
re
Hong
Kon
g-C
hin
a
Kore
a
Esto
nia
Ma
ca
o-C
hin
a
Ja
pa
n
Fin
land
Sw
itze
rla
nd
Chin
ese
Taip
ei
Cana
da
Pola
nd
Neth
erla
nds
Denm
ark
Irela
nd
Germ
any
Austr
ia
Belg
ium
Austr
alia
Latv
ia
Slo
ven
ia
Czech R
ep
ub
lic
Icela
nd
United
Kin
gd
om
Norw
ay
Fra
nce
New
Ze
ala
nd
OE
CD
ave
rag
e
Spa
in
Ru
ssia
n F
ede
ratio
n
Luxe
mb
ou
rg
Italy
Port
ug
al
United
Sta
tes
Sw
ede
n
Slo
vak R
epu
blic
Hung
ary
Isra
el
Gre
ece
Tu
rke
y
Chile
Me
xic
o
Arg
en
tin
a
Bra
zil
Ind
one
sia
%
Percentage of low-performing students
in mathematics Tab I.2.1a 19
Across OECD, 23% of students are low performers (Below Level 2). They can extract relevant information from a single source and can use basic algorithms, formulae, procedures or conventions to solve problems involving whole numbers.
Gender differences remain
20
Gender differences in mathematics performance Fig I.2.25
-50
-40
-30
-20
-10
0
10
20
30
Jo
rdan
Qata
rT
haila
nd
Ma
laysia
Icela
nd
U.A
.E.
Latv
iaS
inga
po
reF
inla
nd
Sw
ede
nB
ulg
aria
Ru
ssia
n F
ed.
Alb
an
iaM
on
ten
eg
roL
ithu
ania
Kaza
kh
sta
nN
orw
ay
Ma
ca
o-C
hin
aS
loven
iaR
om
ania
Pola
nd
Ind
one
sia
Un
ited
Sta
tes
Esto
nia
Ch
inese
Taip
ei
Sha
ngh
ai-
Ch
ina
Belg
ium
Turk
ey
Gre
ece
Fra
nce
Hung
ary
Serb
iaS
lovak R
epu
blic
Vie
tna
mC
ana
da
Ne
the
rla
nds
OE
CD
ave
rag
eP
ort
ug
al
Uru
gua
yC
roa
tia
Isra
el
Cze
ch R
ep
ub
licA
ustr
alia
Un
ited
Kin
gd
om
Sw
itze
rla
nd
Germ
any
Arg
en
tin
aD
enm
ark
Me
xic
oN
ew
Ze
ala
nd
Tunis
iaIr
ela
nd
Ho
ng
Kon
g-C
hin
aS
pa
inB
razil
Ja
pa
nK
ore
aIt
aly
Peru
Austr
iaL
iech
tenste
inC
osta
Ric
aC
hile
Luxe
mb
ou
rgC
olo
mb
ia
Sc
ore
-po
int
dif
fere
nc
e (
bo
ys
-gir
ls)
Boys perform better than girls
Girls perform better than boys
21
Gender differences in reading performance
-80
-70
-60
-50
-40
-30
-20
-10
0Jo
rdan
Qata
rB
ulg
aria
Mo
nte
neg
roF
inla
nd
Slo
ven
iaU
.A.E
.L
ithu
ania
Th
aila
nd
Latv
iaS
wede
nIc
ela
nd
Gre
ece
Cro
atia
No
rwa
yS
erb
iaT
urk
ey
Germ
any
Isra
el
Fra
nce
Esto
nia
Pola
nd
Rom
ania
Ma
laysia
Ru
ssia
n F
ed.
Hung
ary
Slo
vak R
epu
blic
Port
ug
al
Italy
Cze
ch R
ep
ub
licA
rgen
tin
aO
EC
D a
ve
rag
eA
ustr
iaK
aza
kh
sta
nS
witze
rla
nd
Ma
ca
o-C
hin
aU
rug
ua
yC
ana
da
Austr
alia
Ne
w Z
eala
nd
Ch
inese
Taip
ei
Sin
ga
po
reB
elg
ium
Vie
t N
am
Un
ited
Sta
tes
Denm
ark
Tunis
iaB
razil
Luxe
mb
ou
rgS
pa
inIr
ela
nd
Ind
one
sia
Ne
the
rla
nds
Hong
Kon
g-C
hin
aC
osta
Ric
aU
nited
Kin
gd
om
Lie
ch
tenste
inJa
pa
nS
ha
ngh
ai-
Ch
ina
Me
xic
oK
ore
aC
hile
Peru
Co
lom
bia
Alb
an
ia
Sc
ore
-po
int
dif
fere
nce
(b
oys
-gir
ls)
In all countries and economies
girls perform better than boys
Fig I.4.12 22
The share of immigrant students in OECD countries increased from 9% in 2003 to 12% in 2012…
…while the performance disadvantage of immigrant students reduced by 11 score points during the same period (after
accounting for socio-economic factors)
23
-100
-50
0
50
100
150
Austr
alia
Isra
el
United S
tate
sH
un
gary
Hon
g K
ong-C
hin
aS
ingapore
Slo
va
k R
epublic
Can
ada
New
Zea
lan
dC
hin
ese T
aip
ei
Ire
lan
dT
urk
ey
Un
ite
d K
ing
do
mLuxem
bo
urg
Latv
iaC
hile
Arg
entina
Ru
ssia
n F
ed
.C
zech R
ep
ublic
OE
CD
avera
ge
Germ
any
Gre
ece
Norw
ay
Esto
nia
Icela
nd
Italy
Austr
iaN
eth
erla
nd
sS
pain
Fra
nce
Port
ugal
Sw
ed
en
Den
mark
Sw
itze
rla
nd
Belg
ium
Bra
zil
Fin
land
Mexic
oS
han
ghai-
Chin
a
Sco
re p
oin
t d
iffe
ren
ce
before accounting for students' socio-economic status after accounting for students' socio-economic status
Differences in mathematics performance between
students without and with an immigrant background
Students without an immigrant
background perform better
Students with an immigrant
background perform better
Fig II.3.4 24
Disciplinary climate improved
Teacher-student relations improved between 2003 and 2012 in all but one country; and disciplinary climate also improved during the period,
on average across OECD countries and in 27 individual countries
25
Hong Kong-China
Brazil
Uruguay
Albania
Latvia
Lithuania
Chinese Taipei
Thailand Bulgaria
Jordan
UAE Argentina Indonesia
Kazakhstan
Peru
Costa Rica
Montenegro
Tunisia
Qatar
Singapore
Colombia
Malaysia
Russian Fed.
Romania
Viet Nam
Shanghai-China
USA
Poland
New Zealand
Greece
UK
Estonia Finland
Luxembourg
Germany
Austria
Czech Rep.
France
Japan
Turkey Sweden
Hungary Australia
Canada
Chile
Belgium Netherlands
Spain
Switzerland Slovenia
Portugal
Norway
Mexico
Korea
Italy
R² = 0.16
300
350
400
450
500
550
600
650
0 10 20 30 40 50 60 70
Ma
the
ma
tics
perf
orm
an
ce
(sc
ore
po
ints
)
Percentage of students in schools who skipped at least one day of school in the two weeks prior to the PISA test
Countries with large proportions of truants
perform worse in mathematics Fig IV.1.22
Social and emotional dimensions matter too
Students’ Engagement, Drive and Self-Beliefs
are all related to their performance
28
0 20 40 60 80 100
Agree: I feel like I belong at school
Disagree: I feel lonely at school
Agree: I feel happy at school
Agree: Things are ideal in my school
Agree: I am satisfied with my school
%
Korea OECD average United Kingdom
Students' sense of belonging
Percentage of students who agree/disagree with the following statements:
Fig III.2.12 29
Students’ mathematics self-efficacy
Percentage of students who feel very confident or confident about having to do the foll
owing tasks in mathematics:
30 40 50 60 70 80 90 100
Using a <train timetable> to work out howlong it would take to get from one place…
Calculating how much cheaper a TV wouldbe after a 30% discount
Calculating how many square metres oftiles you need to cover a floor
Understanding graphs presented innewspapers
Solving an equation like 3x+5=17
Finding the actual distance between twoplaces on a map with a 1:10 000 scale
Solving an equation like 2(x+3)=(x+3)(x-3)
Calculating the petrol-consumption rate ofa car
%
Korea OECD average United Kingdom
Fig III.4.2 30
Money makes a difference…
…but only up to a point
31
Spending per student from the age of 6 to 15 and
mathematics performance in PISA 2012
Slovak Republic
Czech Republic Estonia
Israel
Poland
Korea
Portugal
New Zealand
Canada Germany
Spain
France
Italy
Singapore
Finland
Japan
Slovenia Ireland
Iceland
Netherlands
Sweden
Belgium
UK
Australia Denmark
United States
Austria
Norway
Switzerland
Luxembourg
Viet Nam
Jordan
Peru
Thailand
Malaysia
Uruguay
Turkey
Colombia
Tunisia
Mexico Montenegro
Brazil
Bulgaria
Chile
Croatia Lithuania
Latvia
Hungary
Shanghai-China
R² = 0.01
R² = 0.37
300
350
400
450
500
550
600
650
0 20 000 40 000 60 000 80 000 100 000 120 000 140 000 160 000 180 000 200 000
Ma
the
ma
tics
perf
orm
an
ce
(sc
ore
po
ints
)
Average spending per student from the age of 6 to 15 (USD, PPPs)
Cumulative expenditure per student less than USD 50 000
Cumulative expenditure per student USD 50 000 or more
Fig IV.1.8
Governance matters Schools with more autonomy over curricula and assessments tend to
perform better than schools with less autonomy where they are part of school systems with more accountability arrangements and greater
teacher-principal collaboration in school management
33
Schools with more autonomy perform better than schools with
less autonomy in systems with more collaboration
Teachers don't participate inmanagement
Teachers participate inmanagement455
460
465
470
475
480
485
Less school autonomy
More school autonomy
Score points
School autonomy for resource allocation x System's level of teachers
participating in school management
Across all participating countries and economies
Fig IV.1.17
Lessons from PISA on successful
education systems
Thank you !
Find out more about PISA at www.pisa.oecd.org
• All national and international publications
• The complete micro-level database
Email: [email protected]
Do you have an idea on how to use this data to improve education in your country?
Would you like to work with us to develop that idea?
Apply to the Thomas J. Alexander
fellowship programme!
http://www.oecd.org/edu/thomasjalexanderfellowship.htm
