PISA 2009 Evaluating systems to improve education

11P

ISA

OE

CD

Pro

gram

me

for

Inte

rnat

iona

l Stu

dent

Ass

essm

ent

Stu

dent

s O

n Li

ne PISA 2009Evaluating systems to improve

education

Programme for International Student Assessment

The yardstick for success is no longer improvement by national standards alone but the best performing

education systems

22P

ISA

OE

CD

Pro

gram

me

for

Inte

rnat

iona

l Stu

dent

Ass

essm

ent

Stu

dent

s O

n Li

ne

1998PISA countries in

2000200120032006200977%81%83%85%86%

Coverage of world economy 87%

PISA 2009 in brief

• Over half a million students… representing 28 million 15-year-olds in 74*

countries/economies

… took an internationally agreed 2-hour test… Goes beyond testing whether students can

reproduce what they were taught……to assess students’ capacity to extrapolate from what

they know and creatively apply their knowledge in novel situations

…and responded to questions on… their personal background, their schools

and their engagement with learning and school• Parents, principals and system leaders provided data

on… school policies, practices, resources and institutional

factors that help explain performance differences .

* Data for Costa Rica, Georgia, India, Malaysia, Malta, Mauritius, Venezuela and Vietnam will be published in December 2011

1960 1970 1980 1990 200240

45

50

55

60

65Routine manual

Nonroutine manual

Routine cog-nitive

Nonroutine ana-lytic

Nonroutine interactive

Changes in skill demand

33P

ISA

OE

CD

Pro

gram

me

for

Inte

rnat

iona

l Stu

dent

Ass

essm

ent

Stu

dent

s O

n Li

ne

1998PISA countries in

2000200120032006200977%81%83%85%86%

Coverage of world economy 87%

PISA 2009 in brief• Key principles

‘Crowd sourcing’ and collaboration– PISA draws together leading expertise and institutions from

participating countries to develop instruments and methodologies…

… guided by governments on the basis of shared policy interests

Cross-national relevance and transferability of policy experiences

– Emphasis on validity across cultures, languages and systems– Frameworks built on well-structured conceptual

understandingof assessment areas and contextual factors

Triangulation across different stakeholder perspectives– Systematic integration of insights from students, parents,

school principals and system-leaders Advanced methods with different grain sizes

– A range of methods to adequately measure intended constructs with different grain sizes to serve different decision-making needs

– Productive feedback, at appropriate levels of detail, to fuel improvement at multiple levels .

44P

ISA

OE

CD

Pro

gram

me

for

Inte

rnat

iona

l Stu

dent

Ass

essm

ent

Stu

dent

s O

n Li

ne

What 15-year-olds can do

55P

ISA

OE

CD

Pro

gram

me

for

Inte

rnat

iona

l Stu

dent

Ass

essm

ent

Stu

dent

s O

n Li

neAverage performanceof 15-year-olds in reading – extrapolate and apply

High reading performance

Low reading performance … 17 countries perform below this line

1525354555440.000

460.000

480.000

500.000

520.000

540.000

560.000

Shanghai-China

KoreaFinlandHong Kong-China

Singapore CanadaNew Zealand

JapanAustralia

NetherlandsBelgiumNorway, EstoniaSwitzerlandPoland,IcelandUnited States LiechtensteinSwedenGermany,

IrelandFrance, Chinese TaipeiDenmarkUnited KingdomHungary,Portugal

Macao-China ItalyLatvia

Slovenia GreeceSpain

Czech RepublicSlovak Republic, CroatiaIsraelLuxembourg,

Austria LithuaniaTurkey

Dubai (UAE) Russian Federation

Chile

Serbia

66P

ISA

OE

CD

Pro

gram

me

for

Inte

rnat

iona

l Stu

dent

Ass

essm

ent

Stu

dent

s O

n Li

neAverage performanceof 15-year-olds in science – extrapolate and apply

Low average performance

Large socio-economic disparities

High average performance



High social equity


High social equity

Strong socio-economic impact on

student performance

Socially equitable distribution of

learning opportunities


Low reading performance

77P

ISA

OE

CD

Pro

gram

me

for

Inte

rnat

iona

l Stu

dent

Ass

essm

ent

Stu

dent

s O

n Li

ne

Durchschnittliche Schülerleistungen im Bereich Mathematik






High social equity


High social equity


student performance





AustraliaBelgiumCanadaChileCzech RepDenmarkFinlandGermanyGreeceHungaryIcelandIrelandIsraelItalyJapanKoreaLuxembourgMexicoNetherlandsNew ZealandNorwayPolandPortugalSpainSwedenSwitzerlandUKUS

2009

1525354555

2009

88P

ISA

OE

CD

Pro

gram

me

for

Inte

rnat

iona

l Stu

dent

Ass

essm

ent

Stu

dent

s O

n Li

ne







High social equity


High social equity


student performance






2009

99P

ISA

OE

CD

Pro

gram

me

for

Inte

rnat

iona

l Stu

dent

Ass

essm

ent

Stu

dent

s O

n Li

ne

Port

ug

al

Sp

ain

Sw

itze

rlan

d

Belg

ium

Kore

a

Lu

xem

bou

rg

Germ

an

y

Gre

ece

Jap

an

Au

stra

lia

Un

ited

Kin

gd

om

New

Zeala

nd

Fra

nce

Neth

erl

an

ds

Den

mark

Italy

Au

stri

a

Cze

ch

Rep

ub

lic

Hu

ng

ary

Norw

ay

Icela

nd

Irela

nd

Mexic

o

Fin

lan

d

Sw

ed

en

Un

ited

Sta

tes

Pola

nd

Slo

vak R

ep

ub

lic

-10

-5

0

5

10

15

Salary as % of GDP/capita Instruction time 1/teaching time 1/class sizePort

ug

al

Sp

ain

Sw

itze

rlan

d

Belg

ium

Kore

a

Lu

xem

bou

rg

Germ

an

y

Gre

ece

Jap

an

Au

stra

lia

Un

ited

Kin

gd

om

New

Zeala

nd

Fra

nce

Neth

erl

an

ds

Den

mark

Italy

Au

stri

a

Cze

ch

Rep

ub

lic

Hu

ng

ary

Norw

ay

Icela

nd

Irela

nd

Mexic

o

Fin

lan

d

Sw

ed

en

Un

ited

Sta

tes

Pola

nd

Slo

vak R

ep

ub

lic

-10

-5

0

5

10

15

Difference with OECD average

High performing systems often prioritize the quality of teachers over the size of classes

Contribution of various factors to upper secondary teacher compensation costsper student as a percentage of GDP per capita (2004)

Percentage points

1010P

ISA

OE

CD

Pro

gram

me

for

Inte

rnat

iona

l Stu

dent

Ass

essm

ent

Stu

dent

s O

n Li

ne







High social equity


High social equity


student performance






2009

1111P

ISA

OE

CD

Pro

gram

me

for

Inte

rnat

iona

l Stu

dent

Ass

essm

ent

Stu

dent

s O

n Li

ne







High social equity


High social equity


student performance






2000

1212P

ISA

OE

CD

Pro

gram

me

for

Inte

rnat

iona

l Stu

dent

Ass

essm

ent

Stu

dent

s O

n Li

ne







High social equity


High social equity


student performance






2000

Other rapid improvers in reading:Peru, Indonesia, Latvia, Israel and Brazil

Rapid improvers in mathematics:Mexico, Brazil, Turkey, Greece, Portugal, Italy

and GermanyRapid improvers in science:

Qatar, Turkey, Portugal, Korea, Brazil, Colombia, Italy, Norway, United States, Poland

1313P

ISA

OE

CD

Pro

gram

me

for

Inte

rnat

iona

l Stu

dent

Ass

essm

ent

Stu

dent

s O

n Li

ne

Changes in performance by type of taskIncrease percentage correct

Multiple-choice - reproducing knowledge

Open-ended - constructing knowledge

0

1

2

3

4

5

6

7

8

9

10

0.8

1.71.7

6.5

OECD Japan

200620062009

2009

1414P

ISA

OE

CD

Pro

gram

me

for

Inte

rnat

iona

l Stu

dent

Ass

essm

ent

Stu

dent

s O

n Li

neChanges in percentage of students who reported having a computer at

home between PISA 2000 and 2009

Lie

ch

ten

ste

in

De

nm

ark

Fin

lan

d

Ice

lan

d

No

rwa

y

Sw

ed

en

Sw

itze

rla

nd

Ho

ng

Ko

ng

-Ch

ina

Ko

rea

Ge

rma

ny

Au

str

alia

Au

str

ia

Ca

na

da

Be

lgiu

m

Po

rtu

ga

l

Cz

ec

h R

ep

ub

lic

Ire

lan

d

Fra

nc

e

Ita

ly

Ne

w Z

ea

lan

d

Isra

el

Po

lan

d

OE

CD

av

era

ge

- 2

7

Hu

ng

ary

Un

ited

Sta

tes

Sp

ain

La

tvia

Gre

ec

e

Ja

pa

n

Bu

lga

ria

Ro

ma

nia

Ru

ss

ian

Fe

de

ratio

n

Ch

ile

Arg

en

tina

Th

aila

nd

Bra

zil

Me

xic

o

Alb

an

ia

Pe

ru

Ind

on

es

ia

0

10

20

30

40

50

60

70

80

90

100

2009 2000Percentage of students

1515P

ISA

OE

CD

Pro

gram

me

for

Inte

rnat

iona

l Stu

dent

Ass

essm

ent

Stu

dent

s O

n Li

ne

Au

stri

aN

orw

ay

Un

ited

Sta

tes

Ne

w Z

ea

lan

dA

ust

ralia

Jap

an

Sw

ed

en

Hu

ng

ary

Ko

rea

Sw

itze

rla

nd

De

nm

ark

Be

lgiu

mO

EC

D a

vera

ge

-25

Fin

lan

dH

on

g K

on

g-C

hin

aC

zech

Re

pu

blic

Ice

lan

dL

iech

ten

ste

inIr

ela

nd

Po

rtu

ga

lIta

lyP

ola

nd

Ge

rma

ny

Me

xico

Isra

el

Th

aila

nd

La

tvia

Ro

ma

nia

Gre

ece

Sp

ain

Bu

lga

ria

Pe

ruR

uss

ian

Fe

de

ratio

nC

hile

Ind

on

esi

aA

lba

nia

Arg

en

tina

Bra

zil0.00

0.05

0.10

0.15

0.20

0.25

2009 2000

Mean Index

Changes in computers-per-student ratio between PISA 2000 and 2009

1616P

ISA

OE

CD

Pro

gram

me

for

Inte

rnat

iona

l Stu

dent

Ass

essm

ent

Stu

dent

s O

n Li

ne

Why PISA 2009 included a digital reading assessment

• “Digital reading” means reading on screen, mostly via the Internet

• Digital reading reading on line is an increasingly important form of reading in the 21st century

• Digital reading has different features to print reading and makes new demands on readers

1717P

ISA

OE

CD

Pro

gram

me

for

Inte

rnat

iona

l Stu

dent

Ass

essm

ent

Stu

dent

s O

n Li

neHow proficient are students in digital reading?

Ko

rea

Jap

an

Au

stra

lia

Ho

ng

Ko

ng

-Ch

ina

Ne

w Z

ea

lan

d

Ma

cao

-Ch

ina

Ire

lan

d

Ice

lan

d

Sw

ed

en

No

rwa

y

Be

lgiu

m

De

nm

ark

Fra

nce

OE

CD

ave

rag

e -

16

Sp

ain

Po

lan

d

Hu

ng

ary

Au

stri

a

Ch

ile

Co

lom

bia

80

60

40

20

0

20

40

60

80

100

Tasks at Level 5 or above typically require the reader to locate, analyse and critically evaluate information, related to an unfamiliar context, in the presence of ambiguity. They require criteria to evaluate the text. Tasks may require navigation across multiple sites without explicit direction, and detailed interrogation of texts in a variety of formats.

Tasks at Level 3 require that the reader integrate information, either by navigating across several sites to find well-defined target information, or by generating simple categories when the task is not explicitly stated. Where evaluation is called for, only the information that is most directly accessible or only part of the available information is required.

Tasks at Level 4 may require the reader to evaluate information from several sources, navigating across several sites comprising texts in a variety of formats, and generating criteria for evaluation in relation to a familiar, personal or practical context. Other tasks at this level demand that the reader interpret complex information according to well-defined criteria in a scientific or technical context.

Percentage of students

Tasks at Level2 typically require the reader to locate and interpret information that is well-defined, usually relating to familiar contexts. They may require navigation across a limited number of sites and the application of web-based navigation tools such as dropdown menus, where explicit directions are provided or only low-level inference is called for. Tasks may require integrating information presented in different formats, recognising examples that fit clearly defined categories.

Students performing below level 2 can scroll and navigate across web pages, as long as explicit directions are provided, and can locate simple pieces of information in a short block of hypertext. Nevertheless, although the digital reading skills of these students are not necessarily negligible, they are performing at levels that are not likely to allow them full access to educational, employment and social opportunities in the 21st century.

1818P

ISA

OE

CD

Pro

gram

me

for

Inte

rnat

iona

l Stu

dent

Ass

essm

ent

Stu

dent

s O

n Li

ne

0 10 20 30 40 500

10

20

30

40

50

Differences between boys and girls compared with print reading

Gender difference in digital reading performance (girls-boys)

Gender difference in print reading performance (girls- boys)

Colombia

Denmark

Hong Kong -ChinaMacao - China

Korea

Spain

New ZealandNorway

Poland

Ireland

Iceland

Australia

Sweden

OECD Average

Belgium

Japan

Austria

Hungary

France

Chile

Girls have a larger advantage in print reading

Girls have a larger advantage in digital reading

1919P

ISA

OE

CD

Pro

gram

me

for

Inte

rnat

iona

l Stu

dent

Ass

essm

ent

Stu

dent

s O

n Li

ne

Navigation

• Navigation is considered to be part of the cognitive process of digital reading

• Tracking and analysing the sequences of pages students visit provide insights into effective navigation behaviours in digital reading

• The index number of relevant pages visited describes how many of the pages judged to be relevant to a task were accessed while the student worked on that .

2020P

ISA

OE

CD

Pro

gram

me

for

Inte

rnat

iona

l Stu

dent

Ass

essm

ent

Stu

dent

s O

n Li

ne

Number of relevant pages visited and digital reading performance

30 35 40 45 50 55350

400

450

500

550

600

Australia

Austria

Belgium

Chile

Colombia

MacaoSpain

France

Ireland

Hungary

Sweden

Hong Kong Japan

Korea

Denmark

Norway

New Zealand

Iceland

Poland

Number of relevant pages visited

Digital reading score

The index number of relevant pages visited describes how many of the pages judged to

be relevant to a task were accessed while the student

worked on that task

2323P

ISA

OE

CD

Pro

gram

me

for

Inte

rnat

iona

l Stu

dent

Ass

essm

ent

Stu

dent

s O

n Li

ne

ICT use at home for leisure and digital reading performance

never or hardly ever Once or twice a month

Once or twice a week

Everyday or almost everyday

450

460

470

480

490

500

510

520

Play one-player games

Play collaborative online games

Use E-mail

Chat on line

Browse the Internet for fun

Download music, films, games or software form the Internet

Publish and maintain a personal page, weblog or blog

Participate in online forums, vir-tual communities or spaces

Dig

ita

l R

ea

din

g P

erf

orm

an

ce

2424P

ISA

OE

CD

Pro

gram

me

for

Inte

rnat

iona

l Stu

dent

Ass

essm

ent

Stu

dent

s O

n Li

ne

ICT use at home for schoolwork and digital reading performance

Never or hardly ever

Once or twice a month


Everyday or almost every-

day

450

460

470

480

490

500

510

520

Browse the Internet for schoolwork

Use e-mail to communicate with other students about schoolwork

Use e-mail to communicate with teachers about schoolwork

Download, upload or browse mate-rial from your school's website

Check the school's website for announcements

Dig

ita

l R

ea

din

g P

erf

orm

an

ce

2525P

ISA

OE

CD

Pro

gram

me

for

Inte

rnat

iona

l Stu

dent

Ass

essm

ent

Stu

dent

s O

n Li

ne

ICT use at school and digital reading performance

Never or hardly ever

Once or twice a month


Everyday or almost ever-

day

360

380

400

420

440

460

480

500

520

Chat on line

Use e-mail

Browse the Internet for schoolwork

Download, upload or browse mate-rial from the school's website

Post your work on the school's website

Play simulations at school

Computer use at school for practice and drilling

Doing individual homework on a school computer

Group work and communication with other students

Dig

ita

l R

ea

din

g p

erf

orm

an

ce

2727P

ISA

OE

CD

Pro

gram

me

for

Inte

rnat

iona

l Stu

dent

Ass

essm

ent

Stu

dent

s O

n Li

ne

Some conclusions• Identifying effective strategies to teach digital reading

skills is an important policy objective Ability to critically evaluate the quality and credibility of

texts, integrate information from multiple texts and – crucially – navigate effectively

• ICT use at home for leisure is – up to a point – positively related to performance, navigation skills and self-confidence in completing high-level ICT tasks

Parents and teachers need to encourage students to use computers so that they can improve their navigation skills but also provide guidance on balancing time spent using computers with time for other activities

• Digital reading can be a lever to reduce the gender gap

The gender gap in digital reading is much smaller than in print reading, and relates to differences in navigation skills between boys and girls

Reading more and reading with enjoyment promotes better reading, and better reading fosters stronger engagement .

2828P

ISA

OE

CD

Pro

gram

me

for

Inte

rnat

iona

l Stu

dent

Ass

essm

ent

Stu

dent

s O

n Li

ne

Some conclusions• Access to computers has improved significantly, both at

home and at school, but important gaps remain between countries and social groups

Strategies that promote wider access to ICT at school can help minimise impact of social background on digital competency gaps, with possible consequences for future employment opportunities

• Schools can do more to integrate ICT into learning Opportunities for students to solve problems using ICT Reading methods that improve students’ ability to distinguish

between relevant and irrelevant material, and to structure, prioritise, distil and summarise texts

• ICT can… Enable students to obtain more regular feedback on their

learning processes Make students more active participants in learning processes

in classrooms and tailor these processes to individual students’ needs

Provide students with up-to-date access to the world’s current research and thinking .

2929P

ISA

OE

CD

Pro

gram

me

for

Inte

rnat

iona

l Stu

dent

Ass

essm

ent

Stu

dent

s O

n Li

ne

What does it all mean?

3030P

ISA

OE

CD

Pro

gram

me

for

Inte

rnat

iona

l Stu

dent

Ass

essm

ent

Stu

dent

s O

n Li

neCommitment to universal achievement

Goals, gateways, instructional

systems

Capacity at point of delivery

Incentives and accountability

Resources where they yield most

A learning system

Coherence

Lessons from PISA on successful

education systems

A commitment to education and the belief that competencies can be learned and therefore all children can achieve

Universal educational standards and personalisation as the approach to heterogeneity in the student body…

…as opposed to a belief that students have different destinations to be met with different expectations, and selection/stratification as the approach to heterogeneity

Clear articulation who is responsible for ensuring student success and to whom

3131P

ISA

OE

CD

Pro

gram

me

for

Inte

rnat

iona

l Stu

dent

Ass

essm

ent

Stu

dent

s O

n Li



systems




A learning system

Coherence


education systems

Clear ambitious goals that are shared across the system and aligned with high stakes gateways and instructional systems

Well established delivery chain through which curricular goals translate into instructional systems, instructional practices and student learning (intended, implemented and achieved)

High level of metacognitive content of instruction

3232P

ISA

OE

CD

Pro

gram

me

for

Inte

rnat

iona

l Stu

dent

Ass

essm

ent

Stu

dent

s O

n Li



systems




A learning system

Coherence


education systems

Capacity at the point of delivery Attracting, developing and retaining high

quality teachers and school leaders and a work organisation in which they can use their potential

Instructional leadership and human resource management in schools

Keeping teaching an attractive profession System-wide career development

3333P

ISA

OE

CD

Pro

gram

me

for

Inte

rnat

iona

l Stu

dent

Ass

essm

ent

Stu

dent

s O

n Li

ne


education systems

Commitment to universal achievement


systems




A learning system

Coherence

Incentives, accountability, knowledge management

Aligned incentive structuresFor students How gateways affect the strength, direction, clarity and nature

of the incentives operating on students at each stage of their education

Degree to which students have incentives to take tough courses and study hard

Opportunity costs for staying in school and performing well

For teachers Make innovations in pedagogy and/or organisation Improve their own performance

and the performance of their colleagues Pursue professional development opportunities

that lead to stronger pedagogical practices A balance between vertical and lateral accountability Effective instruments to manage and share knowledge

and spread innovation – communication within the system and with stakeholders around it

A capable centre with authority and legitimacy to act

3434P

ISA

OE

CD

Pro

gram

me

for

Inte

rnat

iona

l Stu

dent

Ass

essm

ent

Stu

dent

s O

n Li

ne

Systems with more accountability Systems with less

accountability

480

490

500

Schools with less autonomy

Schools with more autonomy

495

School autonomy in re-source allocation

System’s accountability arrangements

PISA score in reading

School autonomy, accountability and student performance

Impact of school autonomy on performance in systems with and without accountability arrangements

3535P

ISA

OE

CD

Pro

gram

me

for

Inte

rnat

iona

l Stu

dent

Ass

essm

ent

Stu

dent

s O

n Li



systems




A learning system

Coherence


education systems Investing resources where they can make

most of a difference Alignment of resources with key challenges

(e.g. attracting the most talented teachers to the most challenging classrooms)

Effective spending choices that prioritise high quality teachers over smaller classes

3636P

ISA

OE

CD

Pro

gram

me

for

Inte

rnat

iona

l Stu

dent

Ass

essm

ent

Stu

dent

s O

n Li



systems




A learning system

Coherence


education systems

A learning system An outward orientation of the system to

keep the system learning, international benchmarks as the ‘eyes’ and ‘ears’ of the system

Recognising challenges and potential future threats to current success, learning from them, designing responses and implementing these

3737P

ISA

OE

CD

Pro

gram

me

for

Inte

rnat

iona

l Stu

dent

Ass

essm

ent

Stu

dent

s O

n Li



systems




A learning system

Coherence


education systems

Coherence of policies and practices Alignment of policies

across all aspects of the system Coherence of policies

over sustained periods of time Consistency of implementation Fidelity of implementation

(without excessive control)

3838P

ISA

OE

CD

Pro

gram

me

for

Inte

rnat

iona

l Stu

dent

Ass

essm

ent

Stu

dent

s O

n Li

ne

Some students learn at high levels

All students need to learn at high levels

Student inclusion

Routine cognitive skills, rote learning

Learning to learn, complex ways of

thinking, ways of workingCurriculum, instruction and assessment

Few years more than secondary

High-level professional knowledge workers

Teacher quality

‘Tayloristic’, hierarchical

Flat, collegial

Work organisation

Primarily to authorities

Primarily to peers and stakeholders

Accountability

Education reform trajectories

The old bureaucratic system The modern enabling system

3939P

ISA

OE

CD

Pro

gram

me

for

Inte

rnat

iona

l Stu

dent

Ass

essm

ent

Stu

dent

s O

n Li

ne

Moderate policy value

High policy value

More difficult Less difficult

Money pits

Must haves

Low-hanging fruits

Quick wins

PISA 2000

The development of PISA

Proliferation of assessment

areas .

Examining individual, institutional and systemic factors associated with quality, equity and efficiency in

education

Measuring student learning outcomes in

key subjects

Extending the range of competencies through which quality is assessed

Monitoring educational progress

Electronic delivery of assessments

Understanding drivers of successful reform trajectories

Understanding the instructional context of learning outcomes – linking how students learn

with what teachers do

‘Democratising PISA’

4040P

ISA

OE

CD

Pro

gram

me

for

Inte

rnat

iona

l Stu

dent

Ass

essm

ent

Stu

dent

s O

n Li

ne

Moderate policy value

High policy value

More difficult Less difficult

Money pits

Must haves

Low-hanging fruits

Quick wins

PISA 2003

Proliferation of assessment

areas .

Examining individual, institutional and systemic factors associated with quality, equity and efficiency in

education

Measuring student learning outcomes in

key subjects and establishing the comparative strengths and

weaknesses of education systems

Monitoring educational progress

Electronic delivery of assessments

Understanding drivers of successful reform trajectories

Understanding the instructional context of learning outcomes – linking how students learn

with what teachers do

Extending the range of competencies through which quality is assessed

‘Democratising PISA’

PISA 2009 Evaluating systems to improve education

Documents

Transcript of PISA 2009 Evaluating systems to improve education