Local, national and international benchmarking for improvement

11A

leja

ndro

Gom

ez P

alm

a, 4

Apr

il 20

13P

ISA

-bas

ed te

st fo

r sch

ools

Local, national and international

benchmarking for improvement

First results from the OECD test for schools based on PISA

School Board MeetingFairfax, VA

4 April 2013Alejandro Gomez Palma

Policy Analyst / PISA Instrument DevelopmentDirectorate for Education and Skills

Programme for International Student Assessment

22A

leja

ndro

Gom

ez P

alm

a, 4

Apr

il 20

13P

ISA

-bas

ed te

st fo

r sch

ools

In the dark all schools and school systems look the same…

But with a little light….

33A

leja

ndro

Gom

ez P

alm

a, 4

Apr

il 20

13P

ISA

-bas

ed te

st fo

r sch

ools

But with a little light….…important differences become apparent….

In the dark all schools and school sytems look the same…

44A

leja

ndro

Gom

ez P

alm

a, 4

Apr

il 20

13P

ISA

-bas

ed te

st fo

r sch

ools

A world of change in the global talent pool

Approximated by percentage of persons with high school or equivalent qualfications in the age groups 55-64, 45-55, 35-44 und 25-34 years

Uni

ted

Stat

es

Czec

h Re

publ

ic

Esto

nia

Ger

man

ySw

itzer

land

Den

mar

k Ca

nada

N

orw

aySw

eden

Ru

ssia

n Fe

dera

tion4

Aust

ria3

Slov

enia

Isra

elSl

ovak

Rep

ublic

New

Zea

land

H

unga

ry

Finl

and

Uni

ted

King

dom

3N

ethe

rland

sLu

xem

bour

g EU

19 a

vera

geO

ECD

ave

rage

Fran

ceAu

stra

lia

Icel

and

Belg

ium

Pola

ndIr

elan

dKo

rea

Chile

2G

reec

eIt

aly

Spai

nTu

rkey

Po

rtug

al

Mex

ico

Braz

il2

0

10

20

30

40

50

60

70

80

90

100

1990s 1980s 1970s 1960s

%

1. Excluding ISCED 3C short programmes 2. Year of reference 20043. Including some ISCED 3C short programmes 3. Year of reference 2003.

13

1

1

27

66A

leja

ndro

Gom

ez P

alm

a, 4

Apr

il 20

13P

ISA

-bas

ed te

st fo

r sch

ools

1998PISA countries in 20002001200320062009

77%81%83%85%86%Coverage of world economy 87%

77A

leja

ndro

Gom

ez P

alm

a, 4

Apr

il 20

13P

ISA

-bas

ed te

st fo

r sch

ools

Mathematics in PISAThe real world The mathematical World

A real situation

A model of reality A mathematical model

Mathematical results

Real results

Understanding, structuring and simplifying the situation

Making the problem amenable to mathematical

treatment

Interpreting the mathematical results

Using relevant mathematical tools to solve the problemValidating

the results

88A

leja

ndro

Gom

ez P

alm

a, 4

Apr

il 20

13P

ISA

-bas

ed te

st fo

r sch

ools

Average performanceof 15-year-olds in reading – extrapolate and apply

High reading performance

Low reading performance … 17 countries perform below this line

1525354555440.000

460.000

480.000

500.000

520.000

540.000

560.000Shanghai-China

KoreaFinlandHong Kong-China

Singapore CanadaNew ZealandJapanAustralia

NetherlandsBelgium Norway, EstoniaSwitzerlandPoland, IcelandUnited States LiechtensteinSwedenGermany,IrelandFrance, Chinese TaipeiDenmarkUnited KingdomHungary,

PortugalMacao-China ItalyLatviaSlovenia GreeceSpainCzech RepublicSlovak Republic, Croatia

IsraelLuxembourg,Austria Lithuania

TurkeyDubai (UAE) Russian Federation

Chile

Serbia

NortheastMidwest

WestSouth

Urban schools

Suburban schools

Performance distribution in US18% do not reach baseline Level 2 (16% when excluding immigrants) (Finland 6%, Canada 9%)

Economic cost: 72 trillion $10% are top performers (Shanghai 20%)

Woodson High School (W.T.), serves middle-class studentsLangley High School, serves upper-class neighbourhoods

Not j

ust a

bout

poo

r sc

hool

s in

poor

ne

ighb

ourh

oods

99A

leja

ndro

Gom

ez P

alm

a, 4

Apr

il 20

13P

ISA

-bas

ed te

st fo

r sch

ools




1525354555440.000

460.000

480.000

500.000

520.000

540.000








Chile

Serbia

NortheastMidwest

WestSouth

Urban schools

Suburban schools



North Star Academy College Preparatory High School, serves students in disadvantaged neighbourhoods (social context like in Chile, Portugal or Shanghai)


Perf

orm

ance

on

PISA

sca

le

300

400

500

600

700

Schools in theUnited States

Schools in theShanghai-China

Schools in Mexico

PISA 2009 Results

300

400

500

600

700

North Star Academy

Pilot Results

10% above

25% above

50% above/below

25% below

10% below

10% above

25% above

50% above/below

25% below

10% below10% above

25% above

50% above/below

25% below

10% below

Reading

300

400

500

600

700

300

400

500

600

700

300

400

500

600

700

300

400

500

600

700

Woodson HS

300

400

500

600

700

BASIS Scottsdale

300

400

500

600

700

Langley High School

300

400

500

600

700

Oakton High School

300

400

500

600

700

BASIS Tucson

1010A

leja

ndro

Gom

ez P

alm

a, 4

Apr

il 20

13P

ISA

-bas

ed te

st fo

r sch

ools




1525354555440.000

460.000

480.000

500.000

520.000

540.000








Chile

Serbia

NortheastMidwest

WestSouth

Urban schools

Suburban schools




1111A

leja

ndro

Gom

ez P

alm

a, 4

Apr

il 20

13P

ISA

-bas

ed te

st fo

r sch

ools

Average performanceof 15-year-olds in science – extrapolate and apply

Low average performanceLarge socio-economic disparities

High average performanceLarge socio-economic disparities

Low average performanceHigh social equity

High average performanceHigh social equity

Strong socio-economic impact on

student performance

Socially equitable distribution of

learning opportunities


Low reading performance

1212A

leja

ndro

Gom

ez P

alm

a, 4

Apr

il 20

13P

ISA

-bas

ed te

st fo

r sch

ools

Durchschnittliche Schülerleistungen im Bereich Mathematik






student performance





AustraliaBelgiumCanadaChileCzech RepDenmarkFinlandGermanyGreeceHungaryIcelandIrelandIsraelItalyJapanKoreaLuxembourgMexicoNetherlandsNew ZealandNorwayPolandPortugalSpainSwedenSwitzerlandUKUS

2009

1525354555

2009

1313A

leja

ndro

Gom

ez P

alm

a, 4

Apr

il 20

13P

ISA

-bas

ed te

st fo

r sch

ools







student performance






2009

1414A

leja

ndro

Gom

ez P

alm

a, 4

Apr

il 20

13P

ISA

-bas

ed te

st fo

r sch

ools

Portu

gal

Spain

Switz

erlan

d

Belg

ium

Kore

a

Luxe

mbo

urg

Germ

any

Gree

ce

Japa

n

Aust

ralia

Unite

d Ki

ngdo

m

New

Zeala

nd

Fran

ce

Neth

erlan

ds

Denm

ark

Italy

Aust

ria

Czec

h Re

publ

ic

Hung

ary

Norw

ay

Icela

nd

Irelan

d

Mexic

o

Finlan

d

Swed

en

Unite

d St

ates

Polan

d

Slov

ak R

epub

lic

-10

-5

0

5

10

15

Salary as % of GDP/capita Instruction time 1/teaching time 1/class sizePo

rtuga

l

Spain

Switz

erlan

d

Belg

ium

Kore

a

Luxe

mbo

urg

Germ

any

Gree

ce

Japa

n

Aust

ralia

Unite

d Ki

ngdo

m

New

Zeala

nd

Fran

ce

Neth

erlan

ds

Denm

ark

Italy

Aust

ria

Czec

h Re

publ

ic

Hung

ary

Norw

ay

Icela

nd

Irelan

d

Mexic

o

Finlan

d

Swed

en

Unite

d St

ates

Polan

d

Slov

ak R

epub

lic

-10

-5

0

5

10

15

Difference with OECD average

High performing systems often prioritize the quality of teachers over the size of classes

Contribution of various factors to upper secondary teacher compensation costsper student as a percentage of GDP per capita (2004)

Percentage points

1515A

leja

ndro

Gom

ez P

alm

a, 4

Apr

il 20

13P

ISA

-bas

ed te

st fo

r sch

ools







student performance






2009

1616A

leja

ndro

Gom

ez P

alm

a, 4

Apr

il 20

13P

ISA

-bas

ed te

st fo

r sch

ools







student performance






2000

1717A

leja

ndro

Gom

ez P

alm

a, 4

Apr

il 20

13P

ISA

-bas

ed te

st fo

r sch

ools







student performance






2000 - 2009

1818A

leja

ndro

Gom

ez P

alm

a, 4

Apr

il 20

13P

ISA

-bas

ed te

st fo

r sch

ools

Changes in performance by type of taskIncrease percentage correct

Multiple-choice - reproducing knowledge

Open-ended - constructing knowledge

0

1

2

3

4

5

6

7

8

9

10

0.81.71.7

6.5

OECD Japan

OECD OECDJapan

Japan

2020A

leja

ndro

Gom

ez P

alm

a, 4

Apr

il 20

13P

ISA

-bas

ed te

st fo

r sch

ools

Policies and practicesLearning climateDisciplineTeacher behaviourParental pressureTeacher-student relationshipsDealing with heterogeneityGrade repetitionPrevalence of trackingExpulsionsAbility grouping

(all subjects)Standards /accountabilityNat. examinationStandardised tests

Policy

System

R

School

R

Equity

E

2121A

leja

ndro

Gom

ez P

alm

a, 4

Apr

il 20

13P

ISA

-bas

ed te

st fo

r sch

ools

Learning from the World’s top performers

23232323P

ISA

OE

CD

Pro

gram

me

for

Inte

rnat

iona

l Stu

dent

Ass

essm

ent

Stro

ng p

erfo

rmer

s an

d su

cces

sful

refo

rmer

sA

ndre

as S

chle

iche

r13

Oct

ober

201

1

Commitment to universal achievement

Goals, gateways, instructional

systems

Capacity at point of delivery

Incentives and accountability

Resources where they yield most

A learning system

Coherence

Lessons from PISA on successful

education systems

A commitment to education and the belief that competencies can be learned and therefore all children can achieve

Universal educational standards and personalisation

…as opposed to a belief that students have different destinations to be met with different expectations, and selection/stratification as the approach to heterogeneity

Clear articulation who is responsible for ensuring student success and to whom

2424P

ISA

OE

CD

Pro

gram

me

for

Inte

rnat

iona

l Stu

dent

Ass

essm

ent

Stro

ng p

erfo

rmer

s an

d su

cces

sful

refo

rmer

sA

ndre

as S

chle

iche

rS

hang

hai,

24 J

uly

2012



systems




A learning system

Coherence


education systems

Clear ambitious goals that are shared across the system and aligned with high stakes gateways and instructional systems

Well established delivery chain through which curricular goals translate into instructional systems, instructional practices and student learning (intended, implemented and achieved)

High level of metacognitive content of instruction

2525P

ISA

OE

CD

Pro

gram

me

for

Inte

rnat

iona

l Stu

dent

Ass

essm

ent

Stro

ng p

erfo

rmer

s an

d su

cces

sful

refo

rmer

sA

ndre

as S

chle

iche

rS

hang

hai,

24 J

uly

2012



systems




A learning system

Coherence


education systems

Capacity at the point of delivery Attracting, developing and retaining high

quality teachers and school leaders and a work organisation in which they can use their potential

Instructional leadership and human resource management in schools and districts

Keeping teaching an attractive profession System-wide career development

Sahl, SIN

2626P

ISA

OE

CD

Pro

gram

me

for

Inte

rnat

iona

l Stu

dent

Ass

essm

ent

Stro

ng p

erfo

rmer

s an

d su

cces

sful

refo

rmer

sA

ndre

as S

chle

iche

rS

hang

hai,

24 J

uly

2012


education systems



systems




A learning system

Coherence Incentives, accountability, knowledge

management Aligned incentive structures

For students How gateways affect the strength, direction, clarity and nature

of the incentives operating on students at each stage of their education

Degree to which students have incentives to take tough courses and study hard

Opportunity costs for staying in school and performing wellFor teachers Make innovations in pedagogy and/or organisation Improve their own performance

and the performance of their colleagues Pursue professional development opportunities

that lead to stronger pedagogical practices A balance between vertical and lateral accountability Effective instruments to manage and share

knowledge and spread innovation – communication within the system and with stakeholders around it

A capable administration with authority and legitimacy to act

2727P

ISA

OE

CD

Pro

gram

me

for

Inte

rnat

iona

l Stu

dent

Ass

essm

ent

Stro

ng p

erfo

rmer

s an

d su

cces

sful

refo

rmer

sA

ndre

as S

chle

iche

rS

hang

hai,

24 J

uly

2012

Systems with more accountability Systems with less

accountability

480

490

500

Schools with less autonomy

Schools with more autonomy

495

School autonomy in re-source allocation

System’s accountability arrangements

PISA score in reading

School autonomy, accountability and student performance

Impact of school autonomy on performance in systems with and without accountability arrangements

Posting achievement data publicly

2828P

ISA

OE

CD

Pro

gram

me

for

Inte

rnat

iona

l Stu

dent

Ass

essm

ent

Stro

ng p

erfo

rmer

s an

d su

cces

sful

refo

rmer

sA

ndre

as S

chle

iche

rS

hang

hai,

24 J

uly

2012

Local responsibility and system-level prescription

System-level prescription‘Tayloristic’ work organisation

Schools leading reformTeachers as ‘knowledge workers’

Schools todayThe industrial

model, detailed prescription of

what schools do

Schools tomorrow?

Building capacity

Finland and other

improvers today

Every school an effective school

Trend in OECD countries

3030P

ISA

OE

CD

Pro

gram

me

for

Inte

rnat

iona

l Stu

dent

Ass

essm

ent

Stro

ng p

erfo

rmer

s an

d su

cces

sful

refo

rmer

sA

ndre

as S

chle

iche

rS

hang

hai,

24 J

uly

2012



systems




A learning system

Coherence


education systems Investing resources where they can make

most of a difference Alignment of resources with key challenges

(e.g. attracting the most talented teachers to the most challenging classrooms)

Effective spending choices that prioritise high quality teachers over smaller classes

CHN

3131P

ISA

OE

CD

Pro

gram

me

for

Inte

rnat

iona

l Stu

dent

Ass

essm

ent

Stro

ng p

erfo

rmer

s an

d su

cces

sful

refo

rmer

sA

ndre

as S

chle

iche

rS

hang

hai,

24 J

uly

2012



systems




A learning system

Coherence


education systems

A learning system An outward orientation to keep the

system learning, technology, international benchmarks as the ‘eyes’ and ‘ears’ of the system

Recognising challenges and potential future threats to current success, learning from them, designing responses and implementing these

SIN

3232P

ISA

OE

CD

Pro

gram

me

for

Inte

rnat

iona

l Stu

dent

Ass

essm

ent

Stro

ng p

erfo

rmer

s an

d su

cces

sful

refo

rmer

sA

ndre

as S

chle

iche

rS

hang

hai,

24 J

uly

2012



systems




A learning system

Coherence


education systems

Coherence of policies and practices Alignment of policies

across all aspects of the system Coherence of policies

over sustained periods of time Consistency of implementation Fidelity of implementation

(without excessive control)

CAN

3333A

leja

ndro

Gom

ez P

alm

a, 4

Apr

il 20

13P

ISA

-bas

ed te

st fo

r sch

ools

A tool for greater resolution for schools and local educators

PISA for Schools

In the U.S.:OECD Test for Schools

(based on PISA)

3535A

leja

ndro

Gom

ez P

alm

a, 4

Apr

il 20

13P

ISA

-bas

ed te

st fo

r sch

ools

The test is not…• A mandated standardised test• Intended to influence – in of itself – everyday

teaching practices• An accountability tool or a tool for “rankings” or

“league tables”

OECD Test for Schools (based on PISA)Uses of the assessment tool

Note: Size of bubbles is proportional to the number of students enrolled at the school.

Schools in the U.S. that participated in PISA 2009 Pilot Schools in the U.S.

-3.0 -2.0 -1.0 0.0 1.0 2.0 3.0250

300

350

400

450

500

550

600

650

700

750

-3.0 -2.0 -1.0 0.0 1.0 2.0 3.0250

300

350

400

450

500

550

600

650

700

750

-3.0 -2.0 -1.0 0.0 1.0 2.0 3.0250

300

350

400

450

500

550

600

650

700

750

U.S. average PISA 2009

U.S

. ave

rage

PI

SA 2

009

Socioeconomic status (ESCS)

Perf

orm

ance

on

PISA

sca

le

Reading

OECD Test for Schools (based on PISA)Overview of results from the pilot




Perf

orm

ance

on

PISA

sca

le

-3.0 -2.0 -1.0 0.0 1.0 2.0 3.0250

300

350

400

450

500

550

600

650

700

750Mathematics


U.S

. ave

rage

PI

SA 2

009






Perf

orm

ance

on

PISA

sca

le

-3.0 -2.0 -1.0 0.0 1.0 2.0 3.0250

300

350

400

450

500

550

600

650

700

750

-3.0 -2.0 -1.0 0.0 1.0 2.0 3.0250

300

350

400

450

500

550

600

650

700

750

-3.0 -2.0 -1.0 0.0 1.0 2.0 3.0250

300

350

400

450

500

550

600

650

700

750Science


U.S

. ave

rage

PI

SA 2

009

-3.0 -2.0 -1.0 0.0 1.0 2.0 3.0250

300

350

400

450

500

550

600

650

700

750

Schools with students from similar socioeconomic backgrounds

Perf

orm

ance

on

PISA

sca

le


300

400

500

600

700

Schools in theUnited States

Schools in theShanghai-China

Schools in Mexico

PISA 2009 Results

300

400

500

600

700

North Star Academy

Pilot Results

10% above

25% above

50% above/below

25% below

10% below

10% above

25% above

50% above/below

25% below

10% below10% above

25% above

50% above/below

25% below

10% below

Reading

300

400

500

600

700

300

400

500

600

700

300

400

500

600

700

300

400

500

600

700

Woodson HS

300

400

500

600

700

BASIS Scottsdale

300

400

500

600

700

Langley High School

300

400

500

600

700

Oakton High School

300

400

500

600

700

BASIS Tucson


What does the same mean mean?

Brazil

United States

United Kingdom

Poland

Japan

Korea

Shanghai-China

Oakton High School

0%20%40%60%80%100%

Brazil

Mexico

United States

OECD average

United Kingdom

Germany

Poland

Singapore

Japan

Canada

Korea

Finland

Shanghai-China

Langley High school

Oakton High School

0% 20% 40% 60% 80% 100%

543 and 543

Reading

Level 1 and be`low Level 2 Level 3 Level 4 Level 5 Level 6

4343A

leja

ndro

Gom

ez P

alm

a, 4

Apr

il 20

13P

ISA

-bas

ed te

st fo

r sch

ools

Content of reports

I. Introduction: Understanding your school’s results

II. What students in your school know and can do in Reading, Mathematics and Science

III. Student engagement and the learning environment at your school (teacher-student relations, disciplinary climate, student confidence and attitudes towards mathematics and science)

IV. Your School Compared with similar schools in Your Country

V. Your School’s Results in an international context

School nameSchool DistrictStateUnited States

How your school compares internationallyOECD Test for SchoolsPilot Trial 2012


4444A

leja

ndro

Gom

ez P

alm

a, 4

Apr

il 20

13P

ISA

-bas

ed te

st fo

r sch

ools

Receiving feedback… District-level information based on

school-level results Reporting on country-specific variables Examples of policies, practices and

insights based on feedback from pilot participants (and results from PISA 2012)

Peer-to-peer exchanges… examples of practices and network-specific follow-up

OECD Test for Schools (based on PISA)What next? Opportunities for development…

4545A

leja

ndro

Gom

ez P

alm

a, 4

Apr

il 20

13P

ISA

-bas

ed te

st fo

r sch

ools

OECD Test for Schools (based on PISA)What now? Next Steps

Starting in September 2013 (for testing) Interested schools and districts can begin

planning now

Availability of the assessment in the United States

To sign up:

http://www.oecd.org/pisa/pisa-basedtestforschools/

[email protected]

OECD will accredit a service provider



mailto:[email protected]

4646A

leja

ndro

Gom

ez P

alm

a, 4

Apr

il 20

13P

ISA

-bas

ed te

st fo

r sch

ools

Thank you !

Find out more about PISA at… OECD www.pisa.oecd.org

– All national and international publications– The complete micro-level database

Email: [email protected]

…and remember:Without data, you are just another person with an opinion

http://www.pisa.oecd.org/

4747A

leja

ndro

Gom

ez P

alm

a, 4

Apr

il 20

13P

ISA

-bas

ed te

st fo

r sch

ools

1 in 3 of the pilot schools are Title I 6 magnets, 6 charters and 1 private

school 3 of 4 schools with 20% or more of

students from diverse ethnic backgrounds

Several schools from the same districts and networks (EdLeader21, Fairfax,…)

105 Schools in the U.S.: 48 school districts in 22 states


Local, national and international benchmarking for improvement

Documents

Transcript of Local, national and international benchmarking for improvement