1

Programme for

International Student

Assessment (PISA) for

Development

Initial Technical Meeting

27 – 28 June 2013

J. Douglas Willms University of New Brunswick

Design and

Technical Issues

I. Measuring Student Performance

Successful

countries have

fewer vulnerable

children. Their skill

distribution is

negatively skewed.

Science Performance

Level 1 Level 3 Level 4 Level 5

Le

ve

l 6

700

Level 6

600500400300100 800

Level 2

200

Proportion of Students

Qatar

Below Level 1

Argentina

LithuaniaHong Kong

Below Level 1

-0.7 -0.6 -0.5 -0.4 -0.3 -0.2 -0.1 -0.0 0.1

300

350

400

450

500

550

Mean Reading Performance

Distribution Skewness

Mean grade 4 reading scores versus skewness of the distribution. Source: Prova Brazil 2007.

Student assessments cover a narrow range of skills.

The ability to

read is a

fundamental skill

for children’s

success at

school, and

ultimately for

their long-term

health and well-

being.

‘The simple view of

reading’ (Rose, 2006)

has two critical,

complementary

dimensions:

Code-related skills – the

ability to recognize and

understand particular

words, and

Language skills – being

able to understand and

interpret spoken and

written language.

The PISA framework is consistent with Bloom’s Taxonomy

for Assessment

Taxonomy for Teaching, Learning,

and Assessing

(revised Bloom’s taxonomy)

PISA Framework for Reading Literacy

Access and Retrieve Integrate and Interpret Reflect and Evaluate

Co

gn

itiv

e P

roc

es

sin

g D

ime

ns

ion

Creating (evaluation)

Locate and organize several

pieces of deeply embedded info

with precision and attention to

detail

Make multiple inferences,

comparisons and contrasts,

detailed and precise

Integrate information

from more than one text

Evaluating (synthesis)

Locate and organize several

pieces of deeply embedded info

Infer which information is

relevant

Critically evaluate or

hypothesize, draw on

specialized knowledge

Analyzing (analysis)

Locate and organize several

pieces of embedded info

Interpret nuances of

meaning, understand and

apply categories in

unfamiliar text

Make hypotheses,

critically evaluate a text

Applying (application)

Locate and recognize relationships

among several pieces

Integrate several pieces of

text to identify main idea,

understand a relationship,

or construe meaning

Understand connections,

comparisons,

explanations

Understanding (comprehension)

Locate 1 or more pieces of

information

Recognize main idea,

understand relationships,

construe meaning

Connect text with outside

knowledge

Remembering (knowledge)

Locate one piece of information in

simple text

Recognize main theme or

purpose in familiar text,

recognize simple idea.

Make simple connection

with everyday knowledge

LESSONS LEARNED: We cannot simply add more items at the lower end of the current PISA

framework, especially if we are to assess the skills of out-of-school students

Many students will not have successfully made the transition from learning-to-

read to reading-to-learn. We need measures of pre-literacy skills, especially

code-related and language skills.

.

15

II. Drivers of Student Performance

Success is Cumulative International test scores (e.g., TIMSS, PISA) are

the cumulative result of children’s learning at

home and at school since birth (or even earlier)

Learning is a function of:

Context

Quality Instruction

Learning Time

Engagement

17

Engagement is a function of:

Context

Quality Instruction

Learning Time

Learning

18

19

Context

Quality Instruction

Learning Time

Engagement begets Learning Learning begets Engagement

Learning

Engagement

LESSONS LEARNED: If we want to understand children’s academic success in low income

countries, we need to assess their opportunities for learning over their school

career. The traditional drivers of school climate measured at age 15 will not

be that helpful. We would learn more from a detailed assessment of school

resources and strong measures of the quality of teaching.

III. Intervention Forecasting

Early Years

Evaluation for

Responsive

Tiered Instruction

A prediction model based

on longitudinal data

LESSONS LEARNED:

The revised PISA academic measures can gives us a portrait of the end

results. However, if we can identify some of the key aspects of school human

and physical resources that students have been exposed to over their

schooling career, we can begin to build prediction models.

Such models are very powerful for allocating resources.

IV.

Allocation of

Resources

Use modern scaling

techniques to scale

variables and

identify cut-points

related to desired

standards.

33

34

Learning resource plot for Brazil

Correlation with SES

0.02

0.25

0.40

-0.16

0.30

0.02

-0.29

-0.20

0.65

-0.23

LESSONS LEARNED: We need a strong measure of socioeconomic

status that can be used cross-nationally.

We also need to build a more comprehensive

model of school resources. Imagine if we had

a model of physical and human resources

that was comparable to our achievement

measures, with 4 or 5 levels. For example,

schools without the basic physical

infrastructure (plumbing, heating, basic

teaching resources) would be at Level 1,

while schools with high levels of resources

(well-equipped library, trained teachers that

demonstrate high-yield strategies) would be

at Level 5.

V. Equality and Equity

Equality refers to differences in the distribution of outcomes among sub-populations, especially

between high- and low-status groups.

Equity refers to fairness – a just

treatment of people from different

sub-populations. 40

41

An Approach for

Assessing Quality,

Equality, and

Inequity

43 RR=1.72 RR=1.54

44 RR=1.28 RR=1.72

LESSONS LEARNED: There are large inequalities in achievement that are entrenched in the social

and economic realities of each country.

The earlier PISA results have taken us some way toward understanding the

extent of inequalities. To move forward we need stronger measures of

inequities - the factors associated with inequalities and how these are

distributed among schools.

5000 10000 15000 20000 25000 30000 350000

10

20

30

40

50

60

70

80

90

100

GDP (2000 International $)

Canada

U.S.

KoreaFinland

Percent

Youth with reading literacy scores below Level 3

Germany

Japan

The

Pathway

to

Success

Student Performance: measures of literacy skills

that stretch the scale down to pre-literacy skills

(e.g., code-related and language skills)

Drivers of Performance: measures that capture

students’ opportunities to learn from an early age.

Intervention Forecasting: models that will better

predict long-term outcomes.

Resource Allocation: a strong measure of

socioeconomic status that can be used cross-

nationally; a comprehensive model of school

resources.

Inequality and Inequity: stronger measures of

inequities - the factors associated with inequalities

and how these are distributed among schools.

http://www.unb.ca/crisp

http://www.ksiresearch.com

http://www.thelearningbar.com

53