Opening the ‘black box’ was melbourne 2 oct 2015

OPENING THE ‘BLACK BOX’: ASSESSING TEACHING AND LEARNING

EXCELLENCE AS A GAME-CHANGE

Dirk Van DammeOECD/EDU/IMEP

2

Inputs Black box Outputs

3


4

Financial inputs in higher education increasing

• Total (public & private) financial investment grew– Between 2005 and 2011 on average across

OECD increase of 10% in per student expenditure and 27% in total expenditure

– With huge differences between countries, increases higher in countries with below-average expenditure, catching up

– Yearly per student expenditure is now 14 KUS$– Total expenditure increased from 1.3% GDP in

2000 to 1.6% GDP in 2011

Inputs

5

-30 -20 -10 0 10 20 30 40 50 60 706 000

10 000

14 000

18 000

22 000

26 000 United States

Switzerland

DenmarkSweden

Norway FinlandNetherlands Germany

JapanIreland

BelgiumFranceAustria

Spain

Israel

United Kingdom

BrazilItaly

KoreaPolandPortugalCzech Republic

HungaryChileSlovak RepublicMexico Estonia

Iceland

Russian Federation

New Zealand

SloveniaR² = 0.0789910852395619

Change in expenditure per student between 2005 and 2011 (%)

Annual expenditure per student (2011,

USD)

OECD aver-age

OECD aver-age

Financial inputs in higher education increasingInputs

6

• Private expenditure has increased a lot– 31% of total expenditure (0.5% GDP) comes from

private sources, mainly tuition fees– Increase from 25% in 2000– Total private expenditure increased with 32%

since 2005– >50% in Israel, US, Australia, Japan, UK, Korea

and Chile

Financial inputs in higher education increasingInputs

7

• Increase in total per student expenditure slows down since crisis– Negative growth in almost half of countries

between 2008 and 2011– Expenditure cannot catch up with increasing

student numbers• Increasing concerns about levels of private

expenditure, student debt– 5 million international students investing even

much higher amounts of money

But strong signs of stagnating fundingInputs

8

Estonia

Slovak Republic

Chile

Hungary

Korea

Czech Republic

Finland

Slovenia

Denmark

Russian Federation

Israel

Japan

United Kingdom

Italy

Poland

OECD averageEU21 average

NetherlandsSwitzerland

France

Sweden

GermanyAustralia

BrazilSpain

NorwayMexico

Belgium

PortugalAustria

United States

IrelandIceland

80

90

10 0

11 0

12 0

13 0

14 0

15 0

Chan ge i n ex pen di tu re Chan ge i n the nu mbe r o f stu den ts (i n ful l-t ime e qui va le nts) Cha nge i n ex pe ndi ture pe r s tu de nt

In dex of c han ge (2 00 8= 100 )

Chart B1.6. Changes in the num be r of students a nd c ha nge s in e x pe nditure per s tude nt by educ ationa l ins titutions , by le ve l of educa tion (2 00 5 , 2 01 0)In d ex o f c ha ng e b etwee n 2 0 05 a n d 20 10 (20 05 = 10 0 , 2 01 0 co ns ta n t p ric es )


9


• Efficiency and value-for-money become very important policy considerations– Both for governments and students/families– Cost of higher education becoming political issue

in many countries• What are students actually ‘buying’?

– Very weak relationship between cost and actual ‘product’, benefits and outcomes

– Value-for-money depends enormously on institution and field of study

10


11


12

• Share of tertiary educated in the population increased, but still huge disparities across countries– 34% of adult population in OECD now has a

tertiary qualification, up from 22% in 2000

Outputs Increases in graduate output

13

Outputs Increases in graduate outputCh

ina

Sout

h Af

rica

Indo

nesia

Braz

ilTu

rkey

Italy

Chile

Mex

ico

Port

ugal

Slov

ak R

epub

licCz

ech

Repu

blic

Colo

mbi

aAu

stria

Hung

ary

Pola

ndSl

oven

iaGr

eece

Germ

any

Latv

iaEU

21 a

vera

geFr

ance

Spai

nO

ECD

aver

age

Neth

erla

nds

Denm

ark

Icel

and

Belg

ium

Swed

enSw

itzer

land

Esto

nia

Norw

ayLu

xem

bour

gFi

nlan

dIre

land

New

Zea

land

Unite

d Ki

ngdo

mAu

stra

liaKo

rea

Unite

d St

ates

Israe

lJa

pan

Cana

daRu

ssia

n Fe

dera

tion

0

10

20

30

40

50

602000 2012%

14

• Employment benefits remain high• On average across OECD academic degree gives a

wage premium of 75% compared to upper secondary educated workers

• Total net present value over lifetime is around 160 000 US$ for man, 100 000 US$ for woman– But with huge differences across fields of study

• Or internal rate of return of 16%• High social benefits in health, interpersonal trust,

volunteering, political participation

Outputs Employment, earnings and social benefits remain high

15

Outputs Employment, earnings and social benefits remain high

Turk

eyD

enm

ark

Spa

inE

ston

iaS

wed

enN

ew Z

eala

ndG

reec

eK

orea

Japa

nC

anad

aS

lova

k R

epub

licP

olan

dN

orw

ayIs

rael

Cze

ch R

epub

licFr

ance

Aus

tralia

Finl

and

OE

CD

ave

rage

Por

tuga

lE

U21

ave

rage

Aus

tria

Uni

ted

Kin

gdom

Net

herla

nds

Italy

Bel

gium

Slo

veni

aG

erm

any

Uni

ted

Sta

tes

Hun

gary

Irela

nd

0

50 000

100 000

150 000

200 000

250 000

300 000

350 000

400 000

450 000

500 000

Private net returns Public net returns

Equ

ival

ent U

SD

16

• Is graduate output higher than the economy’s need for high-skilled labour?– Graduate unemployment– Filtering-down effect?– Over-qualification and over-skilling– Huge field-of-study mismatches

• Is polarization in labour markets, with high employment/high earnings because of skill-biased technological change going to last?

Outputs But increasing concerns about quality of output

17


18


• Enormous differences in the skills equivalent of tertiary qualifications– Foundation skills such as literacy and numeracy– Can societies accept that higher education

graduates have low literacy and numeracy skills?

19


20

Japa

n

Finl

and

Net

herla

nds

Sw

eden

Aus

tralia

Nor

way

Flan

ders

(Bel

gium

)

Eng

land

(UK

)

Eng

land

/N. I

rela

nd (U

K)

Uni

ted

Sta

tes

Cze

ch R

epub

lic

OE

CD

ave

rage

Pol

and

Can

ada

Nor

ther

n Ire

land

(UK

)

Aus

tria

Ger

man

y

Irela

nd

Fran

ce

Den

mar

k

Est

onia

Slo

vak

Rep

ublic

Kor

ea

Rus

sian

Fed

erat

ion

Spa

in

Italy

0

5

10

15

20

25

30

35

40

Below upper secondary education Upper secondary or post-secondary non-tertiary education Tertiary education

Proportion of 25-64 year-olds scoring at PIAAC numeracy level 4 and 5, by educational attainment of the population (2012)


21


• Concerns about the quality and added-value of a university experience– Academically Adrift: limited improvement in

academic skills– What is the relative contribution of selection

versus teaching and learning in the production of high-quality graduates; what is the actual ‘learning gain’

– Doubts on the quality of the teaching and learning experience at elite universities

22


23


24

InputsIncreasing

investments by

taxpayers and families,

but concerns

about value-for-money

Black

box

Outputs

High numbers of graduates,

still high returns, but concerns

about quality

25

InputsIncreasing

investments by


but concerns


Black box

Outputs



about quality

?

26

• What do we know about how quality of teaching and learning results in high-quality output, and socially interesting outcomes?

• Information asymmetry: both public and private financers of higher education have very little understanding of what they actually are spending money for

• Increase of investment has not been accompanied by an empowerment of the input side to make smart choices through better information

Black box

Black box problem is problem of transparency

27

• How to improve transparency?– Performance management systems: essentially

bureaucratic control systems, very rarely comparative and tuned to better decision-making

– Quality assurance arrangements: increasingly relativistic (‘fitness-for-purpose’), bureaucratic and inward oriented (internal QA), unfit for external transparency, nor empowering students

– Student satisfaction surveys: empowering, but perception-based, largely unrelated to teaching and learning quality

Black box


28

• How to improve transparency?– Input measures as proxies for teaching and

learning quality: above a threshold level input has a weak relationship to quality (cfr PISA)

– Research measurement and bibliometric indicators as proxies for teaching and learning quality: most inaccurate, unfair to wide range of institutions and institutional diversity

– Reputations: act as (too?) powerful tools, also in rankings, but inaccurate, often outdated, inimical to newcomers and innovators

Black box


29

• Sound metrics of teaching and learning are very much needed– To reassure governments and families about the value-

for-money of investments– To reward institutions who invest in improving teaching

and learning and are not compensated through other measures

– To value institutional diversification– To reward and foster quality improvement through

mutual learning– To compensate for the over-reliance of rankings on

research and reputation metrics

Black box


30

• What are the risks of not improving transparency?– Erosion of the symbolic power of degrees, the only

monopoly of the higher education sector• Employers turning to alternative modes of selection• Emergence of alternative modes of qualification (employer

credentials, badges, recognition of prior learning, etc.)

Black box


31

Black box


32

• What are the risks of not improving transparency?– Blaming the student becomes the main excuse– Decreasing trust of governments, employers, families

and wider society in the value of higher education• Degree inflation• Concerns about declining standards in some countries

– Gradual erosion of the financial health of higher education institutions if value-for-money concerns are left unanswered

• Financial bubbles of student debt

– Markets no longer accept non-transparency• Cfr Volkswagen

Black box


33

InputsIncreasing

investments by


but concerns


Black box

Outputs



about quality

?

34

InputsIncreasing

investments by


but concerns


Black box

Outputs



about quality

Opening the black box

35

• We need to better understand what happens in the teaching and learning environments in universities– With the investment inputs provided– Explaining the benefits and returns and

understanding why they sometimes fail– In order to incentivize improvements in teaching

and learning– Empower students and families– Restore trust

Black box Opening the black box

36

• Comparative assessment of learning outcomes of graduates is the most promising approach to measure teaching and learning excellence– OECD’s AHELO project– National research projects in Germany, UK, Italy– CLA and various other initiatives in US– OECD-CEA partnership to implement CLA+ in

countries– European Commission supported CALOHEE

project in Tuning framework


37

• Strong resistance by parts of the academic community, but do they have a strong case?– No consensus on academic skills that matter– Risk of standardization– Institutional diversity too large to use limited number of

metrics– Methodological concerns– Cost and burden

• …exactly the same arguments used 20 years ago when the PISA programme was born

• …and very similar to arguments used 10 years ago against measuring research excellence


38

• Developing reliable metrics of teaching and learning excellence in universities is the next big systemic challenge in the development of higher education worldwide

• In the short term universities might think it’s not in their interest and that non-transparency is the better option

• But in the longer term that might be a very risky approach, in which the costs largely exceed the short-term profits


39

• The future progress of universities will not come from– Continued massification and ever higher numbers of

students– Continued increases in public and private funding

• But will be the outcome of better and more visible quality improvements in teaching and learning

• The impact of teaching and learning excellence will be similar to the way research excellence has changed the system

Finally

40

InputsExcellent teaching

and learning

Outputs

Opening the black box

Thank you !

[email protected]/edu/ceri

twitter @VanDammeEDU

41

Opening the ‘black box’ was melbourne 2 oct 2015

Education

Transcript of Opening the ‘black box’ was melbourne 2 oct 2015