1

Reflections on RAE 2008

Richard ThorpeBusiness & Management Sub-panel (i36)r.thorpe@lubs.leeds.ac.uk

2

General Issues

RAE: Research Assessment Exercise Research quality over period 2001–07

to be assessed a longer period than usual

Criteria and working methods published – lesson, read the exam question!

A new grouping of sub-panels into main panel areas main panel 36

Funding to be affected from 2009-10

3

Make up of the panel Panel members plus 3 – sub panels

Accounting and Finance Economics and Econometrics Information Management

Employment Relations Entrepreneurship Innovation Leadership Management Education and Management Learning Management Science Organisational Psychology Public Management and Administration Service Operations Small Business Strategic Management and Technology Technology Management

4

Additional specialist advice Corporate Social Responsibility International Business Tourism The more mathematic aspects of Operational

Research

The panel recognised that outputs couldn't always be characterised as falling neatly into disciplines and that in considering how best they might be distributed and assessed we were sensitive to other categorisations such as theme, sector or function

5

Main

pan

el I: C

hair

David

Otl

ey

34: Economics & econometrics (David Greenaway, Nottingham)

35: Accounting & finance (Andy Stark, MBS)

36: Business & management (Mike Pidd, Lancaster)

37: Library and information mgt(John Feather, Loughborough)

Ray Paul

David BlackabyJane

Broadbent

Structure of main panel I

6

Some history

Previous proper RAEs in 1992, 1996, 2001 Applied a single rating to the whole of a

submission

2001: each output rated as International, National or Sub-national quality 5* implies

>50% output of International standard Very little output rated below National level

Grade applies to all staff submitted, however

7

Main differences from 2001

£0.00

£5.00

£10.00

£15.00

£20.00

£25.00

£30.00

5* 5 4 3 2 1

2001 RAE rating

£000s/head

R monies 2001 Same amount for all staff in

department Leads to cliff edge funding

R monies post-2008 No cliff edge Profile funding

Main panel I Accounting & finance Economics & econometrics Business & management

Studies Library & information

management Staffing rules: no overlap period

8Dual funding system for research

Research income

HEFCE R moniesBased on RAE

Pays for research time &

infrastructure

Other research income, including research councilsPays for research

projects & programmes

9

RAE 2008 quality profile Quality level: 4*, 3*, 2*, 1*, unclassified

Unclassified = zero stars Percentage of research activity in each

category Based on FTE staff submitted No requirement to state % submitted

To apply to research output, the research environment indicators of esteem & impact

10RAE 2008 quality criteria for outputs

Its originality, significance & rigour

As a possible point of reference* in field or sub-field

4* World leading Primary

3* International standard of excellence

Major

2* International quality Contributes

1* National quality Limited contribution

u/c Below national

* To knowledge, theory, policy or practice

11

Judgements of quality Originality

e.g. innovation or distinctiveness of the methodological approach

Data sets used Research questions posed Underlying hypotheses or theoretical framework

Significance Insight and scope of coverage of the work Impact on the discipline in the UK or internationally Extent to which has opened up new areas of research Current or potential impact on policy and practice

Rigour Contextualisation of the work Strength ,appropriateness and intellectual coherence Extent to which the research outcomes are supported

12

Elements of assessment

Researchenvironment

(20%)

Esteem & impact

indicators (10%)

Research outputs: 4 per person (70%)

Quality profile

Weighted and aggregated across each submission

e.g.Research incomePhD studentsStaff development

13RAE 2008: calculation of R monies

Quality level 4* 3* 2* 1* u/c

% research activity

20

25

30

15

10

e.g. Univ of North Midlands enters 50 FTE staff

!!!staff!not papers are stars

and stars for income/FTE :Where

.15.30.25.20 1234

iR

RRRRIncome

i

We do not know what the R values will be

14

RAE 2008 important dates 2008

17th December profiles given to University Vice Chancellors for their institutions

18th December Profiles published in the press for all institutions

2009 4th January institutional feedback given to

UoAs and output, environment and esteem indicators released

March Full submissions published for all institutions which will include papers submitted

15

Research outputs For established staff: 4 per person expected

Unless work is exceptional Or time out from research (apply pro rata rule) Part-timers (apply pro rata rule) Multi-authored work: avoid joint submission from

same department unless work is exceptional Different for early career researchers

Should flag up (possibly) Exceptional work (e.g. potential 4* but not in top-

ranked outlet) Early career researchers People who’ve had time out or part-time (equal

opps) Category C ??

16

The numbers

90 submissions 97 3300 category A FTE staff (3500 on

headcount) 3000 700 category B 50 category C 12,600 outputs 10,000

70% outputs 20% environment 10% esteem

17

Managing the process Of the 12,600 about 10% were cross referred to i

34, 14% to i 35 and 4% to i 37 All outputs were read in detail Each panel member selected 4* outputs and

these were discussed to ensure a common process of calibration

Some submissions were clearly not for Business and Management

For environment and esteem sub panel members were asked to read submissions and profiles were decide by the whole panel

A user member was a member of the panel and where submissions proved evidence of a research environment that was geared to policy and practice appropriately high marks were awarded

18

Research Income In 2001 (a 5 year period) total research income

was £200m In 2008 (a 7 year period) total income was £360m

OST/OSI funding £90m

In 2001 the no of research associates was 500+ In 2008 the number of research associates was 470

In 2001 PhD students numbered 2,600 In 2008 PhD students numbered 3,450, 4.87 per

research active FTE – so room for growth! Some high numbers (e.g. 12) were associated with low quality profiles

19

Issues for HRD Issues relating to ‘field journals’ and their

quality in respect to building a knowledge base Other sub disciplines in a similar position

Hospitality and tourism Entrepreneurship and small business

Issues relating to judging impact in relation to policy and practice

Theory vs Empirical vs Applied Issues relating to Journal ‘quality’ impact

factors and citations Listings of journal quality Published impact factor data

Location of journals Quantitative verses Qualitative

20

Final Thoughts – does HRD need to ‘up its game’? Something about the development of theory in

the field – so what is being added to what? We need to be better connected to the knowledge base

Something about research design. Currently a great deal relies on the use of cases – how well do we theorise from the cases and can we be more innovative

Something about the originality of method – how can we be more rigorous in the methods we use and be able to make claims for policy and practice

Something about being picked up in citations and how we write, e.g. Jeff’s - ‘what a load of bollock’ paper: a story of the hairdresser and his suit’

21

Early career researchers

Central RAE definition “Entered the academic profession on

employment terms that qualified them for submission to RAE 2008 as Category A staff on or after 1st August 2003.”

Submission requirements: Business & Mgt Appointed 1/8/03 to 31/7/05: normally 2

outputs Appointed 1/8/05 or later: normally 1 output Working papers may be submitted The denominator will be adjusted so there is no

point submitting more than this