Using Behavioural Insights and Randomised

Controlled Trials in public policy

Dr Rory Gallagher

CabinetOfficeBehavioural Insights Team

CabinetOfficeBehavioural Insights Team

Behavioural insights (BI) offers new evidence

and frameworks for influencing behaviour

1. Regulation

2. Incentives

3. Information

Behavioural Insights

CabinetOfficeBehavioural Insights Team

BI can help change the way we think about and

approach policy problems

“Our government will find intelligent ways to encourage, support and enable people to make better choices for themselves.”

CabinetOfficeBehavioural Insights Team

• ‘Gold Standard’ in evaluation

• Widely used in medicine, international development

and business, but less so in public policy

• Integral part of BIT’s methodology

RCTs are a good way of evaluating whether our

(BI) interventions are working

.

CabinetOfficeBehavioural Insights Team

Lots of myths about using RCTs in publioc

1. Pointless

2. Expensive2. Expensive

3. Unethical

4. Difficult to run

CabinetOfficeBehavioural Insights Team

• Understand what works

– Evidence-based policy; VfM; prove efficacy

• Context is everything

Why we use RCTs

• Context is everything

– Lab vs field; different policy areas/locations/people

• Details matter

– Small details = big effects; co-design

CabinetOfficeBehavioural Insights Team

Multi-arm trials can highlight important nuances

35.1

35.9

37.2

39.0% paying tax debts after 23 days

33.6

35.1

Control (8,558) UK Norm (8,300) Local Norm (8,403) Debt Norm (8,779) Local + Debt Norm

(8,643)

CabinetOfficeBehavioural Insights Team

Different measures also reveal nuances

5.00%

10.40%

15.40%

Proportion Leaving a Legacy Gift

£3,300 £3,100

£6,610 Size of Legacy Gifts

5.00%

Control Just Ask Passion Ask Control Just Ask Passion Ask

CabinetOfficeBehavioural Insights Team

21

35 35

25

30

35

40

Response rate of Doctors to HMRC letters

Don’t always produce the results you expect

4

21

0

5

10

15

20

Generic HMRC

letter

Tax Health Plan

letter

BI letter BI letter + social

norms

CabinetOfficeBehavioural Insights Team

Don’t always produce the results you want!

22

28

25

20

25

30

% Discontinuing council tax discount

0

5

10

15

Original letter (control) New BI letter New BI letter + signature up

front

CabinetOfficeBehavioural Insights Team

• Operational constraints– Policy interventions?

– Buy-in

– Sample size

– Data sharing

• Political constraints

Constraints and trade-offs

• Political constraints– Established programmes

– Transparency - ‘nil’ results/ ‘failures’

– Time frames

• Trade-offs– Who/how randomise

– No. of arms

– Impact (‘bundle’) vs specific causal effect

CabinetOfficeBehavioural Insights Team

• Randomisation

• Design

Things to watch out for

• Implementation

• Measurement

CabinetOfficeBehavioural Insights Team

BIT paper: Test, Learn, Adapt

BI and RCTs in action:

Two case studiesTwo case studies

CabinetOfficeBehavioural Insights Team

What is this letter about?

CabinetOfficeBehavioural Insights Team

Actions/consequences unclear

Is the

heading

clear? What is a

distress

Contact info

on the back

page

distress

warrant?

Lots of possible

consequences –

diluting effect

Key action –

how contact?

Primes

appeals

Is this info

necessary?

CabinetOfficeBehavioural Insights Team

Court fines

CabinetOfficeBehavioural Insights Team

Court fines

£9

£13

£11£12

£8

£11

£9£10

10

12

14

16

Av

era

ge

am

ou

nt

pa

id (

£) Trial 1

Trial 2

£4

£9 £8 £9

0

2

4

6

8

10

No text Standard Personal Amount Personal +

Amount

Av

era

ge

am

ou

nt

pa

id (

£)

Text condition

CabinetOfficeBehavioural Insights Team

Test, Learn, Adapt – 9 steps

1. Intervention - test effect of text messages on payment of court fines

5 arms = control, standard, personal, amount, personal+ amount

2.Outcome - % individuals paying court fines by the deadline date

3. Unit - randomise at individual level

(rather than different courts or local authority areas)

4. Numbers - to demonstrate a 10% increase in payment rates, we needed a

sample size of 2,000 people.

5. Randomisation - individuals randomly allocated to 5 conditions

6. Administration - Courts Service sent text messages to the individuals

randomly assigned to receive them

7-9. Test, Learn, Adapt - 3 trials, impressive results.

HMCTS exploring national roll-out.

CabinetOfficeBehavioural Insights Team

Getting people into employment

CabinetOfficeBehavioural Insights Team

Getting people into employment

1. Cut down process 2. Commitments 3. Strengths identification

CabinetOfficeBehavioural Insights Team

Getting people into employment

69%

66%65%

70%

80%% off-flow from benefits after 13 weeks

55% 55%57%

56%58%

61%

53%

40%

50%

60%

Feb Mar April May June July Aug

CabinetOfficeBehavioural Insights Team

Pragmatic stepped wedge randomised trial:

job centres across the county

Job

CentreReadiness

Implementation

Order

(TBC)

On-flow

per

month

Off-flow at

13 weeks

Months of trial

0 1 2 3 4 5 6 7 8 9 10 11 12 etc....

1

Ready

Harlow 608 51.70% *M1 *M2 *M3 *M4 *M5 *M6 *M7 *M8 *M9 etc....

2 Chelmsford 727 54.00% *M2 *M3 *M4 *M5 *M6 *M7 *M8 *M9 etc....

3 Colchester 734 52.80% *M3 *M4 *M5 *M6 *M7 *M8 *M9 etc....

4 & 5Canvey & 235 & 55.5% &

*M4 *M5 *M6 *M7 *M8 *M9 etc....

UNCLASSIFIED

4 & 5

Almost

Ready

Canvey &

Rayleigh

235 &

308

55.5% &

57.3%*M4 *M5 *M6 *M7 *M8 *M9 etc....

6 & 7Clacton &

Harwich

441 &

129

48.8% &

47.0%*M5 *M6 *M7 *M8 *M9 etc....

8 Grays 683 50.50% *M6 *M7 *M8 *M9 etc....

9

Not ready

Basildon 901 53.50% *M7 *M8 *M9 etc....

10 & 11Braintree &

Witham

444 &

177

55.2% &

52.2%*M8 *M9 etc....

12 Southend 930 50.20% *M9 etc....

Dark grey = current jobseeker process (control)

Blue = new jobseeker process (intervention)

*MX = measurement of 13 week off flow for cohort which began jobseeker claim in month X of trial (in bold = first month of outcome measure for that cluster)

CabinetOfficeBehavioural Insights Team

• BI is having major impacts

• Provides new frameworks and approach - humility

• RCTs increasing - driving ‘what works’

• Test, Learn, Adapt: agile implementation

Conclusions

• Test, Learn, Adapt: agile implementation

• Context, Details, People … matter

• Early days for BI & RCTS- long way to go, just scratching surface

• Next phase = sustainability and segmentation

• Future enablers = ‘big’ data, digital, transparency, VfM

http://blogs.cabinetoffice.gov.uk/behavioural-insights-team