Evidence based policy

Leeann Brady

Research Officer

14 September 2007

Evidence based policy

• What is evidence?

• What is evidence based policy?

• Examples of evidence based policy.

• How do you go about creating evidence based policy?

• Questions for discussion.

Evidence?• ‘Evidence’ can be seen as what detectives look for in making up their minds, as

Sherlock Holmes said:

“It is a capital mistake to theorise before you have all the evidence. It biases the judgement.”

• Chambers dictionary definition of ‘evidence’:

– Means of proving an unknown or disputed fact– Support for a belief– An indication– Information in a law case– Testimony– Witness or witnesses collectively

• The range of possibilities hints at the difficulties that arise as to what constitutes evidence.

What constitutes evidence based policy?

• The Cabinet Office’s Better Policy Making (2001) identified an evidence based approach to policy development as one which:

– Reviewed existing research– Commissions new research– Consults experts and/or uses internal or external consultants– Considers a wide range of properly costed and appraised

options

Why is evidence based policy important?

• Key benefit is better policy.

• Critics have argued in the past policy decisions were too often driven by short-term political pressures.

• Government, funders and donors increasingly keen to support ‘WHAT WORKS’.

• Demands for an evidence base in Northern Ireland:

“we must be able to measure the contribution made by voluntary and community organisations” (Positive Steps, 2005)

Recognising the role of evidence in the sector

• Evidence base of the sector is limited:

– coverage in official statistics – resources– diversity – resistance to evaluation

• Time constraints – the policy cycle revolves quicker than the research cycle.

• NO GOLD standard for producing evidence based work.

Scepticism when embarking on evidence based policy

• With all evidence, some scepticism about validity is desirable.

• Questions to be asked:

– How relevant is this to what we are seeking to decide?– How representative is this of the population that concerns

us?– How reliable, how well founded theoretically/empirically is it?

• These are tough BUT necessary tests.

Evidence based policy in practice

• Example 1 – Barnardo’s

• Barnardo’s have developed ‘The Evidence Guide’ a modular training pack available at www.barnardos.org.uk/theevidenceguide

• Example 2 – The Cresco Trust

• These organisations seek to demonstrate not just what they do, but also why they do it.

The role of evidence based policy in NICVA

• NICVA recognises the value of expertise in other organisations when building evidence and works closely with other organisations in the voluntary and community sector, government and academia.

• Policy Manifesto.

• Membership survey.

Examples of how NICVA builds evidence with the aim of influencing policy

• State of the Sector

• Skills research

• Public service delivery research

• Positive Steps evaluations

You don’t need to have a research department to build an evidence base

• Your organisation was founded because someone identified a need.

“Voluntary organisations are used to highlighting the need for particular services but not looking at how cost effective their way of delivering that service is.” (Mary Duffy, principal officer in Barnardo’s policy and research unit)

• So how can you build your evidence base?

Get someone else to do the work for you

• NICVA carries out ‘Viewfinder’ surveys.

• There are various mechanisms available to voluntary and community groups to access audits, for example CENI, Business in the Community, NICVA.

• The Science Shop, University of Ulster.

• You can commission research to be carried out by a consultant.

Choosing to do it yourself – overview of the research process used to build evidence

CONCEPTUALISATION

Specify the meaning of the concepts and variables to be studied

POPULATION AND SAMPLING

Whom do you want to be able to draw conclusions about?

CHOICE OF RESEARCH METHOD(S)

•Survey Research

•Qualitative Interviews

•Experimental Research

•Documentary Analysis

OPERATIONALISATION

How will you actually measure the variables under study?

INITIAL IDEA

Source: Adapted from Babbie (2001)

Doing it yourself

1. Search to see what already exists as well as identify gaps.

2. Condense ideas into a small number of core research questions.

3. Check the credibility of your source. Just because it appeared when you ‘googled’ it, it doesn’t mean it is reliable.

An example of one to ‘bookmark’

• Northern Ireland Neighbourhood Information Service

http://www.ninis.nisra.gov.uk/

Choice of research method(s)  Quantitative Methods

(questionnaires)Qualitative methods (interviews, participant observation)

Nature Data expressed in numbers. Data expressed in words.

Main purposes Identifying broader patterns and relationships within the population.

Understanding people’s experiences and perspectives and what motivates them to think and behave as they do.

Analysis Can be analysed using SPSS and Microsoft Excel.

Can be analysed by identifying patterns and themes, as well as outliers. This can be done with a highlighter and hand, as well as software packages such MaxQDA.

Strengths Can make generalisations from a sample to a whole population.

Can help explain why particular groups of people do what they do.

Weaknesses Cannot explain reasons for patterns and relationships.

Cannot generalise.

•A combination of methods is often used.

Things to remember when undertaking research to build an evidence base

• Approach the design of your research clearly and logically:

– Be clear and explicit about what you are interested in.

– Choose something small and manageable.

– Only think about the choice of methods once you have identified your research questions.

QuestionsDo you think evidence based policy is a good or a bad thing for the voluntary and community sector?

Do you think the fact that we have devolved government in Northern Ireland will create a bigger space for evidence based policy?

Are there any obstacles/challenges to your organisation to carry out this sort of work?

Is there any support your organisation would like in regard to building evidence based policy?