11

Experimental Design In Social Experimental Design In Social Research - Do we know for example Research - Do we know for example

what works against crime?what works against crime?

Seminar Seminar Centre for Census and Survey Research, University of Centre for Census and Survey Research, University of

ManchesterManchester5 Dec. 20065 Dec. 2006

Paul MarchantPaul MarchantLeeds Metropolitan UniversityLeeds Metropolitan Universityp.marchant@leedsmet.ac.ukp.marchant@leedsmet.ac.uk

(Paul Baxter from Department of Statistics, University of Leeds is involved (Paul Baxter from Department of Statistics, University of Leeds is involved in developing some of this work)in developing some of this work)

22

The Basic PointThe Basic Point If non-RCTs are used, we need a sound If non-RCTs are used, we need a sound

understanding of the system being studied understanding of the system being studied and a quantitative model to work out what and a quantitative model to work out what is lost and what the effect is.is lost and what the effect is.

The effects being sought may be small so The effects being sought may be small so impact of small systematic errors can be impact of small systematic errors can be important.important.

Use best scientific methods, especially Use best scientific methods, especially when policy implications are costly. when policy implications are costly.

Need rigorous scientific evaluation of Need rigorous scientific evaluation of the implementation of policythe implementation of policy..

33

Science and StatisticsScience and Statistics Science is ‘the belief in the Science is ‘the belief in the

unimportance of experts’ , Richard unimportance of experts’ , Richard Feynman. (He wrote ‘Cargo Cult Feynman. (He wrote ‘Cargo Cult Science’ on what distinguishes Science’ on what distinguishes science from the rest.) science from the rest.)

Science is sceptical enquiry (note Science is sceptical enquiry (note Gorard 2002).Gorard 2002).

‘‘Statistics is the logical glue which Statistics is the logical glue which attaches conclusions to data’.attaches conclusions to data’.

44

Randomised Controlled Trial v. Randomised Controlled Trial v. something elsesomething else

In crime research there is a 5 point In crime research there is a 5 point ‘Scientific Methods Scale’ which ‘Scientific Methods Scale’ which orders trial designs (RCT is the top )orders trial designs (RCT is the top )

While the ordering may be fine there While the ordering may be fine there is no formal indication of what is lost is no formal indication of what is lost by using a 4 rather than a 5. by using a 4 rather than a 5.

A large potential exists it would seem A large potential exists it would seem to draw false inference.to draw false inference.

55

The Randomised Controlled The Randomised Controlled TrialTrial

(A truly marvellous scientific (A truly marvellous scientific invention)invention) Note to avoid bias:Note to avoid bias:

Register trial / Register trial / protocol.protocol.

Allocation is best Allocation is best made tamper-proof. made tamper-proof. (e.g. use ‘concealment’)(e.g. use ‘concealment’)

Use multiple blinding Use multiple blinding of:of: patients, patients, physicians, physicians, assessors, assessors, analysts …analysts …

Population

Take Sample

Randomise to 2 groups

Old Treatment

Compare outcomes (averages) recognising that

these are sample results and subject to sampling variation when applying back to the population

New Treatment

66

Counts of those cured and not Counts of those cured and not cured under the two cured under the two

treatmentstreatmentsCured Not

CuredNew Treatment a b

Control(Standard treatment)

c d

By comparing the ratios of numbers ‘cured’ to ‘not cured’ in the 2 arms of the trial, the Cross Product Ratio (CPR)= (ad)/(cb), it is possible to tell if the new treatment is better.

77

Confidence IntervalsConfidence Intervals However there is However there is sampling variability, sampling variability, because we because we

don’t study everybody of interest; just our don’t study everybody of interest; just our randomrandom samplesample. .

So cannot have perfect knowledge of the effect of So cannot have perfect knowledge of the effect of interest, but only an estimate of it interest, but only an estimate of it within a within a confidence interval (CI)confidence interval (CI). .

Need to know how to calculate the CI appropriately. Need to know how to calculate the CI appropriately. This can be done under assumptions, which seem This can be done under assumptions, which seem reasonable for the case of a clinical RCT and leads reasonable for the case of a clinical RCT and leads to a simple formula for the approximate CI (+/-1.96 to a simple formula for the approximate CI (+/-1.96 standard error) of ln(CPR)standard error) of ln(CPR)(s.e. (ln(CPR)) )(s.e. (ln(CPR)) )22= Var(ln(CPR))= Var(ln(CPR))

== 11 + + 11 + + 11 + + 1 1 a b c d a b c d

88

Crime counts before and after in two Crime counts before and after in two areas one gets a Crime Reduction areas one gets a Crime Reduction Intervention CRI (e.g. 4 on the Methods Intervention CRI (e.g. 4 on the Methods Scale)Scale)

A similar table results. But this is not the same as the RCT set A similar table results. But this is not the same as the RCT set up as: up as:

1 Not randomised, so no statistical equivalence exists at the 1 Not randomised, so no statistical equivalence exists at the start. start.

2 The unit is area, rather than crime event. 2 The unit is area, rather than crime event.

Before After

Treatment Area(Intervention is introduced

between the 2 periods )

a b

Comparison Area(Nothing is changed)

c d

99

Lighting andLighting and crimecrime

There seem to be many ‘theoretical There seem to be many ‘theoretical suggestions’ why lighting might increase suggestions’ why lighting might increase or decrease crime. or decrease crime.

The meta-analysis, HORS251, by The meta-analysis, HORS251, by Farrington and Welsh suggests strongly Farrington and Welsh suggests strongly that lighting beats crime. However my that lighting beats crime. However my contention is that this study contains contention is that this study contains flaws and so we cannot be sure of the flaws and so we cannot be sure of the effect of lighting on crime. (Note also effect of lighting on crime. (Note also HORS252 on CCTV)HORS252 on CCTV)

1010

Forest Plot as HORS 251 Meta-analysisForest Plot as HORS 251 Meta-analysis

reconstructedreconstructed

Odds ratio.156196 1 6.40222

Study % Weight Odds ratio (95% CI)

3.82 (2.28,6.40) Birmingham 0.7

1.72 (1.17,2.52) Stoke 1.8

1.39 (1.04,1.86) Atlanta 3.4

1.44 (1.17,1.77) Dudley 6.6

1.38 (0.97,1.97) Fort Worth 2.3

1.37 (1.06,1.77) Milwaukee 4.4

1.35 (1.23,1.47) Bristol 37.8

1.24 (0.95,1.64) Kansas City 4.2

1.14 (0.62,2.08) Dover 0.9

1.02 (0.75,1.40) Harrisburg 3.5

1.01 (0.89,1.15) New Orleans 21.2

0.94 (0.79,1.12) Portland 11.3

0.75 (0.47,1.18) Indianapolis 1.9

1.23 (1.17,1.31) Overall (95% CI)

1111

But this can’t be right.But this can’t be right. The assumptions for calculating the CIs cannot be The assumptions for calculating the CIs cannot be

correct, in this case. Unit is area not crime. The correct, in this case. Unit is area not crime. The events are not statistically independent within areas.events are not statistically independent within areas.

Too much variation (heterogeneity) exists as seen by Too much variation (heterogeneity) exists as seen by Q-statistic between individual study results Q-statistic between individual study results compared with the uncertainty indicated by compared with the uncertainty indicated by confidence intervals, (if the lighting has the same confidence intervals, (if the lighting has the same effect on crime in every study).effect on crime in every study).

Note there is great variation in crime counts Note there is great variation in crime counts between periods in the comparison areas, where between periods in the comparison areas, where nothing is changed (TAU) (The variance is an order nothing is changed (TAU) (The variance is an order of magnitude greater than the mean). Shows the of magnitude greater than the mean). Shows the heterogeneity is inherent to the natural variation of heterogeneity is inherent to the natural variation of crime. Crime is committed by criminals and it is their crime. Crime is committed by criminals and it is their activitychanging which can cause great variation.activitychanging which can cause great variation.

1212

Pointing out the problemPointing out the problem Marchant (2004), 7 page article in the Marchant (2004), 7 page article in the

British Journal of Criminology drawing British Journal of Criminology drawing attention to the problem. The formula for attention to the problem. The formula for the CIs used must be inappropriate (also the CIs used must be inappropriate (also mentioning other short-comings).mentioning other short-comings).

The authors of HORS251 had 20-page The authors of HORS251 had 20-page response on the next page, justifying the response on the next page, justifying the claim that lighting reduces crime. claim that lighting reduces crime.

But I remain unconvinced by the claim.But I remain unconvinced by the claim.

1313

Study NameStudy Name kkobsobs = = ss22//xx AtlantaAtlanta 58.359458.3594

MilwaukeeMilwaukee 13.754713.7547

PortlandPortland 7.70177.7017

Kansas CityKansas City 9.74499.7449

HarrisburgHarrisburg 1.57481.5748

New OrleansNew Orleans 46.681046.6810

Fort WorthFort Worth 0.29340.2934

IndianapolisIndianapolis 0.04000.0400

DoverDover 4.76474.7647

BristolBristol 44.711644.7116

BirminghamBirmingham 1.53061.5306

DudleyDudley 4.44204.4420

Stoke-on-TrentStoke-on-Trent 0.00830.0083

kkobsobs are extremely are extremely variable and right variable and right skewed. The skewed. The arithmeticarithmetic mean is mean is 15 for these 15 for these comparison areas. comparison areas.

Calculated from the Calculated from the before and after before and after counts in the counts in the comparison areascomparison areas

Correlation between intervention and comparison tends to reduce the effect of overdispersion.

Examine Overdispersion in Comparison Examine Overdispersion in Comparison AreasAreas

1414

Fixing the Heterogeneity Fixing the Heterogeneity Problem Problem

A way of making the problem go away is A way of making the problem go away is simply to increase the uncertainty, i.e. simply to increase the uncertainty, i.e. stretch the CIs . (‘A quasi-Poisson/Binomial stretch the CIs . (‘A quasi-Poisson/Binomial model’). model’).

Here the CIs are stretched by a factor of Here the CIs are stretched by a factor of 2.1. (Equivalent to reducing the events 2.1. (Equivalent to reducing the events counted in every setting by a factor 2.1counted in every setting by a factor 2.12 2 = = 4.4. ). This adjustment has been made by 4.4. ). This adjustment has been made by the authors. (Farrington and Welsh 2006)the authors. (Farrington and Welsh 2006)

Problem solved.... or is it? Is such model Problem solved.... or is it? Is such model plausible? Assumes every study should plausible? Assumes every study should have its CI stretched by the same factor. have its CI stretched by the same factor. This cannot be guaranteed. This cannot be guaranteed.

1515

Some doubtsSome doubts Only relatively few studies.Only relatively few studies. Need to understand the sensitivity of the Need to understand the sensitivity of the

result to assumptions. result to assumptions. How does the variation of crime depend on How does the variation of crime depend on

the level of crime?the level of crime? Publication bias bedevils systematic Publication bias bedevils systematic

reviews, association with ‘positive findings’.reviews, association with ‘positive findings’. The 2 studies which are given as The 2 studies which are given as

statistically significant under the ‘quasi’ –statistically significant under the ‘quasi’ –model also have time series of counts and model also have time series of counts and these do not appear to show convincing these do not appear to show convincing evidence for lighting benefit.evidence for lighting benefit.

1616

The Bristol Study (Shaftoe The Bristol Study (Shaftoe 1994)1994)

1990.01989.01988.01987.01986.0

1500

1400

1300

1200

1100

1000

900

800

700

600

Year

No.

Cri

mes

1311

1464

1160

1237

14241374

142915171414

840840846

727

648

833838

685630

Number of Crimes Reported(in half-year periods)

Brighter Lighting

Control

New lighting introduced from July 87 to March 89 as marked onthe time axis.

Shaftoe said ‘no discernable lighting benefit’ but HORS251 said Shaftoe said ‘no discernable lighting benefit’ but HORS251 said z=6.6 Note: had the data for the year immediately prior to the z=6.6 Note: had the data for the year immediately prior to the introduction of the relighting, i.e. periods 2 and 3, been used introduction of the relighting, i.e. periods 2 and 3, been used rather than unnaturally using periods 1 and 2 which leaves a rather than unnaturally using periods 1 and 2 which leaves a gap of ½ year, the effect found would have been half of that gap of ½ year, the effect found would have been half of that claimed. (Shows large variability.)claimed. (Shows large variability.)

1717

Time Variation in CrimeTime Variation in Crime

It appears that little is known about how It appears that little is known about how crime varies on various scales, in general. crime varies on various scales, in general.

Much more needs to be known about the Much more needs to be known about the occurrence of crime events to know how to occurrence of crime events to know how to analyse them properly to be able find analyse them properly to be able find effects.effects.

Need access to suitable data sets to Need access to suitable data sets to examine this issue. This is on going examine this issue. This is on going research in which myself and colleagues research in which myself and colleagues are engaged. (Plea for data)are engaged. (Plea for data)

A general point: one needs to have A general point: one needs to have knowledge about the system in order to knowledge about the system in order to understand if an intervention changes understand if an intervention changes things. (And in order to design studies)things. (And in order to design studies)

1818

Household studiesHousehold studies In a couple of instances, instead of just counting In a couple of instances, instead of just counting

recorded crimes a, b, c, d in the 4 cells (before, recorded crimes a, b, c, d in the 4 cells (before, after, intervention, comparison), a household after, intervention, comparison), a household survey, before and after, of crimes recalled for survey, before and after, of crimes recalled for the previous period within the 2 areas the previous period within the 2 areas (intervention, comparison) was carried out. (intervention, comparison) was carried out.

One problem is that (unrecognised by authors One problem is that (unrecognised by authors Painter and Farrington) spatial correlation Painter and Farrington) spatial correlation between the occurrence of crime needs to be between the occurrence of crime needs to be considered. (Shared experience of crime on the considered. (Shared experience of crime on the local scale). Gives rise to a Design Effect familiar local scale). Gives rise to a Design Effect familiar in clustered designs. Reduces the precision of the in clustered designs. Reduces the precision of the estimate of the effect.estimate of the effect.

Other problems, e.g. of differential change of Other problems, e.g. of differential change of composition between periods.composition between periods.

1919

Different crimes at the start.Different crimes at the start. Overdispersion clear in number of crimes per Overdispersion clear in number of crimes per

household.household. Differential loss to follow up.Differential loss to follow up. Old people are much less prone to experience crime Old people are much less prone to experience crime

and their number is much reduced due to loss to and their number is much reduced due to loss to follow-up in the comparison area. So the relative follow-up in the comparison area. So the relative composition changes during the experiment. composition changes during the experiment.

Results are very sensitive to the loss or addition of just Results are very sensitive to the loss or addition of just one personone person

But importantly there is But importantly there is correlation between correlation between householdshouseholds, giving extra overdispersion (variability)., giving extra overdispersion (variability).

Essentially it’s a Essentially it’s a non-randomised two-cluster trialnon-randomised two-cluster trial..

The Dudley Study: some problemsThe Dudley Study: some problems

2020

Spatial Correlation (1)Spatial Correlation (1)

An expression can be derived for the variance of An expression can be derived for the variance of

ln(CPR) for a household survey, before and ln(CPR) for a household survey, before and after, intervention-comparison study, i.e. of after, intervention-comparison study, i.e. of the Dudley type. This includes, in addition to the Dudley type. This includes, in addition to the variability between households, the variability between households, bothboth: :

(1)(1) correlations within households between times. correlations within households between times.

(2)(2) correlations correlations betweenbetween households at any one households at any one time. time.

2121

Spatial Correlation (2)Spatial Correlation (2)

What you get basically is the expression you What you get basically is the expression you would get if you ignored the correlation would get if you ignored the correlation between households at one time, i.e. between households at one time, i.e. ignored the spatial correlation, multiplied ignored the spatial correlation, multiplied by the ‘Design Effect’, Deff. (Just as in by the ‘Design Effect’, Deff. (Just as in clustered surveys / trials)clustered surveys / trials)

Deff=(1+(n-1) ρDeff=(1+(n-1) ρss))

ρρss = the spatial correlation = the spatial correlation

n = the number in a cluster, i.e. arean = the number in a cluster, i.e. area

2222

Spatial Correlation (3)Spatial Correlation (3) The spatial correlation was not taken into The spatial correlation was not taken into

account in the Dudley and Stoke analyses account in the Dudley and Stoke analyses thus ignoring the fact that neighbours thus ignoring the fact that neighbours ‘share risk’.‘share risk’.

An expression for the variance of the An expression for the variance of the logarithm of the Cross Product Ratio CPR logarithm of the Cross Product Ratio CPR incorporating spatial correlation is:incorporating spatial correlation is:

1 1 1 1 1 1ln 2 tVar CPR λDeff r

a b c d ab cd

2424

To summarize the issue overdispersion To summarize the issue overdispersion from the Dudley studyfrom the Dudley study

On the matter of overdispersion alone, the On the matter of overdispersion alone, the value obtained from the between household value obtained from the between household variation cannot be correct as the effect of variation cannot be correct as the effect of spatial correlation, i.e. the ‘shared experience’ spatial correlation, i.e. the ‘shared experience’ of nearby households needs to be taken in of nearby households needs to be taken in account, i.e. crime experiences will be linked. account, i.e. crime experiences will be linked. This will increase overdispersion. We would This will increase overdispersion. We would need more than 2 clusters to be able estimate need more than 2 clusters to be able estimate Deff.Deff.

2525

Lack of Equivalence between Lack of Equivalence between AreasAreas

Invariably it is the most crime-ridden Invariably it is the most crime-ridden area that gets the lighting, whereas area that gets the lighting, whereas the relatively crime-free ‘control’ the relatively crime-free ‘control’ area is not re-lit. So there is lack of area is not re-lit. So there is lack of equivalence at the start. One effect equivalence at the start. One effect of this is to allow of this is to allow ‘regression ‘regression towards the mean’towards the mean’ to operate. to operate.

The name The name ‘Control‘Control Area’ is a Area’ is a misnomer. ‘misnomer. ‘ComparisonComparison Area’ is a Area’ is a better name.better name.

2626

Regression towards the Regression towards the meanmean

X The before measurement

Y T

he a

fter

mea

sure

men

t

Cloud ofDataPoints

Line of Equality

0 10050

0

50

100

Line of mean of Y for a given X

2727

The response given to the lack of The response given to the lack of equivalence between the 2 areas. equivalence between the 2 areas. (RTM)(RTM) Farrington and Welsh (2006) claim that RTM is not Farrington and Welsh (2006) claim that RTM is not

a problem because the effect in counted crimes in a problem because the effect in counted crimes in 250 Police ‘Basic Command Units’ going from 250 Police ‘Basic Command Units’ going from 2002/3 to 2003/4 showed only small effect (a few 2002/3 to 2003/4 showed only small effect (a few %). This is perhaps unsurprising as the areas and %). This is perhaps unsurprising as the areas and hence the number of crimes counted are an order hence the number of crimes counted are an order of magnitude larger than in HORS251 so the year of magnitude larger than in HORS251 so the year to year correlation may be expected to be higher to year correlation may be expected to be higher than for the small lighting study areas. than for the small lighting study areas.

Note Wrigley (1995) “This tendency for correlation Note Wrigley (1995) “This tendency for correlation coefficients to increase in magnitude as the size of coefficients to increase in magnitude as the size of the areal unit involved increases has been known the areal unit involved increases has been known since the work of Gehlke and Biehl (1934)”.since the work of Gehlke and Biehl (1934)”.

2828

Natural Log Burglary Rate in Period 1

Na

tura

l Lo

g B

urg

lary

Ra

te in

Pe

rio

d 2

-1.0-1.5-2.0-2.5-3.0-3.5-4.0

-1.0

-1.5

-2.0

-2.5

-3.0

-3.5

Log crime rates in successive periods: data from Tilley Log crime rates in successive periods: data from Tilley et al.et al.

2929

Estimating the effect of RTM Estimating the effect of RTM

On the basis of log normal crime rates it can On the basis of log normal crime rates it can be shown that if the intervention has no be shown that if the intervention has no effect, the expected ln effect, the expected ln CPR CPR = = (1-(1-ρσρσyy//σσxx) ) ln xln x11/x/x2 2

xx11/x/x2 2 is the crime rate ratio; is the crime rate ratio; σσx, x, σσy y the the standard deviations on the log scale and standard deviations on the log scale and ρρ the correlation on the log scale.the correlation on the log scale.

Var( ln Var( ln CPR ) = CPR ) = 2 2 σσyy22(1-(1-ρρ22))

3030

Estimation of the effect of RTMEstimation of the effect of RTM The simple model of crime rates suggests that the The simple model of crime rates suggests that the

high year to year correlation typically 0.95 for the BCU high year to year correlation typically 0.95 for the BCU crime rate data, would indeed give an effect of a few crime rate data, would indeed give an effect of a few %. %.

However the smaller areas used in CRI evaluation However the smaller areas used in CRI evaluation might be expected to have lower correlationmight be expected to have lower correlation

Burglary data from a study of 124 areas has Burglary data from a study of 124 areas has correlation of about 0.8 giving, all else equal, an correlation of about 0.8 giving, all else equal, an expected effect 4 times larger.expected effect 4 times larger.

Note: in general we don’t know the correlation nor Note: in general we don’t know the correlation nor rates being compared for the lighting studies. rates being compared for the lighting studies. However, we do know, whereas the household crime However, we do know, whereas the household crime rate ratio at the start was 1.40 for Dudley, that for rate ratio at the start was 1.40 for Dudley, that for Stoke was 2.51 giving a much larger expected RTM Stoke was 2.51 giving a much larger expected RTM effect.effect.

Without better knowledge we can’t be definite about Without better knowledge we can’t be definite about the impact of RTM but the indications are that the bias the impact of RTM but the indications are that the bias could be important and uncertainty large.could be important and uncertainty large.

3131

Ratio of Burglary Rates x1/ x2

Ln C

PR

3.02.52.01.51.00.50.0

1.0

0.5

0.0

-0.5

-1.0

1

0

Expected natural log of CPR and its CI Expected natural log of CPR and its CI for a set of burglary data.for a set of burglary data.

3232

Insights from a set of police beat dataInsights from a set of police beat data

Crime rate ratios (one year to the next ) Crime rate ratios (one year to the next ) for crime counts for each police beat in for crime counts for each police beat in a file of data have been calculated. It is a file of data have been calculated. It is possible to ratio this with that from the possible to ratio this with that from the next beat in the file thereby next beat in the file thereby constructing a CPR for the pair of beats. constructing a CPR for the pair of beats. This shows that effects of the size This shows that effects of the size claimed for lighting, 20%, are common claimed for lighting, 20%, are common in such comparisons and are in such comparisons and are considerably larger than that expected considerably larger than that expected on the basis of s.e. on the basis of s.e. =√(1/a+1/b+1/c+1/d).=√(1/a+1/b+1/c+1/d).

3333

Strengths and weaknesses of police Strengths and weaknesses of police

beat databeat data The strength is that it does not rely on a The strength is that it does not rely on a

statistical model; overdispersion and RTM statistical model; overdispersion and RTM are automatically taken into account.are automatically taken into account.

The weakness is that this data set may The weakness is that this data set may not represent the situation of the lighting not represent the situation of the lighting studies. But that will be the case for any studies. But that will be the case for any other sets. Yet we do not have sufficient other sets. Yet we do not have sufficient relevant information about the individual relevant information about the individual studies used to make the HORS251 claim. studies used to make the HORS251 claim.

3434

Opportunity of checking the effect of Opportunity of checking the effect of lightinglighting

Many brighter street lighting programmes Many brighter street lighting programmes are being implemented under government are being implemented under government PFI schemes.PFI schemes.

Admittedly non-randomised Admittedly non-randomised implementation but presents an implementation but presents an opportunity to check the claim.opportunity to check the claim.

The checking needs to be done to high The checking needs to be done to high scientific standards.scientific standards.

Perhaps it is possible to link changes of Perhaps it is possible to link changes of night-time brightness measured by night-time brightness measured by satellites to changes in crime. satellites to changes in crime.

3535

General potential consequences of General potential consequences of weak methodsweak methods

Because there is a tendency to report ‘positive Because there is a tendency to report ‘positive effects’ (dissemination bias) and probably even effects’ (dissemination bias) and probably even more so with less rigorous work, one is likely to more so with less rigorous work, one is likely to end up with an even more distorted research end up with an even more distorted research record. record.

This might lead to dubious justification of a bad This might lead to dubious justification of a bad policy.policy.

While it might be possible to estimate the While it might be possible to estimate the effect of the excess variability or the effect of effect of the excess variability or the effect of RTM, it would seem problematic to be confident RTM, it would seem problematic to be confident about adequately adjusting for them.about adequately adjusting for them.

Scientifically stronger methods could avoid Scientifically stronger methods could avoid many problems and may be very cheap relative many problems and may be very cheap relative to policy costs.to policy costs.

3636

“…“…Paul Marchant, statistician at Leeds Metropolitan Paul Marchant, statistician at Leeds Metropolitan University who argues that statistics used in the Home University who argues that statistics used in the Home Office Study 251 could equally be used to show that street Office Study 251 could equally be used to show that street lighting actually increases levels of crime. This is an lighting actually increases levels of crime. This is an argument which the APPLG, alongside the ILE, would hope argument which the APPLG, alongside the ILE, would hope to show as utterly absurd. Of course it is worth noting that to show as utterly absurd. Of course it is worth noting that Paul Marchant is also an astronomer as well as being a Paul Marchant is also an astronomer as well as being a statistician, and that this may lead to some bias in his statistician, and that this may lead to some bias in his interpretation of the statistics he refers to.”interpretation of the statistics he refers to.”

P56 of the March/April 2004 issue of the Lighting P56 of the March/April 2004 issue of the Lighting JournalJournal,, the magazine of the Institution of Lighting the magazine of the Institution of Lighting Engineers.Engineers.

APPLG = The All-Party Parliamentary Lighting GroupAPPLG = The All-Party Parliamentary Lighting GroupILE = The Institution of Lighting EngineersILE = The Institution of Lighting Engineers--------------------------------------------------------------------------------------------------------------------------------------------------------------My view is there is large uncertainty and one cannot be too sure My view is there is large uncertainty and one cannot be too sure what lighting does to crime. (Perhaps on average lighting what lighting does to crime. (Perhaps on average lighting increases crime).increases crime).

My interestMy interest

3737

Some conclusionsSome conclusions A ‘Methods Scale’ seems to suggest that designs A ‘Methods Scale’ seems to suggest that designs

weaker than RCTs might suffice, without indicating weaker than RCTs might suffice, without indicating what is lost. what is lost.

I have indicated some of the problems which result.I have indicated some of the problems which result. I remain to be convinced that the deficiencies can I remain to be convinced that the deficiencies can

be adequately overcome through estimating be adequately overcome through estimating quantitatively the consequences of using a weaker quantitatively the consequences of using a weaker design. (e.g. as in the lighting study and looking at design. (e.g. as in the lighting study and looking at other crime data. )other crime data. )

Weaker designs can be useful in preliminary Weaker designs can be useful in preliminary research but it is doubtful they are adequate when research but it is doubtful they are adequate when there are expensive consequences.there are expensive consequences.

RCTs can be problematic enough! (We need RCTs can be problematic enough! (We need registered trials, published protocols, declaration of registered trials, published protocols, declaration of interest, concealment of allocation, blinding etc…..)interest, concealment of allocation, blinding etc…..)

Evaluations of policies once implemented Evaluations of policies once implemented need to be done to a high scientific standard.need to be done to a high scientific standard.

3838

ReferencesReferences

Farrington D.P. and Welsh B.C. (2002) The Effects of Improved Street Lighting on Crime: A Systematic Review, Home Office Research Study 251, http://www.homeoffice.gov.uk/rds/pdfs2/hors251.pdf

Farrington D.P. and Welsh B.C. (2004) Measuring the Effects of Improved Street Lighting on Crime: A reply to Dr. Marchant The British Journal of Criminology 44 448-467 http://bjc.oupjournals.org/cgi/content/abstract/44/3/448

Farrington D.P. and Welsh B.C. (2006) How Important is Regression to the Mean in Area-Based Crime Prevention Research?, Crime Prevention and Community Safety 8 50

Feynman RP (1985) Cargo Cult science in Surely You’re Joking Mr Feynman Norton New York

Gorard S (2002) Fostering Scepticism: The Importance of Warranting Claims, Evaluation and Research in Education 16 3 p136

Marchant P.R. (2004) A Demonstration that the Claim that Brighter Lighting Reduces Crime is Unfounded The British Journal of Criminology 44 441-447 http://bjc.oupjournals.org/cgi/content/abstract/44/3/441

3939

References continuedReferences continuedMarchant P.R. (2005) What Works? A Critical Note on the

Evaluation of Crime Reduction Initiatives,Crime Prevention and Community Safety 7 7-13

Painter, K. and Farrington, D. P. (1997) The Crime Reducing Effect of Improved Street Lighting: The Dudley Project, in R.V. Clarke ed., Situational Crime Prevention: Successful case studies 209-226 Harrow and Heston, Guilderland NY.

Shaftoe, H (1994) Easton/Ashley, Bristol: Lighting Improvements, in S. Osborn (ed.) Housing Safe Communities: An Evaluation of Recent Initiatives 72-77, Safe Neighbourhoods Unit, London

Tilley N., Pease K., Hough M. and Brown R. (1999) Burglary Prevention: Early Lessons from the Crime Reduction Programme, Crime Reduction Research series Paper1 London Home Office

Wrigley N., Revisiting the Modifiable Areal Unit Problem and Ecological Fallacy pp49-71 in Gould PR, Hoare AG and Cliff AD Eds Diffusing Geography: Essays for Peter Haggett