Jour

nal o

f Cel

l Sci

ence

STICKY WICKET

Replicant II – Tears in the rain

Mole

I’ve seen things you people wouldn’t believe. Attack ships on fire

off the shoulder of Orion. I watched C-beams glitter in the dark

near the Tannhauser gate. All those moments will be lost in

time… like tears in rain…

And then Blatty says, ‘‘Time to die’’ and he dies. In the pouring

rain. Oh man. For those of you just joining us, I’ve been watching

Blade Runner, again. And we’ve been talking about replicants.

Not the artificial humans in the movie, but published results that

can’t be replicated. Replicants. Get it?

There is a growing perception that many, or even most of the

claims made in the literature cannot be replicated. While this

isn’t a new problem (the Greeks complained about this I think,

or they should have – I mean, putting golden crowns in

bathtubs isn’t something you can try to repeat in your hut), the

tried and true approach of letting the community work things

out over time just isn’t fast enough in our modern, I-need-it-

yesterday world. We need to know, up front, what is worthCorrespondence for Mole and his friends can be sent to mole@biologists.com,and may be published in forthcoming issues.

An occasional column in which Mole and other characters share their views on various aspects of life-science research.

� 2014. Published by The Company of Biologists Ltd | Journal of Cell Science (2014) 127, 2123–2125 doi:10.1242/jcs.155598

2123

Jour

nal o

f Cel

l Sci

ence

putting our time into and what isn’t, so that we can moveforward, and now.

Very serious people have been thinking about this seriously.Seriously they have. Does that sound a bit frivolous? Perhaps itshould. In general, whenever I’ve noticed an old problem (such asthis) suddenly coming to the fore as urgent, there is a motivation

that boils down to cold, hard cash behind it. And that’s exactlywhat’s going on here: as our governments embrace austerity (sothat we’ll have enough money to bail out the ultra-rich the next

time they fumble the ball, again) the amount of money availablefor research is shrinking (or at least not growing). So of course wepoint out how, well, stupid this is, since modern, first world

economies are largely fueled by scientific discoveries. This leadsto push-back, noting that, well, no, modern, first world economiesare largely fueled by moving money (in the form of electrons)

around, and besides, the ‘science’ we do doesn’t lead todiscoveries anyway, because the findings are mostly replicants.

Maybe I’m being paranoid. (That said, having grown up in thedays of hippie-dom, I am reminded that just because you’re

paranoid doesn’t mean that they’re not out to get you. Illegitimi

non carborundum!). But it may be a useful exercise to examinethe quite serious recommendations of these serious folks, as some

of them are pretty reasonable. But as we will see, some are not.Let’s look at them one by one.

1. We need to train our people better. See, right off the bat I

completely agree with this. Once upon a time, some, or evenmost, of the students who entered a Ph.D. program did not comeout the other side as Ph.Ds. And once upon a time this was

viewed as a good thing, because there weren’t very many jobs forPh.D. scientists and those who were allowed to continue werethose deemed most likely to succeed. But there are a lot ofreasons why this practice hasn’t continued. One of them is that

programs competed for money and a metric was needed todetermine which ones were most successful, and of course thesimplest metric was the number of entering students who

completed the program. So naturally everyone had to succeed.But more than that, the whole reason to have students was to havehands in the lab, and spending time training and testing students

is now considered more or less a waste of invaluable bench time.An important reason for this comes next. Personally, I think it isessential to drill into trainees the idea that what they publish hasto stand the tests of time if they want to ultimately be successful.

If they have doubts, the work should not be published. Yeh, Iknow, good luck with that. So while I agree that we need to trainpeople much more rigorously, the system itself would have to

change. This is because:2. We need to remove the rewards for publishing results

regardless of their validity. OMG, I’m agreeing again! That almost

never happens. Yes, it would be lovely if we could find a way to dojust that, reward work that is validated and moves the field forward,and not just anything that makes its way into the literature. But how

do we do this? The approach we have generally taken is to use ametric: How often has a finding been cited? If it is cited a lot, it isimportant. And if it has been cited a lot, it may even be a‘landmark’. As we saw last time, though, many ‘landmark’

findings of this sort apparently cannot be replicated – they arereplicants – and this invalidates the entire approach. So we shouldhold off on the decision of the validity of a finding until it has been

deemed useful, right? And how do we do this? One idea is:3. We need to make it is easier to publish negative results.

Holey Moley! Again I agree! This could turn out to be a record.

Well, I agree for the most part. There are really two sorts of

negative result. The first sort is a controlled negative result. Itmay not be absolutely, positively, negative, but it is a result that

convincingly shows that the conclusions of a study are not validlysupported by another study. We really should know about suchresults. For example, a clinical trial is conducted that shows that aparticular drug has no efficacy in a patient population – often a

company that conducts such a study feels no obligation to makethis finding public (I suppose it could hurt their stock value, orthey might be afraid that it would). It would be pretty easy,

though, for regulatory agencies to insist that results of all trials bemade public as a condition of filing with the agency in the firstplace, and this would be invaluable for other researchers

(corporate or academic) in furthering the efforts. Or as anotherexample: attempts by another lab to rigorously test theconclusions of a study produce clear-cut results that cast doubt

on the validity of a result. Generally, journals find such studies,however rigorous, not very interesting (indeed, the journals thatpublished the original finding generally feel that to publish thecounter study would cast them in a bad light, and often demur).

Sure, we can say that the journal is obligated to publish solidevidence that another paper’s conclusions in the same journalmay be wrong, but journals don’t actually have to do anything we

say. So, in general, negative results are either not published, orare relegated to much lower impact journals. But there really issomething we can do about this: cite them. The more we cite

counter-examples in our own papers (even focusing our citationon the one we think is the right one), the value of publishingnegative results will rise, and such increased citation will increase

the demand for such publications. So rather than saying that Beeet al. say that it’s so but Wasp et al. say it isn’t, we could just saythat Wasp et al. tested an idea but found it lacking (why give Beeet al. credit for saying something we think is wrong?). All useful

to think about. But as serious minds have noted, we just don’thave time to check out every little thing before we proceed todevelop the research further. And we certainly cannot hold up

promotions and grants and publications (sorry, stop, reverse that,as Willy Wonka would say) awaiting validation of importantresults. While it is a lovely idea to untether decisions regarding

scientific achievement and its rewards from the status of thepublication, we have no such mechanism. So, the pundits suggest:

4. We need to devise ways to punish publication of invalid

results. Oh boy. I can see how this might be appealing to some.

Currently, the system rewards publication itself, with no penaltyfor publishing something that is simply wrong. If, on the otherhand, publishing something that others cannot replicate carried

with it certain penalties (loss of funding, loss of prestige, beingforced to wear a big loser ‘L’ on your head), we would makesuper-sure that what we published was right, right, right. Right?

To some extent, many of us already link our personal value asscientists to the notion that what we report is correct, and we’ll doanything to show others that we have it right. Whole careers have

been built on working to support an idea that takes years toconvince the community of its validity. When Darwin reportedhis observations that supported the concept of natural selection,he was assailed with counter-examples that might discredit the

idea. Arguably (and this is an argument made, not by me, but byStephen J. Gould), Darwin spent the rest of his career building thecase for natural selection as the basis for the diversity of life. So, I

put it to you, esteemed reader (sorry, when I think of such things,I go all nineteenth century), should we have cut Darwin off at thefirst sign that his idea might not be correct? Yeh, that would have

been bitchin’. (Okay, back to the twenty-first century – way

STICKY WICKET Journal of Cell Science (2014) 127, 2123–2125 doi:10.1242/jcs.155598

2124

Jour

nal o

f Cel

l Sci

ence

better.) Clearly, the problem here is this: who gets to decide thatsomething is correct, or not? Which leads to:

5. Take this out of the hands of the scientific community, who

routinely publish replicants. Now we’re getting seriously serious.Indeed, this has not only been suggested, but pushed – there shouldbe impartial groups that have the mandate to test important findings

to determine if they can be replicated. Perhaps these can even becompanies that do this for a living. Surprise! At least one suchcompany already exists – and bigger surprise!! – the company’s CEO

is one of the people pushing for such validation as a requirement forobtaining future funds. How could anything go wrong? I really,really hope that you find this idea as utterly awful as I do. If you

don’t, let’s talk about it more. If you do, well, stick with me anyway,because you already agree and I need all the help I can get.

Fortunately, a proposed plan to require independent validationof all preliminary results as a prerequisite for government

support of research died an early death. But the idea persists.And this is, in part, because we just don’t know what to do. Hey,this is Mole here. Of course, I have some more ideas. But myreason for wanting to do something is not based on the need to

eradicate the literature of things I don’t agree with. It’s based onwhat Blatty said. Because if we cannot publish what we haveseen, even if someone else hasn’t seen the same thing, then

something that may be genuinely valuable may simply be gone.Lost in time. Like tears in the rain. And I don’t want to see thathappen.

Oh, now I’ve gotten all weepy again. I’m going to go watchWho Framed Roger Rabbit.

STICKY WICKET Journal of Cell Science (2014) 127, 2123–2125 doi:10.1242/jcs.155598

2125