0:00:00.671,0:00:03.456
Hi. So, this chap here,

0:00:03.456,0:00:05.718
he thinks he can tell you the future.

0:00:05.718,0:00:07.697
His name is Nostradamus, although here the Sun have

0:00:07.697,0:00:11.367
made him look a little bit like Sean Connery. (Laughter)

0:00:11.367,0:00:14.270
And like most of you, I suspect, I don't really believe

0:00:14.270,0:00:15.378
that people can see into the future.

0:00:15.378,0:00:18.082
I don't believe in precognition, and every now and then,

0:00:18.082,0:00:21.264
you hear that somebody has been able to predict something that happened in the future,

0:00:21.264,0:00:24.323
and that's probably because it was a fluke, and we only

0:00:24.323,0:00:26.938
hear about the flukes and about the freaks.

0:00:26.938,0:00:31.017
We don't hear about all the times that people got stuff wrong.

0:00:31.017,0:00:33.177
Now we expect that to happen with silly stories

0:00:33.177,0:00:36.316
about precognition, but the problem is,

0:00:36.316,0:00:39.660
we have exactly the same problem in academia

0:00:39.660,0:00:44.161
and in medicine, and in this environment, it costs lives.

0:00:44.161,0:00:47.526
So firstly, thinking just about precognition, as it turns out,

0:00:47.526,0:00:50.055
just last year a researcher called Daryl Bem conducted

0:00:50.055,0:00:51.891
a piece of research where he found evidence

0:00:51.891,0:00:55.549
of precognitive powers in undergraduate students,

0:00:55.549,0:00:58.123
and this was published in a peer-reviewed academic journal

0:00:58.123,0:01:00.404
and most of the people who read this just said, "Okay, well,

0:01:00.404,0:01:02.585
fair enough, but I think that's a fluke, that's a freak, because I know

0:01:02.585,0:01:05.419
that if I did a study where I found no evidence

0:01:05.419,0:01:07.931
that undergraduate students had precognitive powers,

0:01:07.931,0:01:11.483
it probably wouldn't get published in a journal.

0:01:11.483,0:01:14.338
And in fact, we know that that's true, because

0:01:14.338,0:01:16.867
several different groups of research scientists tried

0:01:16.867,0:01:20.387
to replicate the findings of this precognition study,

0:01:20.387,0:01:23.022
and when they submitted it to the exact same journal,

0:01:23.022,0:01:26.174
the journal said, "No, we're not interested in publishing

0:01:26.174,0:01:30.699
replication. We're not interested in your negative data."

0:01:30.699,0:01:33.453
So this is already evidence of how, in the academic

0:01:33.453,0:01:38.323
literature, we will see a biased sample of the true picture

0:01:38.323,0:01:41.790
of all of the scientific studies that have been conducted.

0:01:41.790,0:01:46.219
But it doesn't just happen in the dry academic field of psychology.

0:01:46.219,0:01:50.586
It also happens in, for example, cancer research.

0:01:50.586,0:01:54.663
So in March, 2012, just one month ago, some researchers

0:01:54.663,0:01:57.559
reported in the journal Nature how they had tried

0:01:57.559,0:02:01.405
to replicate 53 different basic science studies looking at

0:02:01.405,0:02:04.960
potential treatment targets in cancer,

0:02:04.960,0:02:07.598
and out of those 53 studies, they were only able

0:02:07.598,0:02:10.674
to successfully replicate six.

0:02:10.674,0:02:15.007
Forty-seven out of those 53 were unreplicable.

0:02:15.007,0:02:18.920
And they say in their discussion that this is very likely

0:02:18.920,0:02:21.559
because freaks get published.

0:02:21.559,0:02:23.655
People will do lots and lots and lots of different studies,

0:02:23.655,0:02:25.775
and the occasions when it works they will publish,

0:02:25.775,0:02:27.454
and the ones where it doesn't work they won't.

0:02:27.454,0:02:31.395
And their first recommendation of how to fix this problem,

0:02:31.395,0:02:34.684
because it is a problem, because it sends us all down blind alleys,

0:02:34.684,0:02:36.390
their first recommendation of how to fix this problem

0:02:36.390,0:02:39.783
is to make it easier to publish negative results in science,

0:02:39.783,0:02:42.690
and to change the incentives so that scientists are

0:02:42.690,0:02:47.042
encouraged to post more of their negative results in public.

0:02:47.042,0:02:50.894
But it doesn't just happen in the very dry world

0:02:50.894,0:02:54.745
of preclinical basic science cancer research.

0:02:54.745,0:02:58.402
It also happens in the very real, flesh and blood

0:02:58.402,0:03:01.993
of academic medicine. So in 1980,

0:03:01.993,0:03:05.001
some researchers did a study on a drug called lorcainide,

0:03:05.001,0:03:07.332
and this was an anti-arrhythmic drug,

0:03:07.332,0:03:09.583
a drug that suppresses abnormal heart rhythms,

0:03:09.598,0:03:11.826
and the idea was, after people have had a heart attack,

0:03:11.826,0:03:13.363
they're quite likely to have abnormal heart rhythms,

0:03:13.363,0:03:15.740
so if we give them a drug that suppresses abnormal heart

0:03:15.740,0:03:19.453
rhythms, this will increase the chances of them surviving.

0:03:19.453,0:03:22.461
Early on its development, they did a very small trial,

0:03:22.461,0:03:24.105
just under a hundred patients.

0:03:24.105,0:03:27.757
Fifty patients got lorcainide, and of those patients, 10 died.

0:03:27.757,0:03:30.800
Another 50 patients got a dummy placebo sugar pill

0:03:30.800,0:03:33.758
with no active ingredient, and only one of them died.

0:03:33.758,0:03:36.407
So they rightly regarded this drug as a failure,

0:03:36.407,0:03:39.276
and its commercial development was stopped, and because

0:03:39.276,0:03:43.624
its commercial development was stopped, this trial was never published.

0:03:43.624,0:03:49.021
Unfortunately, over the course of the next five, 10 years,

0:03:49.021,0:03:52.846
other companies had the same idea about drugs that would

0:03:52.846,0:03:55.438
prevent arrhythmias in people who have had heart attacks.

0:03:55.438,0:03:57.158
These drugs were brought to market. They were prescribed

0:03:57.158,0:04:00.570
very widely because heart attacks are a very common thing,

0:04:00.570,0:04:04.413
and it took so long for us to find out that these drugs

0:04:04.413,0:04:07.324
also caused an increased rate of death

0:04:07.324,0:04:10.071
that before we detected that safety signal,

0:04:10.071,0:04:16.122
over 100,000 people died unnecessarily in America

0:04:16.122,0:04:19.573
from the prescription of anti-arrhythmic drugs.

0:04:19.573,0:04:23.171
Now actually, in 1993,

0:04:23.171,0:04:26.731
the researchers who did that 1980 study, that early study,

0:04:26.731,0:04:30.572
published a mea culpa, an apology to the scientific community,

0:04:30.572,0:04:33.697
in which they said, "When we carried out our study in 1980,

0:04:33.697,0:04:35.633
we thought that the increased death rate that occurred

0:04:35.633,0:04:38.991
in the lorcainide group was an effect of chance."

0:04:38.991,0:04:41.023
The development of lorcainide was abandoned for commercial reasons,

0:04:41.023,0:04:42.661
and this study was never published;

0:04:42.661,0:04:45.047
it's now a good example of publication bias.

0:04:45.047,0:04:46.938
That's the technical term for the phenomenon where

0:04:46.938,0:04:51.176
unflattering data gets lost, gets unpublished, is left

0:04:51.176,0:04:54.547
missing in action, and they say the results described here

0:04:54.547,0:04:59.355
"might have provided an early warning of trouble ahead."

0:04:59.355,0:05:02.568
Now these are stories from basic science.

0:05:02.568,0:05:07.355
These are stories from 20, 30 years ago.

0:05:07.355,0:05:10.502
The academic publishing environment is very different now.

0:05:10.502,0:05:14.497
There are academic journals like "Trials," the open access journal,

0:05:14.497,0:05:17.152
which will publish any trial conducted in humans

0:05:17.152,0:05:20.455
regardless of whether it has a positive or a negative result.

0:05:20.455,0:05:24.424
But this problem of negative results that go missing in action

0:05:24.424,0:05:27.983
is still very prevalent. In fact it's so prevalent

0:05:27.983,0:05:33.834
that it cuts to the core of evidence-based medicine.

0:05:33.834,0:05:36.849
So this is a drug called reboxetine, and this is a drug

0:05:36.849,0:05:39.384
that I myself have prescribed. It's an antidepressant.

0:05:39.384,0:05:41.920
And I'm a very nerdy doctor, so I read all of the studies

0:05:41.920,0:05:44.972
that I could on this drug. I read the one study that was published

0:05:44.972,0:05:47.919
that showed that reboxetine was better than placebo,

0:05:47.919,0:05:49.783
and I read the other three studies that were published

0:05:49.783,0:05:53.354
that showed that reboxetine was just as good as any other antidepressant,

0:05:53.354,0:05:55.541
and because this patient hadn't done well on those other antidepressants,

0:05:55.541,0:05:58.007
I thought, well, reboxetine is just as good. It's one to try.

0:05:58.007,0:06:01.399
But it turned out that I was misled. In fact,

0:06:01.399,0:06:03.848
seven trials were conducted comparing reboxetine

0:06:03.848,0:06:06.560
against a dummy placebo sugar pill. One of them

0:06:06.560,0:06:08.872
was positive and that was published, but six of them

0:06:08.872,0:06:12.920
were negative and they were left unpublished.

0:06:12.920,0:06:14.659
Three trials were published comparing reboxetine

0:06:14.659,0:06:16.885
against other antidepressants in which reboxetine

0:06:16.885,0:06:18.678
was just as good, and they were published,

0:06:18.678,0:06:23.067
but three times as many patients' worth of data was collected

0:06:23.067,0:06:24.938
which showed that reboxetine was worse than

0:06:24.938,0:06:29.639
those other treatments, and those trials were not published.

0:06:29.639,0:06:33.398
I felt misled.

0:06:33.398,0:06:35.528
Now you might say, well, that's an extremely unusual example,

0:06:35.528,0:06:37.536
and I wouldn't want to be guilty of the same kind of

0:06:37.536,0:06:40.517
cherry-picking and selective referencing

0:06:40.517,0:06:42.308
that I'm accusing other people of.

0:06:42.308,0:06:44.192
But it turns out that this phenomenon of publication bias

0:06:44.192,0:06:46.319
has actually been very, very well studied.

0:06:46.319,0:06:48.537
So here is one example of how you approach it.

0:06:48.537,0:06:50.977
The classic model is, you get a bunch of studies where

0:06:50.977,0:06:53.162
you know that they've been conducted and completed,

0:06:53.162,0:06:55.483
and then you go and see if they've been published anywhere

0:06:55.483,0:06:58.346
in the academic literature. So this took all of the trials

0:06:58.346,0:07:00.500
that had ever been conducted on antidepressants

0:07:00.500,0:07:04.142
that were approved over a 15-year period by the FDA.

0:07:04.142,0:07:07.898
They took all of the trials which were submitted to the FDA as part of the approval package.

0:07:07.898,0:07:11.098
So that's not all of the trials that were ever conducted on these drugs,

0:07:11.098,0:07:13.196
because we can never know if we have those,

0:07:13.196,0:07:16.690
but it is the ones that were conducted in order to get the marketing authorization.

0:07:16.690,0:07:19.039
And then they went to see if these trials had been published

0:07:19.039,0:07:21.611
in the peer-reviewed academic literature. And this is what they found.

0:07:21.611,0:07:24.780
It was pretty much a 50-50 split. Half of these trials

0:07:24.780,0:07:28.377
were positive, half of them were negative, in reality.

0:07:28.377,0:07:33.118
But when they went to look for these trials in the peer-reviewed academic literature,

0:07:33.118,0:07:35.352
what they found was a very different picture.

0:07:35.352,0:07:39.724
Only three of the negative trials were published,

0:07:39.724,0:07:44.366
but all but one of the positive trials were published.

0:07:44.366,0:07:48.127
Now if we just flick back and forth between those two,

0:07:48.127,0:07:50.721
you can see what a staggering difference there was

0:07:50.721,0:07:54.171
between reality and what doctors, patients,

0:07:54.171,0:07:56.793
commissioners of health services, and academics

0:07:56.793,0:08:00.074
were able to see in the peer-reviewed academic literature.

0:08:00.074,0:08:04.528
We were misled, and this is a systematic flaw

0:08:04.528,0:08:07.858
in the core of medicine.

0:08:07.858,0:08:10.521
In fact, there have been so many studies conducted on

0:08:10.521,0:08:13.905
publication bias now, over a hundred, that they've been

0:08:13.905,0:08:17.099
collected in a systematic review, published in 2010,

0:08:17.099,0:08:19.865
that took every single study on publication bias

0:08:19.865,0:08:21.164
that they could find.

0:08:21.164,0:08:24.016
Publication bias affects every field of medicine.

0:08:24.016,0:08:28.329
About half of all trials, on average, go missing in action,

0:08:28.329,0:08:31.387
and we know that positive findings are around twice as likely

0:08:31.387,0:08:34.441
to be published as negative findings.

0:08:34.441,0:08:38.502
This is a cancer at the core of evidence-based medicine.

0:08:38.502,0:08:42.373
If I flipped a coin 100 times but then

0:08:42.373,0:08:45.632
withheld the results from you from half of those tosses,

0:08:45.632,0:08:49.032
I could make it look as if I had a coin that always came up heads.

0:08:49.032,0:08:50.838
But that wouldn't mean that I had a two-headed coin.

0:08:50.853,0:08:52.565
That would mean that I was a chancer

0:08:52.565,0:08:55.679
and you were an idiot for letting me get away with it. (Laughter)

0:08:55.679,0:08:59.316
But this is exactly what we blindly tolerate

0:08:59.316,0:09:03.105
in the whole of evidence-based medicine.

0:09:03.105,0:09:07.537
And to me, this is research misconduct.

0:09:07.537,0:09:10.280
If I conducted one study and I withheld

0:09:10.280,0:09:13.280
half of the data points from that one study,

0:09:13.280,0:09:17.987
you would rightly accuse me, essentially, of research fraud.

0:09:17.987,0:09:20.770
And yet, for some reason, if somebody conducts

0:09:20.770,0:09:25.328
10 studies but only publishes the five that give the result that they want,

0:09:25.328,0:09:28.116
we don't consider that to be research misconduct.

0:09:28.116,0:09:30.683
And when that responsibility is diffused between

0:09:30.683,0:09:33.844
a whole network of researchers, academics,

0:09:33.844,0:09:37.372
industry sponsors, journal editors, for some reason

0:09:37.372,0:09:38.825
we find it more acceptable,

0:09:38.825,0:09:42.500
but the effect on patients is damning.

0:09:42.500,0:09:47.518
And this is happening right now, today.

0:09:47.518,0:09:50.229
This is a drug called Tamiflu. Tamiflu is a drug

0:09:50.229,0:09:52.825
which governments around the world have spent billions

0:09:52.825,0:09:55.397
and billions of dollars on stockpiling,

0:09:55.397,0:09:58.545
and we've stockpiled Tamiflu in panic,

0:09:58.545,0:10:02.494
in the belief that it will reduce the rate of complications of influenza.

0:10:02.494,0:10:05.178
Complications is a medical euphemism for pneumonia

0:10:05.178,0:10:09.992
and death. (Laughter)

0:10:09.992,0:10:13.200
Now when the Cochrane systematic reviewers

0:10:13.200,0:10:15.725
were trying to collect together all of the data from all

0:10:15.725,0:10:19.373
of the trials that had ever been conducted on whether Tamiflu actually did this or not,

0:10:19.373,0:10:22.324
they found that several of those trials were unpublished.

0:10:22.324,0:10:24.166
The results were unavailable to them.

0:10:24.166,0:10:28.130
And when they started obtaining the writeups of those trials through various different means,

0:10:28.130,0:10:29.812
through Freedom of Information Act requests, through

0:10:29.812,0:10:34.621
harassing various different organizations, what they found was inconsistent.

0:10:34.621,0:10:37.087
And when they tried to get a hold of the clinical study reports,

0:10:37.087,0:10:40.133
the 10,000-page long documents that have

0:10:40.133,0:10:43.733
the best possible rendition of the information,

0:10:43.733,0:10:46.621
they were told they weren't allowed to have them.

0:10:46.621,0:10:49.304
And if you want to read the full correspondence

0:10:49.304,0:10:52.594
and the excuses and the explanations given by the drug company,

0:10:52.594,0:10:55.311
you can see that written up in this week's edition

0:10:55.311,0:10:59.678
of PLOS Medicine.

0:10:59.678,0:11:03.537
And the most staggering thing of all of this, to me,

0:11:03.537,0:11:06.836
is that not only is this a problem, not only do we recognize

0:11:06.836,0:11:11.031
that this is a problem, but we've had to suffer fake fixes.

0:11:11.031,0:11:14.089
We've had people pretend that this is a problem that's been fixed.

0:11:14.089,0:11:16.277
First of all, we had trials registers, and everybody said,

0:11:16.277,0:11:19.880
oh, it's okay. We'll get everyone to register their trials, they'll post the protocol,

0:11:19.880,0:11:21.904
they'll say what they're going to do before they do it,

0:11:21.904,0:11:24.025
and then afterwards we'll be able to check and see if all the trials which

0:11:24.025,0:11:26.493
have been conducted and completed have been published.

0:11:26.493,0:11:28.689
But people didn't bother to use those registers.

0:11:28.689,0:11:31.308
And so then the International Committee of Medical Journal Editors came along,

0:11:31.308,0:11:32.851
and they said, oh, well, we will hold the line.

0:11:32.851,0:11:35.484
We won't publish any journals, we won't publish any trials,

0:11:35.484,0:11:38.166
unless they've been registered before they began.

0:11:38.166,0:11:41.697
But they didn't hold the line. In 2008, a study was conducted

0:11:41.697,0:11:44.712
which showed that half of all of trials published by journals

0:11:44.712,0:11:47.379
edited by members of the ICMJE

0:11:47.379,0:11:52.192
weren't properly registered, and a quarter of them weren't registered at all.

0:11:52.192,0:11:54.993
And then finally, the FDA Amendment Act was passed

0:11:54.993,0:11:57.342
a couple of years ago saying that everybody who conducts

0:11:57.342,0:12:00.785
a trial must post the results of that trial within one year.

0:12:00.785,0:12:04.881
And in the BMJ, in the first edition of January, 2012,

0:12:04.881,0:12:07.585
you can see a study which looks to see if people kept

0:12:07.585,0:12:11.304
to that ruling, and it turns out that only one in five

0:12:11.304,0:12:14.168
have done so.

0:12:14.168,0:12:17.451
This is a disaster.

0:12:17.451,0:12:21.015
We cannot know the true effects of the medicines

0:12:21.015,0:12:24.231
that we prescribe if we do not have access

0:12:24.231,0:12:27.411
to all of the information.

0:12:27.411,0:12:31.370
And this is not a difficult problem to fix.

0:12:31.370,0:12:36.498
We need to force people to publish all trials

0:12:36.498,0:12:39.469
conducted in humans, including the older trials,

0:12:39.469,0:12:43.414
because the FDA Amendment Act only asks that you publish the trials conducted after 2008,

0:12:43.414,0:12:46.027
and I don't know what world it is in which we're only

0:12:46.027,0:12:50.483
practicing medicine on the basis of trials that completed in the past two years.

0:12:50.483,0:12:52.588
We need to publish all trials in humans,

0:12:52.588,0:12:55.662
including the older trials, for all drugs in current use,

0:12:55.662,0:12:58.578
and you need to tell everyone you know

0:12:58.578,0:13:02.020
that this is a problem and that it has not been fixed.

0:13:02.020,0:13:04.971
Thank you very much. (Applause)

0:13:04.971,0:13:08.244
(Applause)