1
00:00:00,671 --> 00:00:03,456
Hi. So, this chap here,

2
00:00:03,456 --> 00:00:05,718
he thinks he can tell you the future.

3
00:00:05,718 --> 00:00:07,697
His name is Nostradamus, although here the Sun have

4
00:00:07,697 --> 00:00:11,367
made him look a little bit like Sean Connery. (Laughter)

5
00:00:11,367 --> 00:00:14,270
And like most of you, I suspect, I don't really believe

6
00:00:14,270 --> 00:00:15,378
that people can see into the future.

7
00:00:15,378 --> 00:00:18,082
I don't believe in precognition, and every now and then,

8
00:00:18,082 --> 00:00:21,264
you hear that somebody has been able to predict something that happened in the future,

9
00:00:21,264 --> 00:00:24,323
and that's probably because it was a fluke, and we only

10
00:00:24,323 --> 00:00:26,938
hear about the flukes and about the freaks.

11
00:00:26,938 --> 00:00:31,017
We don't hear about all the times that people got stuff wrong.

12
00:00:31,017 --> 00:00:33,177
Now we expect that to happen with silly stories

13
00:00:33,177 --> 00:00:36,316
about precognition, but the problem is,

14
00:00:36,316 --> 00:00:39,660
we have exactly the same problem in academia

15
00:00:39,660 --> 00:00:44,161
and in medicine, and in this environment, it costs lives.

16
00:00:44,161 --> 00:00:47,526
So firstly, thinking just about precognition, as it turns out,

17
00:00:47,526 --> 00:00:50,055
just last year a researcher called Daryl Bem conducted

18
00:00:50,055 --> 00:00:51,891
a piece of research where he found evidence

19
00:00:51,891 --> 00:00:55,549
of precognitive powers in undergraduate students,

20
00:00:55,549 --> 00:00:58,123
and this was published in a peer-reviewed academic journal

21
00:00:58,123 --> 00:01:00,404
and most of the people who read this just said, "Okay, well,

22
00:01:00,404 --> 00:01:02,585
fair enough, but I think that's a fluke, that's a freak, because I know

23
00:01:02,585 --> 00:01:05,419
that if I did a study where I found no evidence

24
00:01:05,419 --> 00:01:07,931
that undergraduate students had precognitive powers,

25
00:01:07,931 --> 00:01:11,483
it probably wouldn't get published in a journal.

26
00:01:11,483 --> 00:01:14,338
And in fact, we know that that's true, because

27
00:01:14,338 --> 00:01:16,867
several different groups of research scientists tried

28
00:01:16,867 --> 00:01:20,387
to replicate the findings of this precognition study,

29
00:01:20,387 --> 00:01:23,022
and when they submitted it to the exact same journal,

30
00:01:23,022 --> 00:01:26,174
the journal said, "No, we're not interested in publishing

31
00:01:26,174 --> 00:01:30,699
replication. We're not interested in your negative data."

32
00:01:30,699 --> 00:01:33,453
So this is already evidence of how, in the academic

33
00:01:33,453 --> 00:01:38,323
literature, we will see a biased sample of the true picture

34
00:01:38,323 --> 00:01:41,790
of all of the scientific studies that have been conducted.

35
00:01:41,790 --> 00:01:46,219
But it doesn't just happen in the dry academic field of psychology.

36
00:01:46,219 --> 00:01:50,586
It also happens in, for example, cancer research.

37
00:01:50,586 --> 00:01:54,663
So in March, 2012, just one month ago, some researchers

38
00:01:54,663 --> 00:01:57,559
reported in the journal Nature how they had tried

39
00:01:57,559 --> 00:02:01,405
to replicate 53 different basic science studies looking at

40
00:02:01,405 --> 00:02:04,960
potential treatment targets in cancer,

41
00:02:04,960 --> 00:02:07,598
and out of those 53 studies, they were only able

42
00:02:07,598 --> 00:02:10,674
to successfully replicate six.

43
00:02:10,674 --> 00:02:15,007
Forty-seven out of those 53 were unreplicable.

44
00:02:15,007 --> 00:02:18,920
And they say in their discussion that this is very likely

45
00:02:18,920 --> 00:02:21,559
because freaks get published.

46
00:02:21,559 --> 00:02:23,655
People will do lots and lots and lots of different studies,

47
00:02:23,655 --> 00:02:25,775
and the occasions when it works they will publish,

48
00:02:25,775 --> 00:02:27,454
and the ones where it doesn't work they won't.

49
00:02:27,454 --> 00:02:31,395
And their first recommendation of how to fix this problem,

50
00:02:31,395 --> 00:02:34,684
because it is a problem, because it sends us all down blind alleys,

51
00:02:34,684 --> 00:02:36,390
their first recommendation of how to fix this problem

52
00:02:36,390 --> 00:02:39,783
is to make it easier to publish negative results in science,

53
00:02:39,783 --> 00:02:42,690
and to change the incentives so that scientists are

54
00:02:42,690 --> 00:02:47,042
encouraged to post more of their negative results in public.

55
00:02:47,042 --> 00:02:50,894
But it doesn't just happen in the very dry world

56
00:02:50,894 --> 00:02:54,745
of preclinical basic science cancer research.

57
00:02:54,745 --> 00:02:58,402
It also happens in the very real, flesh and blood

58
00:02:58,402 --> 00:03:01,993
of academic medicine. So in 1980,

59
00:03:01,993 --> 00:03:05,001
some researchers did a study on a drug called lorcainide,

60
00:03:05,001 --> 00:03:07,332
and this was an anti-arrhythmic drug,

61
00:03:07,332 --> 00:03:09,583
a drug that suppresses abnormal heart rhythms,

62
00:03:09,598 --> 00:03:11,826
and the idea was, after people have had a heart attack,

63
00:03:11,826 --> 00:03:13,363
they're quite likely to have abnormal heart rhythms,

64
00:03:13,363 --> 00:03:15,740
so if we give them a drug that suppresses abnormal heart

65
00:03:15,740 --> 00:03:19,453
rhythms, this will increase the chances of them surviving.

66
00:03:19,453 --> 00:03:22,461
Early on its development, they did a very small trial,

67
00:03:22,461 --> 00:03:24,105
just under a hundred patients.

68
00:03:24,105 --> 00:03:27,757
Fifty patients got lorcainide, and of those patients, 10 died.

69
00:03:27,757 --> 00:03:30,800
Another 50 patients got a dummy placebo sugar pill

70
00:03:30,800 --> 00:03:33,758
with no active ingredient, and only one of them died.

71
00:03:33,758 --> 00:03:36,407
So they rightly regarded this drug as a failure,

72
00:03:36,407 --> 00:03:39,276
and its commercial development was stopped, and because

73
00:03:39,276 --> 00:03:43,624
its commercial development was stopped, this trial was never published.

74
00:03:43,624 --> 00:03:49,021
Unfortunately, over the course of the next five, 10 years,

75
00:03:49,021 --> 00:03:52,846
other companies had the same idea about drugs that would

76
00:03:52,846 --> 00:03:55,438
prevent arrhythmias in people who have had heart attacks.

77
00:03:55,438 --> 00:03:57,158
These drugs were brought to market. They were prescribed

78
00:03:57,158 --> 00:04:00,570
very widely because heart attacks are a very common thing,

79
00:04:00,570 --> 00:04:04,413
and it took so long for us to find out that these drugs

80
00:04:04,413 --> 00:04:07,324
also caused an increased rate of death

81
00:04:07,324 --> 00:04:10,071
that before we detected that safety signal,

82
00:04:10,071 --> 00:04:16,122
over 100,000 people died unnecessarily in America

83
00:04:16,122 --> 00:04:19,573
from the prescription of anti-arrhythmic drugs.

84
00:04:19,573 --> 00:04:23,171
Now actually, in 1993,

85
00:04:23,171 --> 00:04:26,731
the researchers who did that 1980 study, that early study,

86
00:04:26,731 --> 00:04:30,572
published a mea culpa, an apology to the scientific community,

87
00:04:30,572 --> 00:04:33,697
in which they said, "When we carried out our study in 1980,

88
00:04:33,697 --> 00:04:35,633
we thought that the increased death rate that occurred

89
00:04:35,633 --> 00:04:38,991
in the lorcainide group was an effect of chance."

90
00:04:38,991 --> 00:04:41,023
The development of lorcainide was abandoned for commercial reasons,

91
00:04:41,023 --> 00:04:42,661
and this study was never published;

92
00:04:42,661 --> 00:04:45,047
it's now a good example of publication bias.

93
00:04:45,047 --> 00:04:46,938
That's the technical term for the phenomenon where

94
00:04:46,938 --> 00:04:51,176
unflattering data gets lost, gets unpublished, is left

95
00:04:51,176 --> 00:04:54,547
missing in action, and they say the results described here

96
00:04:54,547 --> 00:04:59,355
"might have provided an early warning of trouble ahead."

97
00:04:59,355 --> 00:05:02,568
Now these are stories from basic science.

98
00:05:02,568 --> 00:05:07,355
These are stories from 20, 30 years ago.

99
00:05:07,355 --> 00:05:10,502
The academic publishing environment is very different now.

100
00:05:10,502 --> 00:05:14,497
There are academic journals like "Trials," the open access journal,

101
00:05:14,497 --> 00:05:17,152
which will publish any trial conducted in humans

102
00:05:17,152 --> 00:05:20,455
regardless of whether it has a positive or a negative result.

103
00:05:20,455 --> 00:05:24,424
But this problem of negative results that go missing in action

104
00:05:24,424 --> 00:05:27,983
is still very prevalent. In fact it's so prevalent

105
00:05:27,983 --> 00:05:33,834
that it cuts to the core of evidence-based medicine.

106
00:05:33,834 --> 00:05:36,849
So this is a drug called reboxetine, and this is a drug

107
00:05:36,849 --> 00:05:39,384
that I myself have prescribed. It's an antidepressant.

108
00:05:39,384 --> 00:05:41,920
And I'm a very nerdy doctor, so I read all of the studies

109
00:05:41,920 --> 00:05:44,972
that I could on this drug. I read the one study that was published

110
00:05:44,972 --> 00:05:47,919
that showed that reboxetine was better than placebo,

111
00:05:47,919 --> 00:05:49,783
and I read the other three studies that were published

112
00:05:49,783 --> 00:05:53,354
that showed that reboxetine was just as good as any other antidepressant,

113
00:05:53,354 --> 00:05:55,541
and because this patient hadn't done well on those other antidepressants,

114
00:05:55,541 --> 00:05:58,007
I thought, well, reboxetine is just as good. It's one to try.

115
00:05:58,007 --> 00:06:01,399
But it turned out that I was misled. In fact,

116
00:06:01,399 --> 00:06:03,848
seven trials were conducted comparing reboxetine

117
00:06:03,848 --> 00:06:06,560
against a dummy placebo sugar pill. One of them

118
00:06:06,560 --> 00:06:08,872
was positive and that was published, but six of them

119
00:06:08,872 --> 00:06:12,920
were negative and they were left unpublished.

120
00:06:12,920 --> 00:06:14,659
Three trials were published comparing reboxetine

121
00:06:14,659 --> 00:06:16,885
against other antidepressants in which reboxetine

122
00:06:16,885 --> 00:06:18,678
was just as good, and they were published,

123
00:06:18,678 --> 00:06:23,067
but three times as many patients' worth of data was collected

124
00:06:23,067 --> 00:06:24,938
which showed that reboxetine was worse than

125
00:06:24,938 --> 00:06:29,639
those other treatments, and those trials were not published.

126
00:06:29,639 --> 00:06:33,398
I felt misled.

127
00:06:33,398 --> 00:06:35,528
Now you might say, well, that's an extremely unusual example,

128
00:06:35,528 --> 00:06:37,536
and I wouldn't want to be guilty of the same kind of

129
00:06:37,536 --> 00:06:40,517
cherry-picking and selective referencing

130
00:06:40,517 --> 00:06:42,308
that I'm accusing other people of.

131
00:06:42,308 --> 00:06:44,192
But it turns out that this phenomenon of publication bias

132
00:06:44,192 --> 00:06:46,319
has actually been very, very well studied.

133
00:06:46,319 --> 00:06:48,537
So here is one example of how you approach it.

134
00:06:48,537 --> 00:06:50,977
The classic model is, you get a bunch of studies where

135
00:06:50,977 --> 00:06:53,162
you know that they've been conducted and completed,

136
00:06:53,162 --> 00:06:55,483
and then you go and see if they've been published anywhere

137
00:06:55,483 --> 00:06:58,346
in the academic literature. So this took all of the trials

138
00:06:58,346 --> 00:07:00,500
that had ever been conducted on antidepressants

139
00:07:00,500 --> 00:07:04,142
that were approved over a 15-year period by the FDA.

140
00:07:04,142 --> 00:07:07,898
They took all of the trials which were submitted to the FDA as part of the approval package.

141
00:07:07,898 --> 00:07:11,098
So that's not all of the trials that were ever conducted on these drugs,

142
00:07:11,098 --> 00:07:13,196
because we can never know if we have those,

143
00:07:13,196 --> 00:07:16,690
but it is the ones that were conducted in order to get the marketing authorization.

144
00:07:16,690 --> 00:07:19,039
And then they went to see if these trials had been published

145
00:07:19,039 --> 00:07:21,611
in the peer-reviewed academic literature. And this is what they found.

146
00:07:21,611 --> 00:07:24,780
It was pretty much a 50-50 split. Half of these trials

147
00:07:24,780 --> 00:07:28,377
were positive, half of them were negative, in reality.

148
00:07:28,377 --> 00:07:33,118
But when they went to look for these trials in the peer-reviewed academic literature,

149
00:07:33,118 --> 00:07:35,352
what they found was a very different picture.

150
00:07:35,352 --> 00:07:39,724
Only three of the negative trials were published,

151
00:07:39,724 --> 00:07:44,366
but all but one of the positive trials were published.

152
00:07:44,366 --> 00:07:48,127
Now if we just flick back and forth between those two,

153
00:07:48,127 --> 00:07:50,721
you can see what a staggering difference there was

154
00:07:50,721 --> 00:07:54,171
between reality and what doctors, patients,

155
00:07:54,171 --> 00:07:56,793
commissioners of health services, and academics

156
00:07:56,793 --> 00:08:00,074
were able to see in the peer-reviewed academic literature.

157
00:08:00,074 --> 00:08:04,528
We were misled, and this is a systematic flaw

158
00:08:04,528 --> 00:08:07,858
in the core of medicine.

159
00:08:07,858 --> 00:08:10,521
In fact, there have been so many studies conducted on

160
00:08:10,521 --> 00:08:13,905
publication bias now, over a hundred, that they've been

161
00:08:13,905 --> 00:08:17,099
collected in a systematic review, published in 2010,

162
00:08:17,099 --> 00:08:19,865
that took every single study on publication bias

163
00:08:19,865 --> 00:08:21,164
that they could find.

164
00:08:21,164 --> 00:08:24,016
Publication bias affects every field of medicine.

165
00:08:24,016 --> 00:08:28,329
About half of all trials, on average, go missing in action,

166
00:08:28,329 --> 00:08:31,387
and we know that positive findings are around twice as likely

167
00:08:31,387 --> 00:08:34,441
to be published as negative findings.

168
00:08:34,441 --> 00:08:38,502
This is a cancer at the core of evidence-based medicine.

169
00:08:38,502 --> 00:08:42,373
If I flipped a coin 100 times but then

170
00:08:42,373 --> 00:08:45,632
withheld the results from you from half of those tosses,

171
00:08:45,632 --> 00:08:49,032
I could make it look as if I had a coin that always came up heads.

172
00:08:49,032 --> 00:08:50,838
But that wouldn't mean that I had a two-headed coin.

173
00:08:50,853 --> 00:08:52,565
That would mean that I was a chancer

174
00:08:52,565 --> 00:08:55,679
and you were an idiot for letting me get away with it. (Laughter)

175
00:08:55,679 --> 00:08:59,316
But this is exactly what we blindly tolerate

176
00:08:59,316 --> 00:09:03,105
in the whole of evidence-based medicine.

177
00:09:03,105 --> 00:09:07,537
And to me, this is research misconduct.

178
00:09:07,537 --> 00:09:10,280
If I conducted one study and I withheld

179
00:09:10,280 --> 00:09:13,280
half of the data points from that one study,

180
00:09:13,280 --> 00:09:17,987
you would rightly accuse me, essentially, of research fraud.

181
00:09:17,987 --> 00:09:20,770
And yet, for some reason, if somebody conducts

182
00:09:20,770 --> 00:09:25,328
10 studies but only publishes the five that give the result that they want,

183
00:09:25,328 --> 00:09:28,116
we don't consider that to be research misconduct.

184
00:09:28,116 --> 00:09:30,683
And when that responsibility is diffused between

185
00:09:30,683 --> 00:09:33,844
a whole network of researchers, academics,

186
00:09:33,844 --> 00:09:37,372
industry sponsors, journal editors, for some reason

187
00:09:37,372 --> 00:09:38,825
we find it more acceptable,

188
00:09:38,825 --> 00:09:42,500
but the effect on patients is damning.

189
00:09:42,500 --> 00:09:47,518
And this is happening right now, today.

190
00:09:47,518 --> 00:09:50,229
This is a drug called Tamiflu. Tamiflu is a drug

191
00:09:50,229 --> 00:09:52,825
which governments around the world have spent billions

192
00:09:52,825 --> 00:09:55,397
and billions of dollars on stockpiling,

193
00:09:55,397 --> 00:09:58,545
and we've stockpiled Tamiflu in panic,

194
00:09:58,545 --> 00:10:02,494
in the belief that it will reduce the rate of complications of influenza.

195
00:10:02,494 --> 00:10:05,178
Complications is a medical euphemism for pneumonia

196
00:10:05,178 --> 00:10:09,992
and death. (Laughter)

197
00:10:09,992 --> 00:10:13,200
Now when the Cochrane systematic reviewers

198
00:10:13,200 --> 00:10:15,725
were trying to collect together all of the data from all

199
00:10:15,725 --> 00:10:19,373
of the trials that had ever been conducted on whether Tamiflu actually did this or not,

200
00:10:19,373 --> 00:10:22,324
they found that several of those trials were unpublished.

201
00:10:22,324 --> 00:10:24,166
The results were unavailable to them.

202
00:10:24,166 --> 00:10:28,130
And when they started obtaining the writeups of those trials through various different means,

203
00:10:28,130 --> 00:10:29,812
through Freedom of Information Act requests, through

204
00:10:29,812 --> 00:10:34,621
harassing various different organizations, what they found was inconsistent.

205
00:10:34,621 --> 00:10:37,087
And when they tried to get a hold of the clinical study reports,

206
00:10:37,087 --> 00:10:40,133
the 10,000-page long documents that have

207
00:10:40,133 --> 00:10:43,733
the best possible rendition of the information,

208
00:10:43,733 --> 00:10:46,621
they were told they weren't allowed to have them.

209
00:10:46,621 --> 00:10:49,304
And if you want to read the full correspondence

210
00:10:49,304 --> 00:10:52,594
and the excuses and the explanations given by the drug company,

211
00:10:52,594 --> 00:10:55,311
you can see that written up in this week's edition

212
00:10:55,311 --> 00:10:59,678
of PLOS Medicine.

213
00:10:59,678 --> 00:11:03,537
And the most staggering thing of all of this, to me,

214
00:11:03,537 --> 00:11:06,836
is that not only is this a problem, not only do we recognize

215
00:11:06,836 --> 00:11:11,031
that this is a problem, but we've had to suffer fake fixes.

216
00:11:11,031 --> 00:11:14,089
We've had people pretend that this is a problem that's been fixed.

217
00:11:14,089 --> 00:11:16,277
First of all, we had trials registers, and everybody said,

218
00:11:16,277 --> 00:11:19,880
oh, it's okay. We'll get everyone to register their trials, they'll post the protocol,

219
00:11:19,880 --> 00:11:21,904
they'll say what they're going to do before they do it,

220
00:11:21,904 --> 00:11:24,025
and then afterwards we'll be able to check and see if all the trials which

221
00:11:24,025 --> 00:11:26,493
have been conducted and completed have been published.

222
00:11:26,493 --> 00:11:28,689
But people didn't bother to use those registers.

223
00:11:28,689 --> 00:11:31,308
And so then the International Committee of Medical Journal Editors came along,

224
00:11:31,308 --> 00:11:32,851
and they said, oh, well, we will hold the line.

225
00:11:32,851 --> 00:11:35,484
We won't publish any journals, we won't publish any trials,

226
00:11:35,484 --> 00:11:38,166
unless they've been registered before they began.

227
00:11:38,166 --> 00:11:41,697
But they didn't hold the line. In 2008, a study was conducted

228
00:11:41,697 --> 00:11:44,712
which showed that half of all of trials published by journals

229
00:11:44,712 --> 00:11:47,379
edited by members of the ICMJE

230
00:11:47,379 --> 00:11:52,192
weren't properly registered, and a quarter of them weren't registered at all.

231
00:11:52,192 --> 00:11:54,993
And then finally, the FDA Amendment Act was passed

232
00:11:54,993 --> 00:11:57,342
a couple of years ago saying that everybody who conducts

233
00:11:57,342 --> 00:12:00,785
a trial must post the results of that trial within one year.

234
00:12:00,785 --> 00:12:04,881
And in the BMJ, in the first edition of January, 2012,

235
00:12:04,881 --> 00:12:07,585
you can see a study which looks to see if people kept

236
00:12:07,585 --> 00:12:11,304
to that ruling, and it turns out that only one in five

237
00:12:11,304 --> 00:12:14,168
have done so.

238
00:12:14,168 --> 00:12:17,451
This is a disaster.

239
00:12:17,451 --> 00:12:21,015
We cannot know the true effects of the medicines

240
00:12:21,015 --> 00:12:24,231
that we prescribe if we do not have access

241
00:12:24,231 --> 00:12:27,411
to all of the information.

242
00:12:27,411 --> 00:12:31,370
And this is not a difficult problem to fix.

243
00:12:31,370 --> 00:12:36,498
We need to force people to publish all trials

244
00:12:36,498 --> 00:12:39,469
conducted in humans, including the older trials,

245
00:12:39,469 --> 00:12:43,414
because the FDA Amendment Act only asks that you publish the trials conducted after 2008,

246
00:12:43,414 --> 00:12:46,027
and I don't know what world it is in which we're only

247
00:12:46,027 --> 00:12:50,483
practicing medicine on the basis of trials that completed in the past two years.

248
00:12:50,483 --> 00:12:52,588
We need to publish all trials in humans,

249
00:12:52,588 --> 00:12:55,662
including the older trials, for all drugs in current use,

250
00:12:55,662 --> 00:12:58,578
and you need to tell everyone you know

251
00:12:58,578 --> 00:13:02,020
that this is a problem and that it has not been fixed.

252
00:13:02,020 --> 00:13:04,971
Thank you very much. (Applause)

253
00:13:04,971 --> 00:13:08,244
(Applause)