WEBVTT 00:00:00.671 --> 00:00:03.456 Hi. So, this chap here, 00:00:03.456 --> 00:00:05.718 he thinks he can tell you the future. 00:00:05.718 --> 00:00:07.697 His name is Nostradamus, although here the Sun have 00:00:07.697 --> 00:00:11.367 made him look a little bit like Sean Connery. (Laughter) NOTE Paragraph 00:00:11.367 --> 00:00:14.270 And like most of you, I suspect, I don't really believe 00:00:14.270 --> 00:00:15.378 that people can see into the future. 00:00:15.378 --> 00:00:18.082 I don't believe in precognition, and every now and then, 00:00:18.082 --> 00:00:21.264 you hear that somebody has been able to predict something that happened in the future, 00:00:21.264 --> 00:00:24.323 and that's probably because it was a fluke, and we only 00:00:24.323 --> 00:00:26.938 hear about the flukes and about the freaks. 00:00:26.938 --> 00:00:31.017 We don't hear about all the times that people got stuff wrong. 00:00:31.017 --> 00:00:33.177 Now we expect that to happen with silly stories 00:00:33.177 --> 00:00:36.316 about precognition, but the problem is, 00:00:36.316 --> 00:00:39.660 we have exactly the same problem in academia 00:00:39.660 --> 00:00:44.161 and in medicine, and in this environment, it costs lives. NOTE Paragraph 00:00:44.161 --> 00:00:47.526 So firstly, thinking just about precognition, as it turns out, 00:00:47.526 --> 00:00:50.055 just last year a researcher called Daryl Bem conducted 00:00:50.055 --> 00:00:51.891 a piece of research where he found evidence 00:00:51.891 --> 00:00:55.549 of precognitive powers in undergraduate students, 00:00:55.549 --> 00:00:58.123 and this was published in a peer-reviewed academic journal 00:00:58.123 --> 00:01:00.404 and most of the people who read this just said, "Okay, well, 00:01:00.404 --> 00:01:02.585 fair enough, but I think that's a fluke, that's a freak, because I know 00:01:02.585 --> 00:01:05.419 that if I did a study where I found no evidence 00:01:05.419 --> 00:01:07.931 that undergraduate students had precognitive powers, 00:01:07.931 --> 00:01:11.483 it probably wouldn't get published in a journal. 00:01:11.483 --> 00:01:14.338 And in fact, we know that that's true, because 00:01:14.338 --> 00:01:16.867 several different groups of research scientists tried 00:01:16.867 --> 00:01:20.387 to replicate the findings of this precognition study, 00:01:20.387 --> 00:01:23.022 and when they submitted it to the exact same journal, 00:01:23.022 --> 00:01:26.174 the journal said, "No, we're not interested in publishing 00:01:26.174 --> 00:01:30.699 replication. We're not interested in your negative data." 00:01:30.699 --> 00:01:33.453 So this is already evidence of how, in the academic 00:01:33.453 --> 00:01:38.323 literature, we will see a biased sample of the true picture 00:01:38.323 --> 00:01:41.790 of all of the scientific studies that have been conducted. NOTE Paragraph 00:01:41.790 --> 00:01:46.219 But it doesn't just happen in the dry academic field of psychology. 00:01:46.219 --> 00:01:50.586 It also happens in, for example, cancer research. 00:01:50.586 --> 00:01:54.663 So in March, 2012, just one month ago, some researchers 00:01:54.663 --> 00:01:57.559 reported in the journal Nature how they had tried 00:01:57.559 --> 00:02:01.405 to replicate 53 different basic science studies looking at 00:02:01.405 --> 00:02:04.960 potential treatment targets in cancer, 00:02:04.960 --> 00:02:07.598 and out of those 53 studies, they were only able 00:02:07.598 --> 00:02:10.674 to successfully replicate six. 00:02:10.674 --> 00:02:15.007 Forty-seven out of those 53 were unreplicable. 00:02:15.007 --> 00:02:18.920 And they say in their discussion that this is very likely 00:02:18.920 --> 00:02:21.559 because freaks get published. 00:02:21.559 --> 00:02:23.655 People will do lots and lots and lots of different studies, 00:02:23.655 --> 00:02:25.775 and the occasions when it works they will publish, 00:02:25.775 --> 00:02:27.454 and the ones where it doesn't work they won't. 00:02:27.454 --> 00:02:31.395 And their first recommendation of how to fix this problem, 00:02:31.395 --> 00:02:34.684 because it is a problem, because it sends us all down blind alleys, 00:02:34.684 --> 00:02:36.390 their first recommendation of how to fix this problem 00:02:36.390 --> 00:02:39.783 is to make it easier to publish negative results in science, 00:02:39.783 --> 00:02:42.690 and to change the incentives so that scientists are 00:02:42.690 --> 00:02:47.042 encouraged to post more of their negative results in public. NOTE Paragraph 00:02:47.042 --> 00:02:50.894 But it doesn't just happen in the very dry world 00:02:50.894 --> 00:02:54.745 of preclinical basic science cancer research. 00:02:54.745 --> 00:02:58.402 It also happens in the very real, flesh and blood 00:02:58.402 --> 00:03:01.993 of academic medicine. So in 1980, 00:03:01.993 --> 00:03:05.001 some researchers did a study on a drug called lorcainide, 00:03:05.001 --> 00:03:07.332 and this was an anti-arrhythmic drug, 00:03:07.332 --> 00:03:09.583 a drug that suppresses abnormal heart rhythms, 00:03:09.598 --> 00:03:11.826 and the idea was, after people have had a heart attack, 00:03:11.826 --> 00:03:13.363 they're quite likely to have abnormal heart rhythms, 00:03:13.363 --> 00:03:15.740 so if we give them a drug that suppresses abnormal heart 00:03:15.740 --> 00:03:19.453 rhythms, this will increase the chances of them surviving. 00:03:19.453 --> 00:03:22.461 Early on its development, they did a very small trial, 00:03:22.461 --> 00:03:24.105 just under a hundred patients. 00:03:24.105 --> 00:03:27.757 Fifty patients got lorcainide, and of those patients, 10 died. 00:03:27.757 --> 00:03:30.800 Another 50 patients got a dummy placebo sugar pill 00:03:30.800 --> 00:03:33.758 with no active ingredient, and only one of them died. 00:03:33.758 --> 00:03:36.407 So they rightly regarded this drug as a failure, 00:03:36.407 --> 00:03:39.276 and its commercial development was stopped, and because 00:03:39.276 --> 00:03:43.624 its commercial development was stopped, this trial was never published. NOTE Paragraph 00:03:43.624 --> 00:03:49.021 Unfortunately, over the course of the next five, 10 years, 00:03:49.021 --> 00:03:52.846 other companies had the same idea about drugs that would 00:03:52.846 --> 00:03:55.438 prevent arrhythmias in people who have had heart attacks. 00:03:55.438 --> 00:03:57.158 These drugs were brought to market. They were prescribed 00:03:57.158 --> 00:04:00.570 very widely because heart attacks are a very common thing, 00:04:00.570 --> 00:04:04.413 and it took so long for us to find out that these drugs 00:04:04.413 --> 00:04:07.324 also caused an increased rate of death 00:04:07.324 --> 00:04:10.071 that before we detected that safety signal, 00:04:10.071 --> 00:04:16.122 over 100,000 people died unnecessarily in America 00:04:16.122 --> 00:04:19.573 from the prescription of anti-arrhythmic drugs. NOTE Paragraph 00:04:19.573 --> 00:04:23.171 Now actually, in 1993, 00:04:23.171 --> 00:04:26.731 the researchers who did that 1980 study, that early study, 00:04:26.731 --> 00:04:30.572 published a mea culpa, an apology to the scientific community, 00:04:30.572 --> 00:04:33.697 in which they said, "When we carried out our study in 1980, 00:04:33.697 --> 00:04:35.633 we thought that the increased death rate that occurred 00:04:35.633 --> 00:04:38.991 in the lorcainide group was an effect of chance." 00:04:38.991 --> 00:04:41.023 The development of lorcainide was abandoned for commercial reasons, 00:04:41.023 --> 00:04:42.661 and this study was never published; 00:04:42.661 --> 00:04:45.047 it's now a good example of publication bias. 00:04:45.047 --> 00:04:46.938 That's the technical term for the phenomenon where 00:04:46.938 --> 00:04:51.176 unflattering data gets lost, gets unpublished, is left 00:04:51.176 --> 00:04:54.547 missing in action, and they say the results described here 00:04:54.547 --> 00:04:59.355 "might have provided an early warning of trouble ahead." NOTE Paragraph 00:04:59.355 --> 00:05:02.568 Now these are stories from basic science. 00:05:02.568 --> 00:05:07.355 These are stories from 20, 30 years ago. 00:05:07.355 --> 00:05:10.502 The academic publishing environment is very different now. 00:05:10.502 --> 00:05:14.497 There are academic journals like "Trials," the open access journal, 00:05:14.497 --> 00:05:17.152 which will publish any trial conducted in humans 00:05:17.152 --> 00:05:20.455 regardless of whether it has a positive or a negative result. 00:05:20.455 --> 00:05:24.424 But this problem of negative results that go missing in action 00:05:24.424 --> 00:05:27.983 is still very prevalent. In fact it's so prevalent 00:05:27.983 --> 00:05:33.834 that it cuts to the core of evidence-based medicine. 00:05:33.834 --> 00:05:36.849 So this is a drug called reboxetine, and this is a drug 00:05:36.849 --> 00:05:39.384 that I myself have prescribed. It's an antidepressant. 00:05:39.384 --> 00:05:41.920 And I'm a very nerdy doctor, so I read all of the studies 00:05:41.920 --> 00:05:44.972 that I could on this drug. I read the one study that was published 00:05:44.972 --> 00:05:47.919 that showed that reboxetine was better than placebo, 00:05:47.919 --> 00:05:49.783 and I read the other three studies that were published 00:05:49.783 --> 00:05:53.354 that showed that reboxetine was just as good as any other antidepressant, 00:05:53.354 --> 00:05:55.541 and because this patient hadn't done well on those other antidepressants, 00:05:55.541 --> 00:05:58.007 I thought, well, reboxetine is just as good. It's one to try. 00:05:58.007 --> 00:06:01.399 But it turned out that I was misled. In fact, 00:06:01.399 --> 00:06:03.848 seven trials were conducted comparing reboxetine 00:06:03.848 --> 00:06:06.560 against a dummy placebo sugar pill. One of them 00:06:06.560 --> 00:06:08.872 was positive and that was published, but six of them 00:06:08.872 --> 00:06:12.920 were negative and they were left unpublished. 00:06:12.920 --> 00:06:14.659 Three trials were published comparing reboxetine 00:06:14.659 --> 00:06:16.885 against other antidepressants in which reboxetine 00:06:16.885 --> 00:06:18.678 was just as good, and they were published, 00:06:18.678 --> 00:06:23.067 but three times as many patients' worth of data was collected 00:06:23.067 --> 00:06:24.938 which showed that reboxetine was worse than 00:06:24.938 --> 00:06:29.639 those other treatments, and those trials were not published. 00:06:29.639 --> 00:06:33.398 I felt misled. NOTE Paragraph 00:06:33.398 --> 00:06:35.528 Now you might say, well, that's an extremely unusual example, 00:06:35.528 --> 00:06:37.536 and I wouldn't want to be guilty of the same kind of 00:06:37.536 --> 00:06:40.517 cherry-picking and selective referencing 00:06:40.517 --> 00:06:42.308 that I'm accusing other people of. 00:06:42.308 --> 00:06:44.192 But it turns out that this phenomenon of publication bias 00:06:44.192 --> 00:06:46.319 has actually been very, very well studied. 00:06:46.319 --> 00:06:48.537 So here is one example of how you approach it. 00:06:48.537 --> 00:06:50.977 The classic model is, you get a bunch of studies where 00:06:50.977 --> 00:06:53.162 you know that they've been conducted and completed, 00:06:53.162 --> 00:06:55.483 and then you go and see if they've been published anywhere 00:06:55.483 --> 00:06:58.346 in the academic literature. So this took all of the trials 00:06:58.346 --> 00:07:00.500 that had ever been conducted on antidepressants 00:07:00.500 --> 00:07:04.142 that were approved over a 15-year period by the FDA. 00:07:04.142 --> 00:07:07.898 They took all of the trials which were submitted to the FDA as part of the approval package. 00:07:07.898 --> 00:07:11.098 So that's not all of the trials that were ever conducted on these drugs, 00:07:11.098 --> 00:07:13.196 because we can never know if we have those, 00:07:13.196 --> 00:07:16.690 but it is the ones that were conducted in order to get the marketing authorization. 00:07:16.690 --> 00:07:19.039 And then they went to see if these trials had been published 00:07:19.039 --> 00:07:21.611 in the peer-reviewed academic literature. And this is what they found. 00:07:21.611 --> 00:07:24.780 It was pretty much a 50-50 split. Half of these trials 00:07:24.780 --> 00:07:28.377 were positive, half of them were negative, in reality. 00:07:28.377 --> 00:07:33.118 But when they went to look for these trials in the peer-reviewed academic literature, 00:07:33.118 --> 00:07:35.352 what they found was a very different picture. 00:07:35.352 --> 00:07:39.724 Only three of the negative trials were published, 00:07:39.724 --> 00:07:44.366 but all but one of the positive trials were published. 00:07:44.366 --> 00:07:48.127 Now if we just flick back and forth between those two, 00:07:48.127 --> 00:07:50.721 you can see what a staggering difference there was 00:07:50.721 --> 00:07:54.171 between reality and what doctors, patients, 00:07:54.171 --> 00:07:56.793 commissioners of health services, and academics 00:07:56.793 --> 00:08:00.074 were able to see in the peer-reviewed academic literature. 00:08:00.074 --> 00:08:04.528 We were misled, and this is a systematic flaw 00:08:04.528 --> 00:08:07.858 in the core of medicine. NOTE Paragraph 00:08:07.858 --> 00:08:10.521 In fact, there have been so many studies conducted on 00:08:10.521 --> 00:08:13.905 publication bias now, over a hundred, that they've been 00:08:13.905 --> 00:08:17.099 collected in a systematic review, published in 2010, 00:08:17.099 --> 00:08:19.865 that took every single study on publication bias 00:08:19.865 --> 00:08:21.164 that they could find. 00:08:21.164 --> 00:08:24.016 Publication bias affects every field of medicine. 00:08:24.016 --> 00:08:28.329 About half of all trials, on average, go missing in action, 00:08:28.329 --> 00:08:31.387 and we know that positive findings are around twice as likely 00:08:31.387 --> 00:08:34.441 to be published as negative findings. NOTE Paragraph 00:08:34.441 --> 00:08:38.502 This is a cancer at the core of evidence-based medicine. 00:08:38.502 --> 00:08:42.373 If I flipped a coin 100 times but then 00:08:42.373 --> 00:08:45.632 withheld the results from you from half of those tosses, 00:08:45.632 --> 00:08:49.032 I could make it look as if I had a coin that always came up heads. 00:08:49.032 --> 00:08:50.838 But that wouldn't mean that I had a two-headed coin. 00:08:50.853 --> 00:08:52.565 That would mean that I was a chancer 00:08:52.565 --> 00:08:55.679 and you were an idiot for letting me get away with it. (Laughter) 00:08:55.679 --> 00:08:59.316 But this is exactly what we blindly tolerate 00:08:59.316 --> 00:09:03.105 in the whole of evidence-based medicine. 00:09:03.105 --> 00:09:07.537 And to me, this is research misconduct. 00:09:07.537 --> 00:09:10.280 If I conducted one study and I withheld 00:09:10.280 --> 00:09:13.280 half of the data points from that one study, 00:09:13.280 --> 00:09:17.987 you would rightly accuse me, essentially, of research fraud. 00:09:17.987 --> 00:09:20.770 And yet, for some reason, if somebody conducts 00:09:20.770 --> 00:09:25.328 10 studies but only publishes the five that give the result that they want, 00:09:25.328 --> 00:09:28.116 we don't consider that to be research misconduct. 00:09:28.116 --> 00:09:30.683 And when that responsibility is diffused between 00:09:30.683 --> 00:09:33.844 a whole network of researchers, academics, 00:09:33.844 --> 00:09:37.372 industry sponsors, journal editors, for some reason 00:09:37.372 --> 00:09:38.825 we find it more acceptable, 00:09:38.825 --> 00:09:42.500 but the effect on patients is damning. NOTE Paragraph 00:09:42.500 --> 00:09:47.518 And this is happening right now, today. 00:09:47.518 --> 00:09:50.229 This is a drug called Tamiflu. Tamiflu is a drug 00:09:50.229 --> 00:09:52.825 which governments around the world have spent billions 00:09:52.825 --> 00:09:55.397 and billions of dollars on stockpiling, 00:09:55.397 --> 00:09:58.545 and we've stockpiled Tamiflu in panic, 00:09:58.545 --> 00:10:02.494 in the belief that it will reduce the rate of complications of influenza. 00:10:02.494 --> 00:10:05.178 Complications is a medical euphemism for pneumonia 00:10:05.178 --> 00:10:09.992 and death. (Laughter) 00:10:09.992 --> 00:10:13.200 Now when the Cochrane systematic reviewers 00:10:13.200 --> 00:10:15.725 were trying to collect together all of the data from all 00:10:15.725 --> 00:10:19.373 of the trials that had ever been conducted on whether Tamiflu actually did this or not, 00:10:19.373 --> 00:10:22.324 they found that several of those trials were unpublished. 00:10:22.324 --> 00:10:24.166 The results were unavailable to them. 00:10:24.166 --> 00:10:28.130 And when they started obtaining the writeups of those trials through various different means, 00:10:28.130 --> 00:10:29.812 through Freedom of Information Act requests, through 00:10:29.812 --> 00:10:34.621 harassing various different organizations, what they found was inconsistent. 00:10:34.621 --> 00:10:37.087 And when they tried to get a hold of the clinical study reports, 00:10:37.087 --> 00:10:40.133 the 10,000-page long documents that have 00:10:40.133 --> 00:10:43.733 the best possible rendition of the information, 00:10:43.733 --> 00:10:46.621 they were told they weren't allowed to have them. 00:10:46.621 --> 00:10:49.304 And if you want to read the full correspondence 00:10:49.304 --> 00:10:52.594 and the excuses and the explanations given by the drug company, 00:10:52.594 --> 00:10:55.311 you can see that written up in this week's edition 00:10:55.311 --> 00:10:59.678 of PLOS Medicine. NOTE Paragraph 00:10:59.678 --> 00:11:03.537 And the most staggering thing of all of this, to me, 00:11:03.537 --> 00:11:06.836 is that not only is this a problem, not only do we recognize 00:11:06.836 --> 00:11:11.031 that this is a problem, but we've had to suffer fake fixes. 00:11:11.031 --> 00:11:14.089 We've had people pretend that this is a problem that's been fixed. 00:11:14.089 --> 00:11:16.277 First of all, we had trials registers, and everybody said, 00:11:16.277 --> 00:11:19.880 oh, it's okay. We'll get everyone to register their trials, they'll post the protocol, 00:11:19.880 --> 00:11:21.904 they'll say what they're going to do before they do it, 00:11:21.904 --> 00:11:24.025 and then afterwards we'll be able to check and see if all the trials which 00:11:24.025 --> 00:11:26.493 have been conducted and completed have been published. 00:11:26.493 --> 00:11:28.689 But people didn't bother to use those registers. 00:11:28.689 --> 00:11:31.308 And so then the International Committee of Medical Journal Editors came along, 00:11:31.308 --> 00:11:32.851 and they said, oh, well, we will hold the line. 00:11:32.851 --> 00:11:35.484 We won't publish any journals, we won't publish any trials, 00:11:35.484 --> 00:11:38.166 unless they've been registered before they began. 00:11:38.166 --> 00:11:41.697 But they didn't hold the line. In 2008, a study was conducted 00:11:41.697 --> 00:11:44.712 which showed that half of all of trials published by journals 00:11:44.712 --> 00:11:47.379 edited by members of the ICMJE 00:11:47.379 --> 00:11:52.192 weren't properly registered, and a quarter of them weren't registered at all. 00:11:52.192 --> 00:11:54.993 And then finally, the FDA Amendment Act was passed 00:11:54.993 --> 00:11:57.342 a couple of years ago saying that everybody who conducts 00:11:57.342 --> 00:12:00.785 a trial must post the results of that trial within one year. 00:12:00.785 --> 00:12:04.881 And in the BMJ, in the first edition of January, 2012, 00:12:04.881 --> 00:12:07.585 you can see a study which looks to see if people kept 00:12:07.585 --> 00:12:11.304 to that ruling, and it turns out that only one in five 00:12:11.304 --> 00:12:14.168 have done so. NOTE Paragraph 00:12:14.168 --> 00:12:17.451 This is a disaster. 00:12:17.451 --> 00:12:21.015 We cannot know the true effects of the medicines 00:12:21.015 --> 00:12:24.231 that we prescribe if we do not have access 00:12:24.231 --> 00:12:27.411 to all of the information. NOTE Paragraph 00:12:27.411 --> 00:12:31.370 And this is not a difficult problem to fix. 00:12:31.370 --> 00:12:36.498 We need to force people to publish all trials 00:12:36.498 --> 00:12:39.469 conducted in humans, including the older trials, 00:12:39.469 --> 00:12:43.414 because the FDA Amendment Act only asks that you publish the trials conducted after 2008, 00:12:43.414 --> 00:12:46.027 and I don't know what world it is in which we're only 00:12:46.027 --> 00:12:50.483 practicing medicine on the basis of trials that completed in the past two years. 00:12:50.483 --> 00:12:52.588 We need to publish all trials in humans, 00:12:52.588 --> 00:12:55.662 including the older trials, for all drugs in current use, 00:12:55.662 --> 00:12:58.578 and you need to tell everyone you know 00:12:58.578 --> 00:13:02.020 that this is a problem and that it has not been fixed. 00:13:02.020 --> 00:13:04.971 Thank you very much. (Applause) 00:13:04.971 --> 00:13:08.244 (Applause)