1
00:00:06,664 --> 00:00:11,023
In 2011, a team of physicists reported
a startling discovery:

2
00:00:11,023 --> 00:00:13,544
neutrinos traveled faster 
than the speed of light

3
00:00:13,544 --> 00:00:15,885
by 60 billionths of a second

4
00:00:15,885 --> 00:00:20,574
in their 730 kilometer trip from Geneva
to a detector in Italy.

5
00:00:20,574 --> 00:00:25,316
Despite six months of double checking,
the bizarre discovery refused to yield.

6
00:00:25,316 --> 00:00:27,804
But rather than celebrating 
a physics revolution,

7
00:00:27,804 --> 00:00:30,075
the researchers published a cautious paper

8
00:00:30,075 --> 00:00:35,103
arguing for continued research in an
effort to explain the observed anomaly.

9
00:00:35,103 --> 00:00:41,274
In time, the error was tracked to a single
incorrectly connected fiber optic cable.

10
00:00:41,274 --> 00:00:45,745
This example reminds us that real
science is more than static textbooks.

11
00:00:45,745 --> 00:00:49,646
Instead, researchers around the world
are continuously publishing

12
00:00:49,646 --> 00:00:51,265
their latest discoveries

13
00:00:51,265 --> 00:00:54,965
with each paper adding 
to the scientific conversation.

14
00:00:54,965 --> 00:00:57,425
Published studies 
can motivate future research,

15
00:00:57,425 --> 00:00:59,223
inspire new products,

16
00:00:59,223 --> 00:01:01,335
and inform government policy.

17
00:01:01,335 --> 00:01:05,144
So it's important that we have confidence
in the published results.

18
00:01:05,144 --> 00:01:06,726
If their conclusions are wrong,

19
00:01:06,726 --> 00:01:07,955
we risk time,

20
00:01:07,955 --> 00:01:09,115
resources,

21
00:01:09,115 --> 00:01:12,285
and even our health in the pursuit
of false leads.

22
00:01:12,285 --> 00:01:13,914
When findings are significant,

23
00:01:13,914 --> 00:01:17,005
they are frequently double-checked
by other researchers,

24
00:01:17,005 --> 00:01:18,996
either by reanalyzing the data

25
00:01:18,996 --> 00:01:21,886
or by redoing the entire experiment.

26
00:01:21,886 --> 00:01:25,496
For example, it took repeated 
investigation of the CERN data

27
00:01:25,496 --> 00:01:28,766
before the timing error was tracked down.

28
00:01:28,766 --> 00:01:33,206
Unfortunately, there are currently neither
the resources nor professional incentives

29
00:01:33,206 --> 00:01:38,617
to double check the more than 1 million
scientific papers published annually.

30
00:01:38,617 --> 00:01:42,725
Even when papers are challenged,
the results are not reassuring.

31
00:01:42,725 --> 00:01:46,297
Recent studies that examined dozens
of published pharmaceutical papers

32
00:01:46,297 --> 00:01:50,807
managed to replicate the results of
less than 25% of them.

33
00:01:50,807 --> 00:01:54,587
And similar results have been found
in other scientific disciplines.

34
00:01:54,587 --> 00:01:58,316
There are a variety of sources 
for irreproducible results.

35
00:01:58,316 --> 00:02:03,597
Errors could hide in their original
design, execution, or analysis of the data.

36
00:02:03,597 --> 00:02:04,807
Unknown factors,

37
00:02:04,807 --> 00:02:08,237
such as patients' undisclosed condition
in a medical study,

38
00:02:08,237 --> 00:02:11,909
can produce results that are 
not repeatable in new test subjects.

39
00:02:11,909 --> 00:02:15,887
And sometimes, the second research group
can't reproduce the original results

40
00:02:15,887 --> 00:02:20,368
simply because they don't know
exactly what the original group did.

41
00:02:20,368 --> 00:02:23,730
However, some problems might stem
from systematic decisions

42
00:02:23,730 --> 00:02:26,018
in how we do science.

43
00:02:26,018 --> 00:02:26,918
Researchers,

44
00:02:26,918 --> 00:02:28,548
the institutions that employ them,

45
00:02:28,548 --> 00:02:31,258
and the scientific journals 
that publish findings

46
00:02:31,258 --> 00:02:34,688
are expected to produce 
big results frequently.

47
00:02:34,688 --> 00:02:37,038
Important papers can advance careers,

48
00:02:37,038 --> 00:02:38,964
generate media interest,

49
00:02:38,964 --> 00:02:41,098
and secure essential funding,

50
00:02:41,098 --> 00:02:45,478
so there's slim motivation for researchers
to challenge their own exciting results.

51
00:02:45,478 --> 00:02:47,644
In addition, little incentive exists

52
00:02:47,644 --> 00:02:51,608
to publish results unsupportive
of the expected hypothesis.

53
00:02:51,608 --> 00:02:55,008
That results in a deluge of agreement
between what was expected

54
00:02:55,008 --> 00:02:56,615
and what was found.

55
00:02:56,615 --> 00:03:00,069
In rare occasions, this can even lead
to deliberate fabrication,

56
00:03:00,069 --> 00:03:04,779
such as in 2013, when a researcher
spiked rabbit blood with human blood

57
00:03:04,779 --> 00:03:08,888
to give false evidence that 
his HIV vaccine was working.

58
00:03:08,888 --> 00:03:10,889
The publish or perish mindset

59
00:03:10,889 --> 00:03:15,410
can also compromise academic journals'
traditional peer-review processes

60
00:03:15,410 --> 00:03:16,722
which are safety checks

61
00:03:16,722 --> 00:03:20,459
where experts examine submitted papers
for potential shortcomings.

62
00:03:20,459 --> 00:03:21,540
The current system,

63
00:03:21,540 --> 00:03:24,199
which might involve only one
or two reviewers,

64
00:03:24,199 --> 00:03:26,149
can be woefully ineffective.

65
00:03:26,149 --> 00:03:28,579
That was demonstrated in a 1998 study

66
00:03:28,579 --> 00:03:32,649
where eight weaknesses were deliberately
inserted into papers,

67
00:03:32,649 --> 00:03:36,289
but only around 25% 
were caught upon review.

68
00:03:36,289 --> 00:03:40,841
Many scientists are working toward
improving reproducibility in their fields.

69
00:03:40,841 --> 00:03:42,940
There's a push to make researchers
raw data,

70
00:03:42,940 --> 00:03:44,560
experimental procedures,

71
00:03:44,560 --> 00:03:47,728
and analytical techniques 
more openly available

72
00:03:47,728 --> 00:03:50,560
in order to ease replication efforts.

73
00:03:50,560 --> 00:03:53,029
The peer review process can also
be strengthened

74
00:03:53,029 --> 00:03:57,101
to more efficiently weed out weak papers
prior to publication.

75
00:03:57,101 --> 00:04:00,020
And we could temper the pressure
to find big results

76
00:04:00,020 --> 00:04:04,191
by publishing more papers that fail
to confirm the original hypothesis,

77
00:04:04,191 --> 00:04:08,670
an event that happens far more than
current scientific literature suggests.

78
00:04:08,670 --> 00:04:12,311
Science always has, and always will,
encounter some false starts

79
00:04:12,311 --> 00:04:15,391
as part of the collective acquisition
of new knowledge.

80
00:04:15,391 --> 00:04:18,310
Finding ways to improve 
the reproducibility of our results

81
00:04:18,310 --> 00:04:21,601
can help us weed out those false starts
more effectively,

82
00:04:21,601 --> 00:04:24,672
keeping us moving steadily toward
exciting new discoveries.