WEBVTT

00:00:06.664 --> 00:00:11.023
In 2011, a team of physicists reported
a startling discovery:

00:00:11.023 --> 00:00:13.544
neutrinos traveled faster 
than the speed of light

00:00:13.544 --> 00:00:15.885
by 60 billionths of a second

00:00:15.885 --> 00:00:20.574
in their 730 kilometer trip from Geneva
to a detector in Italy.

00:00:20.574 --> 00:00:25.316
Despite six months of double checking,
the bizarre discovery refused to yield.

00:00:25.316 --> 00:00:27.804
But rather than celebrating 
a physics revolution,

00:00:27.804 --> 00:00:30.075
the researchers published a cautious paper

00:00:30.075 --> 00:00:35.103
arguing for continued research in an
effort to explain the observed anomaly.

00:00:35.103 --> 00:00:41.274
In time, the error was tracked to a single
incorrectly connected fiber optic cable.

00:00:41.274 --> 00:00:45.745
This example reminds us that real
science is more than static textbooks.

00:00:45.745 --> 00:00:49.646
Instead, researchers around the world
are continuously publishing

00:00:49.646 --> 00:00:51.265
their latest discoveries

00:00:51.265 --> 00:00:54.965
with each paper adding 
to the scientific conversation.

00:00:54.965 --> 00:00:57.425
Published studies 
can motivate future research,

00:00:57.425 --> 00:00:59.223
inspire new products,

00:00:59.223 --> 00:01:01.335
and inform government policy.

00:01:01.335 --> 00:01:05.144
So it's important that we have confidence
in the published results.

00:01:05.144 --> 00:01:06.726
If their conclusions are wrong,

00:01:06.726 --> 00:01:07.955
we risk time,

00:01:07.955 --> 00:01:09.115
resources,

00:01:09.115 --> 00:01:12.285
and even our health in the pursuit
of false leads.

00:01:12.285 --> 00:01:13.914
When findings are significant,

00:01:13.914 --> 00:01:17.005
they are frequently double-checked
by other researchers,

00:01:17.005 --> 00:01:18.996
either by reanalyzing the data

00:01:18.996 --> 00:01:21.886
or by redoing the entire experiment.

00:01:21.886 --> 00:01:25.496
For example, it took repeated 
investigation of the CERN data

00:01:25.496 --> 00:01:28.766
before the timing error was tracked down.

00:01:28.766 --> 00:01:33.206
Unfortunately, there are currently neither
the resources nor professional incentives

00:01:33.206 --> 00:01:38.617
to double check the more than 1 million
scientific papers published annually.

00:01:38.617 --> 00:01:42.725
Even when papers are challenged,
the results are not reassuring.

00:01:42.725 --> 00:01:46.297
Recent studies that examined dozens
of published pharmaceutical papers

00:01:46.297 --> 00:01:50.807
managed to replicate the results of
less than 25% of them.

00:01:50.807 --> 00:01:54.587
And similar results have been found
in other scientific disciplines.

00:01:54.587 --> 00:01:58.316
There are a variety of sources 
for irreproducible results.

00:01:58.316 --> 00:02:03.597
Errors could hide in their original
design, execution, or analysis of the data.

00:02:03.597 --> 00:02:04.807
Unknown factors,

00:02:04.807 --> 00:02:08.237
such as patients' undisclosed condition
in a medical study,

00:02:08.237 --> 00:02:11.909
can produce results that are 
not repeatable in new test subjects.

00:02:11.909 --> 00:02:15.887
And sometimes, the second research group
can't reproduce the original results

00:02:15.887 --> 00:02:20.368
simply because they don't know
exactly what the original group did.

00:02:20.368 --> 00:02:23.730
However, some problems might stem
from systematic decisions

00:02:23.730 --> 00:02:26.018
in how we do science.

00:02:26.018 --> 00:02:26.918
Researchers,

00:02:26.918 --> 00:02:28.548
the institutions that employ them,

00:02:28.548 --> 00:02:31.258
and the scientific journals 
that publish findings

00:02:31.258 --> 00:02:34.688
are expected to produce 
big results frequently.

00:02:34.688 --> 00:02:37.038
Important papers can advance careers,

00:02:37.038 --> 00:02:38.964
generate media interest,

00:02:38.964 --> 00:02:41.098
and secure essential funding,

00:02:41.098 --> 00:02:45.478
so there's slim motivation for researchers
to challenge their own exciting results.

00:02:45.478 --> 00:02:47.644
In addition, little incentive exists

00:02:47.644 --> 00:02:51.608
to publish results unsupportive
of the expected hypothesis.

00:02:51.608 --> 00:02:55.008
That results in a deluge of agreement
between what was expected

00:02:55.008 --> 00:02:56.615
and what was found.

00:02:56.615 --> 00:03:00.069
In rare occasions, this can even lead
to deliberate fabrication,

00:03:00.069 --> 00:03:04.779
such as in 2013, when a researcher
spiked rabbit blood with human blood

00:03:04.779 --> 00:03:08.888
to give false evidence that 
his HIV vaccine was working.

00:03:08.888 --> 00:03:10.889
The publish or perish mindset

00:03:10.889 --> 00:03:15.410
can also compromise academic journals'
traditional peer-review processes

00:03:15.410 --> 00:03:16.722
which are safety checks

00:03:16.722 --> 00:03:20.459
where experts examine submitted papers
for potential shortcomings.

00:03:20.459 --> 00:03:21.540
The current system,

00:03:21.540 --> 00:03:24.199
which might involve only one
or two reviewers,

00:03:24.199 --> 00:03:26.149
can be woefully ineffective.

00:03:26.149 --> 00:03:28.579
That was demonstrated in a 1998 study

00:03:28.579 --> 00:03:32.649
where eight weaknesses were deliberately
inserted into papers,

00:03:32.649 --> 00:03:36.289
but only around 25% 
were caught upon review.

00:03:36.289 --> 00:03:40.841
Many scientists are working toward
improving reproducibility in their fields.

00:03:40.841 --> 00:03:42.940
There's a push to make researchers
raw data,

00:03:42.940 --> 00:03:44.560
experimental procedures,

00:03:44.560 --> 00:03:47.728
and analytical techniques 
more openly available

00:03:47.728 --> 00:03:50.560
in order to ease replication efforts.

00:03:50.560 --> 00:03:53.029
The peer review process can also
be strengthened

00:03:53.029 --> 00:03:57.101
to more efficiently weed out weak papers
prior to publication.

00:03:57.101 --> 00:04:00.020
And we could temper the pressure
to find big results

00:04:00.020 --> 00:04:04.191
by publishing more papers that fail
to confirm the original hypothesis,

00:04:04.191 --> 00:04:08.670
an event that happens far more than
current scientific literature suggests.

00:04:08.670 --> 00:04:12.311
Science always has, and always will,
encounter some false starts

00:04:12.311 --> 00:04:15.391
as part of the collective acquisition
of new knowledge.

00:04:15.391 --> 00:04:18.310
Finding ways to improve 
the reproducibility of our results

00:04:18.310 --> 00:04:21.601
can help us weed out those false starts
more effectively,

00:04:21.601 --> 00:04:24.672
keeping us moving steadily toward
exciting new discoveries.