Is there a reproducibility crisis in science? - Matt Anticole
-
0:07 - 0:11In 2011, a team of physicists reported
a startling discovery - -
0:11 - 0:14neutrinos traveled faster
than the speed of light -
0:14 - 0:16by 60 billionths of a second
-
0:16 - 0:21in their 730 kilometer trip from Geneva
to a detector in Italy. -
0:21 - 0:25Despite six months of double checking,
the bizarre discovery refused to yield. -
0:25 - 0:28But rather than celebrating
a physics revolution, -
0:28 - 0:30the researchers published a cautious paper
-
0:30 - 0:35arguing for continued research in an
effort to explain the observed anomaly. -
0:35 - 0:41In time, the error was tracked to a single
incorrectly connected fiber optic cable. -
0:41 - 0:46This example reminds us that real
science is more than static textbooks. -
0:46 - 0:50Instead, researchers around the world
are continuously publishing -
0:50 - 0:51their latest discoveries
-
0:51 - 0:55with each paper adding
to the scientific conversation. -
0:55 - 0:57Published studies
can motivate future research, -
0:57 - 0:59inspire new products,
-
0:59 - 1:01and inform government policy.
-
1:01 - 1:05So it's important that we have confidence
in the published results. -
1:05 - 1:07If their conclusions are wrong,
-
1:07 - 1:08we risk time,
-
1:08 - 1:09resources,
-
1:09 - 1:12and even our health in the pursuit
of false leads. -
1:12 - 1:14When findings are significant,
-
1:14 - 1:17they are frequently double-checked
by other researchers, -
1:17 - 1:19either by reanalyzing the data
-
1:19 - 1:22or by redoing the entire experiment.
-
1:22 - 1:25For example, it took repeated
investigation of the CERN data -
1:25 - 1:29before the timing era was tracked down.
-
1:29 - 1:33Unfortunately, there are currently neither
the resources nor professional incentives -
1:33 - 1:39to double-check the more than 1 million
scientific papers published annually. -
1:39 - 1:43Even when papers are challenged,
the results are not reassuring. -
1:43 - 1:46Recent studies that examined dozens
of published pharmaceutical papers -
1:46 - 1:51managed to replicate the results of
less than 25% of them. -
1:51 - 1:55And similar results have been found
in other scientific disciplines. -
1:55 - 1:58There are a variety of sources
for irreproducible results. -
1:58 - 2:04Errors could hide in their original
design, execution or analysis of the data. -
2:04 - 2:05Unknown factors,
-
2:05 - 2:08such as patients' undisclosed condition
in a medical study, -
2:08 - 2:12can produce results that are
not repeatable in new test subjects. -
2:12 - 2:16And sometimes, the second research group
can't reproduce the original results -
2:16 - 2:20simply because they don't know
exactly what the original group did. -
2:20 - 2:24However, some problems might stem
from systematic decisions -
2:24 - 2:26in how we do science.
-
2:26 - 2:27Researchers,
-
2:27 - 2:29the institutions that employ them,
-
2:29 - 2:31and the scientific journals
that publish findings -
2:31 - 2:35are expected to produce
big results frequently. -
2:35 - 2:37Important papers can advance careers,
-
2:37 - 2:39generate media interest,
-
2:39 - 2:41and secure essential funding,
-
2:41 - 2:45so there's slim motivation for researchers
to challenge their own exciting results. -
2:45 - 2:48In addition, little incentive exists
-
2:48 - 2:52to publish results unsupportive
of the expected hypothesis. -
2:52 - 2:55That results in a deluge of agreement
between what was expected -
2:55 - 2:57and what was found.
-
2:57 - 3:00In rare occasions, this can even lead
to deliberate fabrication, -
3:00 - 3:05such as in 2013, when a researcher
spiked rabbit blood with human blood -
3:05 - 3:09to give false evidence that
his HIV vaccine was working. -
3:09 - 3:11The publish or perish mindset
-
3:11 - 3:15can also compromise academic journals'
traditional peer-review processes -
3:15 - 3:17which are safety checks
-
3:17 - 3:20where experts examine submitted papers
for potential shortcomings. -
3:20 - 3:22The current system,
-
3:22 - 3:24which might involve only one
or two reviewers, -
3:24 - 3:26can be woefully ineffective.
-
3:26 - 3:29That was demonstrated in a 1998 study
-
3:29 - 3:33where eight weaknesses were deliberately
inserted into papers, -
3:33 - 3:36but only around 25%
were caught upon review. -
3:36 - 3:41Many scientists are working toward
improving reproducibility in their fields. -
3:41 - 3:43There's a push to make researchers
raw data, -
3:43 - 3:45experimental procedures,
-
3:45 - 3:48and analytical techniques
more openly available -
3:48 - 3:51in order to ease replication efforts.
-
3:51 - 3:53The peer review process can also
be strengthened -
3:53 - 3:57to more efficiently weed out weak papers
prior to publication. -
3:57 - 4:00And we could temper the pressure
to find big results -
4:00 - 4:04by publishing more papers that fail
to confirm the original hypothesis, -
4:04 - 4:09an even that happens far more than
current scientific literature suggests. -
4:09 - 4:12Science always has, and always will,
encounter some false starts -
4:12 - 4:15as part of the collective acquisition
of new knowledge. -
4:15 - 4:18Finding ways to improve
the reproducibility of our results -
4:18 - 4:22can help us weed out those false starts
more effectively, -
4:22 - 4:25keeping us moving steadily toward
exciting new discoveries.
- Title:
- Is there a reproducibility crisis in science? - Matt Anticole
- Description:
-
View full lesson: http://ed.ted.com/lessons/scientific-irreproducibility-matthew-anticole
Lesson by Matthew Anticole, animation by Brett Underhill
- Video Language:
- English
- Team:
- closed TED
- Project:
- TED-Ed
- Duration:
- 04:47
Jessica Ruby approved English subtitles for 1608 04 A Anticole Matthew ScientificIrreproducibility (correct version) | ||
Jessica Ruby accepted English subtitles for 1608 04 A Anticole Matthew ScientificIrreproducibility (correct version) | ||
Jessica Ruby edited English subtitles for 1608 04 A Anticole Matthew ScientificIrreproducibility (correct version) | ||
Jennifer Cody edited English subtitles for 1608 04 A Anticole Matthew ScientificIrreproducibility (correct version) |