Is there a reproducibility crisis in science? - Matt Anticole

View full lesson: http://ed.ted.com/lessons/is-there-a-reproducibility-crisis-in-science-matt-anticole

Published scientific studies can motivate research, inspire products, and inform policy. However, recent studies that examined dozens of published pharmaceutical papers managed to replicate the results of less than 25% of them — and similar results have been found in other scientific disciplines. How do we combat this crisis of scientific irreproducibility? Matt Anticole investigates.

Lesson by Matt Anticole, animation by Brett Underhill.

In 2011, a team of physicists reported
a startling discovery:

neutrinos traveled faster
than the speed of light

by 60 billionths of a second

in their 730 kilometer trip from Geneva
to a detector in Italy.

Despite six months of double checking,
the bizarre discovery refused to yield.

But rather than celebrating
a physics revolution,

the researchers published a cautious paper

arguing for continued research in an
effort to explain the observed anomaly.

In time, the error was tracked to a single
incorrectly connected fiber optic cable.

This example reminds us that real
science is more than static textbooks.

Instead, researchers around the world
are continuously publishing

their latest discoveries

with each paper adding
to the scientific conversation.

Published studies
can motivate future research,

inspire new products,

and inform government policy.

So it's important that we have confidence
in the published results.

If their conclusions are wrong,

we risk time,

resources,

and even our health in the pursuit
of false leads.

When findings are significant,

they are frequently double-checked
by other researchers,

either by reanalyzing the data

or by redoing the entire experiment.

For example, it took repeated
investigation of the CERN data

before the timing error was tracked down.

Unfortunately, there are currently neither
the resources nor professional incentives

to double check the more than 1 million
scientific papers published annually.

Even when papers are challenged,
the results are not reassuring.

Recent studies that examined dozens
of published pharmaceutical papers

managed to replicate the results of
less than 25% of them.

And similar results have been found
in other scientific disciplines.

There are a variety of sources
for irreproducible results.

Errors could hide in their original
design, execution, or analysis of the data.

Unknown factors,

such as patients' undisclosed condition
in a medical study,

can produce results that are
not repeatable in new test subjects.

And sometimes, the second research group
can't reproduce the original results

simply because they don't know
exactly what the original group did.

However, some problems might stem
from systematic decisions

in how we do science.

Researchers,

the institutions that employ them,

and the scientific journals
that publish findings

are expected to produce
big results frequently.

Important papers can advance careers,

generate media interest,

and secure essential funding,

so there's slim motivation for researchers
to challenge their own exciting results.

In addition, little incentive exists

to publish results unsupportive
of the expected hypothesis.

That results in a deluge of agreement
between what was expected

and what was found.

In rare occasions, this can even lead
to deliberate fabrication,

such as in 2013, when a researcher
spiked rabbit blood with human blood

to give false evidence that
his HIV vaccine was working.

The publish or perish mindset

can also compromise academic journals'
traditional peer-review processes

which are safety checks

where experts examine submitted papers
for potential shortcomings.

The current system,

which might involve only one
or two reviewers,

can be woefully ineffective.

That was demonstrated in a 1998 study

where eight weaknesses were deliberately
inserted into papers,

but only around 25%
were caught upon review.

Many scientists are working toward
improving reproducibility in their fields.

There's a push to make researchers
raw data,

experimental procedures,

and analytical techniques
more openly available

in order to ease replication efforts.

The peer review process can also
be strengthened

to more efficiently weed out weak papers
prior to publication.

And we could temper the pressure
to find big results

by publishing more papers that fail
to confirm the original hypothesis,

an event that happens far more than
current scientific literature suggests.

Science always has, and always will,
encounter some false starts

as part of the collective acquisition
of new knowledge.

Finding ways to improve
the reproducibility of our results

can help us weed out those false starts
more effectively,

keeping us moving steadily toward
exciting new discoveries.