'Robot scientist' Eve finds that less than one third of scientific
results are reproducible

Date:
April 6, 2022
Source:
University of Cambridge
Summary:
Researchers have used a combination of automated text analysis
and the 'robot scientist' Eve to semi-automate the process of
reproducing research results. The problem of lack of reproducibility
is one of the biggest crises facing modern science.



FULL STORY ========================================================================== Researchers have used a combination of automated text analysis and
the 'robot scientist' Eve to semi-automate the process of reproducing
research results.

The problem of lack of reproducibility is one of the biggest crises
facing modern science.


==========================================================================
The researchers, led by the University of Cambridge, analysed more than
12,000 research papers on breast cancer cell biology. After narrowing the
set down to 74 papers of high scientific interest, less than one third --
22 papers -- were found to be reproducible. In two cases, Eve was able
to make serendipitous discoveries.

The results, reported in the journal Royal Society Interface, demonstrate
that it is possible to use robotics and artificial intelligence to help
address the reproducibility crisis.

A successful experiment is one where another scientist, in a different laboratory under similar conditions, can achieve the same result. But
more than 70% of researchers have tried and failed to reproduce another scientist's experiments, and more than half have failed to reproduce
some of their own experiments: this is the reproducibility crisis.

"Good science relies on results being reproducible: otherwise, the
results are essentially meaningless," said Professor Ross King from
Cambridge's Department of Chemical Engineering and Biotechnology,
who led the research. "This is particularly critical in biomedicine:
if I'm a patient and I read about a promising new potential treatment,
but the results aren't reproducible, how am I supposed to know what to
believe? The result could be people losing trust in science." Several
years ago, King developed the robot scientist Eve, a computer/robotic
system that uses techniques from artificial intelligence (AI) to carry
out scientific experiments.



==========================================================================
"One of the big advantages of using machines to do science is they're
more precise and record details more exactly than a human can," said
King. "This makes them well-suited to the job of attempting to reproduce scientific results." As part of a project funded by DARPA, King and his colleagues from the UK, US and Sweden designed an experiment that uses
a combination of AI and robotics to help address the reproducibility
crisis, by getting computers to read scientific papers and understand
them, and getting Eve to attempt to reproduce the experiments.

For the current paper, the team focused on cancer research. "The cancer literature is enormous, but no one ever does the same thing twice,
making reproducibility a huge issue," said King. "Given the vast sums of
money spent on cancer research, and the sheer number of people affected
by cancer worldwide, it's an area where we urgently need to improve reproducibility." From an initial set of more than 12,000 published
scientific papers, the researchers used automated text mining techniques
to extract statements related to a change in gene expression in response
to drug treatment in breast cancer.

From this set, 74 papers were selected.

Two different human teams used Eve and two breast cancer cell lines
and attempted to reproduce the 74 results. Statistically significant
evidence for repeatability was found for 43 papers, meaning that the
results were replicable under identical conditions; and significant
evidence for reproducibility or robustness was found in 22 papers,
meaning the results were replicable by different scientists under similar conditions. In two cases, the automation made serendipitous discoveries.

While only 22 out of 74 papers were found to be reproducible in this experiment, the researchers say that this does not mean that the remaining papers are not scientifically reproducible or robust. "There are lots
of reasons why a particular result may not be reproducible in another
lab," said King. "Cell lines can sometimes change their behaviour
in different labs under different conditions, for instance. The most
important difference we found was that it matters who does the experiment, because every person is different." King says that this work shows that automated and semi-automated techniques could be an important tool to
help address the reproducibility crisis, and that reproducibility should
become a standard part of the scientific process.

"It's quite shocking how big of an issue reproducibility is in science,
and it's going to need a complete overhaul in the way that a lot of
science is done," said King. "We think that machines have a key role
to play in helping to fix it." The research was also funded by the
Engineering and Physical Sciences Research Council (EPSRC), part of UK
Research and Innovation (UKRI).


========================================================================== Story Source: Materials provided by University_of_Cambridge. The original
text of this story is licensed under a Creative_Commons_License. Note:
Content may be edited for style and length.


========================================================================== Journal Reference:
1. Katherine Roper, A. Abdel-Rehim, Sonya Hubbard, Martin Carpenter,
Andrey
Rzhetsky, Larisa Soldatova, Ross D. King. Testing the
reproducibility and robustness of the cancer biology literature
by robot. Journal of The Royal Society Interface, 2022; 19 (189)
DOI: 10.1098/rsif.2021.0821 ==========================================================================

Link to news story: https://www.sciencedaily.com/releases/2022/04/220406101746.htm

--- up 5 weeks, 2 days, 10 hours, 50 minutes
* Origin: -=> Castle Rock BBS <=- Now Husky HPT Powered! (1:317/3)