Reproducibility
Demonstrate that research results are reliable and not due to bias or chance
What is reproducibility?
Reproducibility is the degree to which other researchers can achieve the same results using the same dataset and analysis as the original researchers. Research is reproducible when other researchers can achieve the results again with high reliability.
Reproducible research requires a detailed description of the methods used in the study, as well as all the underlying data and code, to be openly available. Reproducibility is essential to good science. It demonstrates that research results are objective and reliable and not due to bias or chance.
download our FREE guide
How to make your research more reproducible
Fill in the form below to uncover actionable tips you can implement today to make your research more reproducible.
Reproducibility
Reproducibility vs. repeatability
Reproducibility, repeatability, and replicability are related terms often used interchangeably. While various research communities have embraced these concepts, the terminology between disciplines differs.
At F1000, we define reproducible research as research that can be executed by different researchers using the same data to achieve the same results. This definition aligns with the Claerbout and Karrenbach definition first outlined in 1992. One study found the Claerbout and Karrebach definition is the most used across disciplines.
Differentiating between repeatability, reproducibility, and replicability relies on two critical factors:
- Who is conducting the research?
- Are they using the original dataset or new data?
Why is reproducibility so important?
Benefits of reproducibility for the research community
Science is dependent on the sharing of information. The growth of open research and open data has led to increased calls for researchers to improve the reproducibility of their research. Sharing data, code, and detailed research methods accelerate scientific discovery by making more elements of research available to all researchers. Aside from faster progress, reproducibility offers numerous other benefits for the research community.
Reproducible research strengthens scientific evidence and the reliability of results
When researchers can reproduce a study, they lend credibility to the findings of the original researchers. The results of the original research can be deemed reliable, and there is more evidence supporting the conclusions. This makes it more likely that research findings will be used to make a real-world impact by informing policy or practice.
Reproducible research increases trust in science
When scientists cannot reproduce results, other researchers and the public lose trust in the scientific process. With so much public funding going towards research and so much research informing public policy, research must be reproducible so the public can trust science.
Reproducible research enables efficiency in research
How reproducibility increases efficiency is twofold. Firstly, the more reproducible research is, the greater the odds that the research, or at least parts of it, can be reused by other researchers in the future. Additionally, by publishing negative results, researchers can help other researchers avoid wasting time on analyses that will not return the expected results.
Reproducible research helps to minimize misinformation
Reviewers can spot mistakes more easily when researchers embrace transparency in the research process, provide detailed documentation of their methods and analyses, and share their research materials. This helps to avoid misinformation that can limit the replicability of research and, instead, leads to more accurate research papers.
Benefits of reproducibility for researchers
It’s clear that the broader research community benefits from reproducibility. Yet, much of the burden for making research reproducible falls to the original researcher. How can researchers themselves benefit from making their research reproducible?
Reproducible research has the potential for a greater impact
Researchers who produce reproducible research share their underlying data and methods. Sharing detailed research data is associated with higher citation rates as other researchers use and credit the data in their projects.
Reproducible research facilitates in-depth peer review
Reproducible research can result in higher-quality, faster peer review as reviewers have access to the data and analytical processes described in the manuscript. This increases the probability that errors are caught during the peer review process and reduces back-and-forth between authors and reviewers.
Reproducible research enables iterative science
Science is an iterative process, with many tasks repeated time and again. Reproducible research enables researchers to reuse previous research materials to execute similar research tasks more efficiently in new projects.
Reproducible research enables collaboration and reuse
Reproducible research opens the door to new partnerships to develop existing projects further. It also leads to greater reuse of research. Designing reproducible workflows and sharing all research outputs openly allows others to develop a deeper understanding of the work and build upon it.
What is the reproducibility crisis?
The reproducibility crisis refers to a current state in research in which the results of many studies are difficult or impossible to reproduce. The reproducibility crisis raises important questions about research practice and the validity of research findings. As such, the reproducibility crisis has been a prominent topic of conversation in recent years, particularly within psychology and the life sciences.
Significantly, one study revealed that in biology alone, over 70% of researchers could not reproduce the findings of other scientists, and approximately 60% of researchers could not reproduce their own findings.
What are the barriers to reproducible research?
So, what has caused the reproducibility crisis? Despite the apparent benefits of reproducible research, researchers face various challenges when making it reproducible. These barriers include:
Lack of recognition and incentives
The novelty bias in academic publishing and university hiring and promotion criteria hinders reproducibility. Researchers are rewarded for publishing novel findings in high-impact journals and not for publishing null or confirmatory results. This makes it challenging to encourage researchers to go to the extra effort of reproducing research and leads to an under-reporting of studies that produce seemingly insignificant results.
Unwillingness to share methods, data, and research materials
Some researchers resist sharing their data and research materials, which is essential to improving reproducibility in research. Researchers may avoid sharing to reuse the data to perform new analyses and research without fear of scooping by other researchers. However, by publishing data in a repository, researchers can receive credit when others use their data and establish an embargo period for reuse.
Reproducibility requires additional time and skills
Making research reproducible is sometimes easier said than done. Working reproducibly requires a greater time investment and skills that aren’t always taught at university. To work in a reproducible manner, researchers may need to learn how to use new software and tools, develop data and software engineering skills, and become expert project managers.
Poor research practices and study design
Poor research practices can also cause irreproducibility, such as unclear methodologies, inaccurate statistical or data analysis, and insufficient efforts to minimize biases. Poor study design also makes it less likely that research will be reproducible.
Expand your open research knowledge
Peer review is not just quality control, it is the backbone of modern scientific and academic publishing, ensuring the validity and credibility of research articles. While it may seem like…
The peer review process is a fundamental component of scholarly publishing, ensuring the quality and credibility of academic research. After submitting your manuscript to a publishing venue, it undergoes rigorous…
Peer review is an integral part of scholarly communication and academic publishing. A key player in this process is the peer reviewer, who is typically a recognized expert in the…