Step 3 - Methodology and Protocols

Last updated on 2024-12-10 | Edit this page

Estimated time: 15 minutes

One of the top reasons that reproducibility studies failed was the lack of detail in methodology and protocols.

Literature Review

Have you kept details of your literature review?

While you may not be publishing all the details of your literature review, it can be helpful for your future self to retain these details. You may want to use part of this data for your next study, and need to remember why you discounted a paper, or if you felt there was bias unaccounted for in a study.

  • What databases did you search?

  • What search terms did you use?

  • What filters did you use?

  • Why did you include or dismiss certain papers?

Preregistration

When you preregister your research, you are specifying your research plan in advance of your study and submitting it to a registry.

Preregistration separates hypothesis-generating (exploratory) from hypothesis-testing (confirmatory) research. Both are important. But the same data cannot be used to generate and test a hypothesis, which can happen unintentionally and reduce the credibility of your results. Addressing this problem through planning improves the quality and transparency of your research. This helps you clearly report your study and helps others who may wish to build on it.

Prominent registries include:

Registered Reports

You may want to consider a Registered Report

The Center for Open Science provides the following definition:

“Registered Reports is a publishing format that emphasizes the importance of the research question and the quality of methodology by conducting peer review prior to data collection. High quality protocols are then provisionally accepted for publication if the authors follow through with the registered methodology.

This format is designed to reward best practices in adhering to the hypothetico-deductive model of the scientific method. It eliminates a variety of questionable research practices, including low statistical power, selective reporting of results, and publication bias, while allowing complete flexibility to report serendipitous findings. ”

Here is a list of participating journals with details on their options.

Transparent Methodologies and Protocols

Clarity in your methodology is important, and can offer an insight into how you have reduced bias in your research. Has that methodology come from a different area of the discipline? If so, why was it adopted?

Have you described your protocols in depth? Part of being reproducible is explaining your steps in such depth that someone can replicate your work.

Consider what detail to include - In addition to detailed procedures, equipment, and instruments, protocols should also contain study objectives, reasoning for experimental design, reasoning for chosen sample sizes, safety precautions, and how results were calculated and reported, including statistical analysis and any rules for predefining and documenting excluded data to avoid bias.

Have you explained why you chose that protocol, took those steps?

This really did come up time and time again when talking to researchers on how they established trust in a paper. This lesson is a good opportunity to point out where in your institute researchers can get assistance with methodology and statistics, or relevant training.

Publishing your Protocol

Benefits to publishing your protocols:

  • Builds trust in your work

  • Enables discovery

  • Expands your publication records

  • Offers an early opportunity for peer review, with an opportunity for feedback and improvements

  • It creates an early record of your novel methodologies, software, and/or innovations

Where can you publish your protocols?

  • Protocols.io is dedicated to protocol publication.

  • PLOS PLOS is a nonprofit, Open Access publisher for science and medicine.

  • Open Science Framework (OSF) - A free open source project management tool. You can register all types of review protocols including scoping reviews.

  • Cochrane Library - Includes protocols of Cochrane Reviews for the medicine and other healthcare specialties that are planned or in progress.

  • Many journals will publish protocols - check the journals in your field. Instructions for authors will often have guidelines for the level of detail expected in the protocols they publish or where the protocol should be published.

See methods & protocols - ReproducibiliTeach for the differences between protocol journals and protocol repositories (at 23:33).

What is your next step?

References

The Turing Way Community. (updated 2023) Methods and Protocols . Github.com Retrieved April 11, 2024, from https://book.the-turing-way.org/reproducible-research/rdm/rdm-methods.html?highlight=protocol#open-methods-protocols licenced under CC-BY 4.0 licence

Center for Open Science (updated 2024) Home. COS Retrieved April 11, 2024 from https://www.cos.io/ licenced under CC-BY 4.0 licence

Center for Open Science (updated 2024) Registered Reports. COS Retrieved April 11, 2024 from https://www.cos.io/initiatives/registered-reports licenced under CC-BY 4.0 licence

In this lesson, we have learnt:

  • What details to keep about our literature reviews

  • About Preregistrations and where to submit

  • About Registered Reports formats and when they are appropriate

  • What details to include about your methodologies and protocols

  • Why and where to publish your protocols

We build trust in our knowledge by:

  • By being open on our plans in our registered reports and preregistrations

  • Publishing our protocols

  • Being detailed in our methodologies

We retain knowledge using:

  • Detailed recording of our literature review, so we can understand why those papers were reviewed

  • By saving our protocols for use later

We build business continuity by:

  • By saving our protocols for use later

  • By sharing our registered reports and preregistrations

  • By sharing with the team why those papers were reviewed in your literature review