Open Science

Outline

• Definitions
• Rationale
• Summary
• FAQ
• Food for thought
• Practical exercises

This image was created by Scriberia for The Turing Way community and is used under a CC-BY licence

• Reproducible research - A result is reproducible when the same analysis steps performed on the same dataset consistently produces the same answer.
• Replicable research - A result is replicable when the same analysis performed on different datasets produces qualitatively similar answers.
• Robust research - A result is robust when the same dataset is subjected to different analysis workflows to answer the same research question and a qualitatively similar or identical answer is produced.
• Generalisable research - A result is generalisable when it is not dependent on a particular dataset nor a particular version of the analysis pipeline.

• Computational reproducibility: When detailed information is provided about code, software, hardware and implementation details.
• Empirical reproducibility: When detailed information is provided about non-computational empirical scientific experiments and observations. In practice, this is enabled by making the data and details of how it was collected freely available.
• Statistical reproducibility: When detailed information is provided, for example, about the choice of statistical tests, model parameters, and threshold values. This mostly relates to pre-registration of study design to prevent p-value hacking and other manipulations.

A problem with any one of these three types of reproducibility [...] can be enough to derail the process of establishing scientific facts. Each type calls for different remedies, from improving existing communication standards and reporting (empirical reproducibility) to making computational environments available for replication purposes (computational reproducibility) to the statistical assessment of repeated results for validation purposes (statistical reproducibility).

Stodden 2014 [Online; accessed 20 May 2021]

The Open Science Training Handbook

An article (about computational result) is advertising, not scholarship. The actual scholarship is the full software environment, code and data, that produced the result. Buckheit & Donoho 1995

Marwick et al. 2017

This image was created by Scriberia for The Turing Way community and is used under a CC-BY licence

The Turing Way

-> F. Markowetz - 5 selfish reasons to work reproducibly [Online; accessed 20 May 2021]

The Open Science Training Handbook

How computers broke science – and what we can do to fix it [Online; accessed 20 May 2021]

"Overall the report introduces the concept of reproducibility as a continuum of practices".

Peng

"It is posited that the reproducibility of results has value both as a mechanism to ensure good science based on truthful claims, and as a driver of further discovery and innovation." Reproducibility of scientific results in the EU

A likely culprit for this disconnect [embracing, but not practicing open science principles] is an academic reward system that does not sufficiently incentivize open practices. Nosek et al. 2015

"Journal policies can be evaluated based on the degree to which they comply with the TOP Guidelines. This TOP Factor is a metric that reports the steps that a journal is taking to implement open science practices, practices that are based on the core principles of the scientific community. It is an alternative way to assess journal qualities, and is an improvement over traditional metrics that measure mean citation rates. The TOP Factor is transparent [...] and will be responsive to community feedback." TOP Guidelines

Read more about TOP Factor here.

Screenshot from 2021-05-19

Reward system

This image was created by Scriberia for The Turing Way community and is used under a CC-BY licence

Education

This image was created by Scriberia for The Turing Way community and is used under a CC-BY licence

Wilson et al. 2014

Wilson et al. 2014

• Definitions of reproducible, replicable, robust and generalisable studies
• Definitions of empirical, computational and statistical reproducibility
• Reproducibility at the core of the scientific method
• Issues with research reproducibility
• Ways to promoting an open research culture
• How to make reproducible research
• What is a research compendium

Everything is in the paper; anyone can reproduce this from there!

This is one of the most common misconceptions. Even having an extremely detailed description of the methods and workflows employed to reach the final result will not be sufficient in most cases to reproduce it. This can be due to several aspects, including different computational environments, differences in the software versions, implicit biases that were not clearly stated, etc.

I don’t have the time to learn and establish a reproducible workflow.

In addition to a significant number of freely available online services that can be combined and facilitate the setting up of an entire workflow, spending the time and effort to put this together will increase both the scientific validity of the final results as well as minimize the time of re-running or extending it in further studies.

* The Open Science Training Handbook

The basic building blocks of archaeological knowledge are non-replicable observations, this does not mean we are immune to the reproducibility crisis.

But archaeological research is not just a list of sites and artefacts. In order to extract understanding from our irreproducible corpus of material, we subject it to an extraordinary range of analytical methods, most of which are replicable. This is where archaeologists have been most active in promoting reproducibility, as part of a larger trend towards open science.

We can’t rerun the history that produced that material, or even the process through which we obtained it. So how can we obtain reproducible results from non-replicable observations?

J. Roe. 2016. Does archaeology have a reproducibility crisis? [Online; accessed 20 May 2021]

-> Consider how the concepts of empirical and computational reproducibility apply to Archaeology.

Further readings

• Roebroeks et al. 2017. Uneven Data Quality and the Earliest Occupation of Europe—the Case of Untermassfeld (Germany). Journal of Paleolithic Archaeology 1, 5-31

Reading list

• Marwick, B. Computational Reproducibility in Archaeological Research: Basic Principles and a Case Study of Their Implementation. J Archaeol Method Theory 24, 424–450 (2017). https://doi.org/10.1007/s10816-015-9272-9

References

• Carney & Davies 2020
• Clarkson et al. 2017
  • Addressing Reproducibility in Archaeology: Our Three-Pronged Approach
• Nature collection: Challenges in irreproducible research
• Nosek et al. 2015
• Wilson et al. 2014

• F. Markowetz - 5 selfish reasons to work reproducibly
• K. A. Baggerly - The Importance of Reproducible Research in High-Throughput Biology

Outline

• Create a research compendium
• Publish a research compendium (Zenodo / OSF)