Open Science

Outline

• Peer review
• Metrics
• Summary
• FAQ
• Food for thought

The act of publishing is a quid pro quo in which authors receive credit and acknowledgment in exchange for disclosure of their scientific findings.

Knowledge is built upon those findings and thus it is necessary to have in place a system to evaluate both research and researchers.

Both research and researcher are evaluated through two primary methods: peer review and metrics, the first qualitative and the latter quantitative.

Peer review

Peer review is used primarily to judge pieces of research. It is the formal quality assurance mechanism whereby scholarly manuscripts (e.g., journal articles, books, grant applications and conference papers) are made subject to the scrutiny of others, whose feedback and judgements are then used to improve works and make final decisions regarding selection (for publication, grant allocation or speaking time).

Metrics

Once they have passed peer review, research publications are then often the primary measure of a researcher's work ('Publish or Perish'). [...] However, assessing the quality of publications is difficult and subjective. [...] General assessment is often based on metrics such as the number of citations publications garner (h-index), or even the perceived level of prestige of the journal it was published in (quantified by the Journal Impact Factor).

The peer review system judges the validity, significance and originality of the work, rather than who has done it.

VoYS. Peer review. The nuts and bolts. A guide for early career researchers

VoYS. Peer review. The nuts and bolts. A guide for early career researchers

• "Referee systems are not as old as historical accounts often claim. Investigators of nature communicated their findings without scientific referees for centuries. Deciding whom and what to trust usually depended on personal knowledge among close-knit groups of researchers".
• "The first referee systems that we would recognize as such were set in place by English scientific societies in the early nineteenth century".
• "The notion [of referees as supreme scientific gatekeepers] emerged in around 1900. It was exactly then that some began to wonder whether referee systems might be fundamentally flawed. In this sense, peer review has always been broken".

Csiszar 2016

• Unreliability and inconsistency: Reliant upon the vagaries of human judgement, the objectivity, reliability, and consistency of peer review are subject to question.
• Delay and expense: The period from submission to publication at many journals can often exceed one year, with much of this time taken up by peer review.
• Lack of accountability and risks of subversion: The “black-box” nature of traditional peer review gives reviewers, editors and even authors a lot of power to potentially subvert the process.
• Social and publication biases: Peer reviewers may be subject to social biases on the grounds of gender, nationality, institutional affiliation, language and discipline
• Lack of incentives: Traditional peer review provides little in the way of incentives for reviewers, whose work is almost exclusively unpaid and whose anonymous contributions cannot be recognised and hence rewarded
• Wastefulness: Potentially valuable additional information (Reviewer comments & disagreements, Authors replies) is wasted.

Ross-Hellauer 2017

The Open Science Training Handbook

OPEN PEER REVIEW

At its most basic (Open identities), reviewers know who the authors are and the authors know who the reviewers are. It can also mean inclusion of the reviewers’ names and/or reports alongside the published paper (Open reports), comments from others at pre-publication stage, or various combinations of these.

• Greater accountability and reduced opportunity for bias or inappropriate actions. Reviewers can be given public credit for their work.
• Potential reviewers may be more likely to decline to review. Revealing reviewer identity may lead to animosity from authors, damaged relationships and repercussions for job prospects, promotion and grant funding.

An umbrella term for a number of overlapping ways that peer review models can be adapted in line with the aims of Open Science. Ross-Hellauer 2017

Main traits (combined, or not):

• Open identities - where both authors and reviewers are aware of each other’s identities (i.e., non-blinded)
• Open reports - where review reports are published alongside the relevant article.

An umbrella term for a number of overlapping ways that peer review models can be adapted in line with the aims of Open Science. Ross-Hellauer 2017

Optional traits:

• Open participation - where members of the wider community are able to contribute to the review process
• Open interaction - where direct reciprocal discussion between author(s) and reviewers, and/or between reviewers, is allowed and encouraged
• Open pre-review manuscripts - where manuscripts are made immediately available in advance of any formal peer review procedures
• Open final-version commenting: Review or commenting on final “version of record” publications.
• Open platforms (“decoupled review”): Review is facilitated by a different organizational entity than the venue of publication.

Nature

• Leder, D., Hermann, R., Hüls, M. et al. A 51,000-year-old engraved bone reveals Neanderthals’ capacity for symbolic behaviour. Nat Ecol Evol (2021). https://doi.org/10.1038/s41559-021-01487-z

F1000

• Ross-Hellauer 2017

Peer Community In (PCI)

• Unreliability and inconsistency: Open identities and open reports are theorized to lead to better reviews.
• Delay and expense: Open pre-review manuscripts sharply reduce the time before research is first publicly available and may increase the overall quality of initial submissions.
• Lack of accountability and risks of subversion: Open identities and reports can increase accountability through increased transparency and by making any conflicts of interest more immediately apparent to authors and future readers.
• Social and publication biases: Open reports adds another layer of quality assurance, allowing the wider community to scrutinize reviews.
• Lack of incentives: Open reports linked to open identities enable higher visibility for peer review activities.
• Wastefulness: Open reports make currently invisible but potentially useful scholarly information available for re-use, as well as providing young researchers a guide.

Ross-Hellauer 2017

• Open identities fosters greater accountability amongst reviewers and reduces the opportunities for bias or undisclosed conflicts of interest.
• Open reports add another layer of quality assurance, allowing the wider community to scrutinize reviews to examine decision-making processes.
• In combination, open identities and open reports are theorized to lead to better reviews, as the thought of having their name publicly connected to a work or seeing their review published encourages reviewers to be more thorough.
• Open identities and open reports enable reviewers to gain public credit for their review work, thus incentivising this vital activity and allowing review work to be cited in other publications and in career development activities linked to promotion and tenure.
• Open participation could overcome problems associated with editorial selection of reviewers (e.g., biases, closed-networks, elitism). Especially for early career researchers who do not yet receive invitations to review, such open processes may also present a chance to build their research reputation and practice their review skills.

Being a peer reviewer presents researchers with opportunities for engaging with novel research, building academic networks and expertise, and refining their own writing skills. It is a crucial element of quality control for academic work. Yet, in general, researchers do not often receive formal training in how to do peer review. Even where researchers believe themselves confident with traditional peer review, however, the many forms of open peer review present new challenges and opportunities.

Open peer review hence aims to bring greater transparency, accountability, inclusivity and/or efficiency to the restricted model of traditional peer review.

• Social biases - Open identities removes anonymity conditions for reviewers (single-blind) or authors and reviewers (double-blind) which are traditionally in place to counteract social biases - status, history, or affiliations of the author(s) might influence the reviewers' judgements (e.g., by discouraging reviewers from making strong criticisms, especially against higher-status colleagues).
  • Reviewers should constantly question their assumptions to ensure their judgements reflect only the quality of the manuscript, and not the status, history, or affiliations of the author(s).
• Subjectivity - Authors and reviewers may subjectively agree or disagree on how to present the results and/or what needs improvement, amendment or correction. In open identities and/or open reports, the transparency could exacerbate such differences and the difficulties to manage emotional reactions.

Given these issues, potential reviewers may be more likely to decline to review.

Curry 2019 [Online access, 16 Jul '21]

• Problems with the traditional peer review system
• Definition of Open peer review
• Advantages and disadvantages of Open peer review
• Metrics (citation indexes, impact factor)
• Problems with metrics and (Leiden Manifesto, DORA)
• Alternatives metrics
• Future direction of the EC / The 'Open Research Europe' platform

Is research evaluation fair?

Research evaluation is as fair as its methods and evaluation techniques. Metrics and altmetrics try to measure research quality with research output quantity, which can be accurate, but does not have to be.

Science is a method whereby a notion proffered by anyone must be supported by experimental data. This means that if somebody else is interested in checking up on the notion presented, that person must be allowed access to instructions as to how the original experiments were done. Then he can check things out for himself. It is not allowable in science to make a statement of fact based solely on your own opinion.

Claims made by scientists, in contrast to those made by movie critics or theologians, can be separated from the scientists who make them. It isn’t important to know who Issac Newton was. He discovered that force is equal to mass times acceleration. He was an antisocial, crazy person who wanted to burn down his parents’ house. But force is still equal to mass times acceleration. It can be demonstrated by anybody with a pool table and familiar with Newton's concepts. K. Mullis, 1998, Dancing naked in the mind field

However, social biases seem to be well rooted in such a 'prestige economy'.

References

• Csiszar, A. Peer review: Troubled from the start. Nature 532, 306–308 (2016). https://doi.org/10.1038/532306a
• Hicks, Wouters, Waltman, de Rijcke, Rafols, Nature, April 23, 2015 http://www.nature.com/news/bibliometrics-the-leiden-manifesto-for-research-metrics-1.17351)

Further resources

• Blackmore, P., & Kandiko, C. B. (2011). Motivation in academic life: a prestige economy. Research in Post-Compulsory Education, 16(4), 399–411. https://doi.org/doi:10.1080/13596748.2011.626971
• Directorate-General for Research and Innovation (European Commission): Evaluation of Research Careers Fully Acknowledging Open Science Practices: Rewards, Incentives and/or Recognition for Researchers Practicing Open Science. Report, 14 November 2017: https://doi.org/10.2777/75255