When peer review is askew, what do we do?

Guest post by Madeleine Ball

Peer review helps improve the quality of research before it is published, but it’s also important to help the public make sense of evidence. Supporting diverse author and reviewer identities is key to ensuring that high-quality scientific evidence is available in decision-making. As Nobel Laureate Jennifer Doudna puts it: “If you want to have the best scientific outcomes, you need a lot of different brains working on it” (1). That’s why any discussion of Identity in Peer Review, the theme of Peer Review Week 2021, needs to include diversity. But diversity isn’t the only aspect of identity.

As an early career researcher (ECR) and Voice of Young Science (VoYS) member, I ask: how does identity in all its facets impact the quality of science that gets published? I discuss issues with blinding and diversity in peer review, and ultimately look at whether including non-peer-reviewed preprints in science publishing can have beneficial effects on reducing bias, encouraging representation and increasing the quality of published material.

Blinding identities can be good for getting ECRs involved in peer review

Blinding means hiding the identities of the authors, reviewers or both. Single-blinding is when reviewer identities are hidden from the author, or vice versa. Double-blinding is where neither peer reviewers nor authors know who the other is. Blinding is intended to reduce identity bias, a known issue in science. A 2012 study showed that an identical lab manager job application is rated as more competent when submitted with a male name as opposed to a female name (2). Most journals in science operate a single-blind system where reviewers are anonymous. Although this presents the potential for bias against authors, totally removing blinding could put a reviewer in an uncomfortable position, where they may be less critical even when it is required, especially if they’re an ECR. Single-blinding can help give ECR reviewers the confidence to get involved without fear of criticising a senior peer. Double-blinding can do this too, but  it may imbue a false sense of security, as bias can seep in at many points in the peer review process (3). 

Opening up the “black box” of peer review can create a better environment for reviewers with underrepresented identities

A blanket policy that required all reviews to be conducted by an equal number of reviewers from underrepresented groups would place relatively more of the burden of reviewing onto a smaller number of reviewers. That’s because reviewers from underrepresented groups are just that: underrepresented – whether they be women, LGBT+ researchers, ECRs or researchers from the Global South. Underrepresentation has been exacerbated by the pandemic. For example, the burden of childcare during lockdowns has disproportionately fallen on women researchers, leaving less time to do peer review. Ironically, in our attempt to include different identities, we could place this burden on the very people we are trying to uplift. With an increasing number of papers being submitted every day (4) and a shortening turnaround time for researchers to provide reviews, a deeper look at peer review is necessary to support representation. As peer review largely happens behind closed doors, unable to be seen or judged by a varied audience of scientists who could give valuable comments, the responsibility for individual reviewers and editors is high. Opening up the “black box” of peer review, encouraging more openness and accountability, is the first step to creating a better environment for more reviewers to get involved.

Preprints are an opportunity to improve quality if they are identified as not being peer-reviewed

At the start of this post, I questioned whether the current structure of peer review supports inclusion and diversity, and how it affects the quality of research outputs. I suggest that community-led peer review of preprints could be a solution, if preprints are identified as not yet formally peer-reviewed. The pandemic has resulted in the increasing use of preprints, where authors have felt it was in the public interest to disseminate Covid-19 research quickly. This has inevitably led to instances of poor-quality research getting through. However, within the science Twitter community, rapid critical commentary on preprints provided a transparent peer review-like process (5). Current peer review practices have not always prevented bad science from leaking through; even worse, some fraudulent or unreliable science has been published with the seal of peer-reviewed approval and trusted by the media and public alike. Overall, I believe wider scrutiny of preprints by the scientific community can improve the quality of research published, as long as preprints are used as an opportunity for added scrutiny not publicity. 

As the number of submissions for peer-review is quickly becoming overwhelming, including preprints as part of discussion before formal peer review can soften the barrier to entry that peer review alone can create. Allowing diverse contributions to spread the burden may ultimately contribute to a higher quality of publication, whilst also embedding the expectation of what good quality peer review looks like to the public, empowering more people to scrutinise scientific evidence.   

References

1. The Nobel Prize in Chemistry 2020 [Internet]. NobelPrize.org. [cited 2021 Aug 19]. Available from: https://www.nobelprize.org/prizes/chemistry/2020/doudna/167713-jennifer-doudna-interview-february-2021/ 

2. Moss-Racusin CA, Dovidio JF, Brescoll VL, Graham MJ, Handelsman J. Science faculty’s subtle gender biases favor male students. PNAS. 2012 Oct 9;109(41):16474–9. https://doi.org/10.1073/pnas.1211286109 

3. The Fractured Logic of Blinded Peer Review in Journals [Internet]. Absolutely Maybe. 2017 [cited 2021 Aug 23]. Available from: https://absolutelymaybe.plos.org/2017/10/31/the-fractured-logic-of-blinded-peer-review-in-journals/ 

4. Squazzoni F, Bravo G, Grimaldo F, Garcıa-Costa D, Farjam M, Mehmani B. Only Second-Class Tickets for Women in the COVID-19 Race. A Study on Manuscript Submissions and Reviews in 2329 Elsevier Journals [Internet]. Rochester, NY: Social Science Research Network; 2020 Oct [cited 2021 Aug 20]. Report No.: ID 3712813. Available from: https://papers.ssrn.com/abstract=3712813 

5. Fraser N, Brierley L, Dey G, Polka JK, Pálfy M, Nanni F, et al. The evolving role of preprints in the dissemination of COVID-19 research and their impact on the science communication landscape. PLOS Biology. 2021 Apr 2;19(4):e3000959. https://doi.org/10.1371/journal.pbio.3000959 

About the Author

Madeleine Ball is a Voice of Young Science member and PhD student at the University of Cambridge, UK, studying chromatin organisation during development via the use of super-resolution microscopy. She has an interest in communicating challenging science to the general public, bridging the gap between scientists and policymakers and discussing ways to improve research quality. If you are interested in joining the public discussion about research quality as an ECR and supporting your peers to do the same, then sign up for VoYS to get involved: Voice of Young Science – Sense about Science