Peer review, preprints, and pandemics

At the height of the Covid-19 pandemic in the summer of 2020, the UK’s Science Media Centre (SMC) was at the eye of the media storm, facing unprecedented pressure to promote research at the preprint stage rather than after peer review. Subsequently, Fiona Fox, SMC Director, wrote a blog describing her team’s experience and suggesting what this may mean for science PR and reporting, originally titled ‘What should press officers advise on preprints during a pandemic?’ As part of F1000’s and Taylor & Francis’ discussion and debate for Peer Review Week 2022, Fiona and the SMC have kindly allowed us to reproduce this thought-provoking piece on the F1000 Open Thinking blog.

We are also delighted that the SMC’s Senior Media Manager, Tom Sheldon, will be joining our online debate Peer Pressure: Does Peer Review Help or Hinder Science Reporting. The full panel is as follows:

• Jodie Bell, Senior Press & Media Relations Manager, Taylor & Francis
• Boyana Konforti, Director of Strategic Initiatives, F1000
• Parth Patel, Senior Research Fellow, IPPR; media commentator; Wellcome Open Research author
• Rachael Pells, freelance journalist in science and research (Times, Nature, Wired, iPaper)
• Tom Sheldon, Senior Media Manager, Science Media Centre
• Richard Van Noorden, Features Editor, Nature
• The Chair of the panel Martin Wilson, Head of Content, Taylor & Francis; former Head of Product, BBC Digital Creativity

To read Tom’s argument in Nature that ‘The scientific community must take measures to keep preprints from distorting the public’s understanding of science’, please go to https://www.nature.com/articles/d41586-018-05789-4

Tom concludes:

How can we have preprints and support good journalism? Should scientific societies or preprint advocates develop guidelines for what should and should not be posted as a preprint? Should there be a preprint moratorium on any research with public-health implications? Should universities or researchers ever publicize a preprint? Should all preprints be emblazoned with a warning aimed at journalists that work has not been peer reviewed?

Preprints could bring great prizes for science. But these questions must be brought up now, so that public understanding is not damaged as preprints flourish.

Join us at the Peer Pressure: Does Peer Review Help or Hinder Science Reporting? online discussion to explore these questions and more on September 21^st (for full details and to book your place, please go to this registration link).

By Fiona Fox

This blog was first published in July 2020 under the title ‘What should press advice on preprints during a pandemic?’ It is kindly republished on the Open Thinking blog in its original form, with a new title and subtitles to aid reading.  The blog contains the thoughts of the author rather than representing the work or policy of the Science Media Centre.

SMC and concerns about reporting preprints prior to the COVID-19 pandemic

Last year, under the leadership of my colleague Tom Sheldon, the SMC spent many months in discussion with the research community about the impact of preprints on science in the media.

Many of you discussed this with us at the time, but here’s a quick and dirty recap. We felt that in the enthusiasm to embrace the benefits for research, scientists had neglected to consider the impact preprints might have on the way science gets to the wider public, especially preprints on medical and clinical research findings with implications for public health. The system for getting these kinds of findings to the wider public had been honed over many years. It involved waiting for findings to be peer-reviewed and accepted by a journal, then issuing a press release under embargo, allowing journalists time to read the paper, speak to authors, seek third-party comments, and maybe attend a press conference.

Our concern, shared by many in university and journal press offices, was that findings of importance to public health could end up being reported to the wider public at a stage when the data was unchecked and possibly wrong. As a press office charged with getting more accurate and measured science to the public, it was obvious that this was something we should care about. After extensive discussions with scientists and press officers, we led a process of drawing up guidelines for them. In short, they urge authors and press officers to hold back from publicising preprints to the wider public until findings are published in a journal, and urge press officers not to press release preprints.

COVID-19 and preprints

Those of you who read my last blog on COVID may remember my new email sign off: #humilityneeded. Just as well, because having drawn up guidelines urging scientists and press officers not to publicise preprints, the SMC has spent the last five months facing a barrage of COVID preprints – and sometimes concluding that the best thing to do is to bring them into the full glare of the media. We’ve gathered third-party comments on them and, on occasion, have even run press conferences on them.

So how has this happened? Most of you, I suspect, will already know that almost all the standard rules of science publishing seem to have been torn up in the face of this global pandemic. Mostly this is a good thing. Global health leaders like Jeremy Farrar have long argued that it is morally untenable to allow evidence that can help scientists better control new infectious diseases to languish in sluggish publication schedules in the middle of an outbreak. With over half a million deaths and counting and so many unanswered questions, speed is essential.

And speed we certainly have. Since the WHO declared COVID-19 a Public Health Emergency of International Concern (PHEIC) at the end of January, research has been happening at an extraordinary pace. Some estimate that almost 20,000 new papers on COVID have appeared during that time and studies that usually take 100 days from submission to acceptance are now taking six days.

Some of these of course are papers fast-tracked for publication. They present some challenges for journal press officers and the SMC because they are often issued for immediate release which means less time to seek third-party comments or organise press briefings. But these findings have still been through peer review and the other checks and balances that go with good journal publishing.

It’s the deluge of preprints that we are struggling with, not least because we were only just getting our heads round the issue before COVID-19 and all our guidelines seem to have been made sorely inadequate almost overnight.

Pandemic preprints, press reporting, and social media

I first realised this in January when things had become so busy that the SMC’s team of five press officers had begun working a seven day week on a rota system. I was leading on the Saturday and getting a handover from a colleague who flagged up a new modelling preprint, from Glasgow and Lancaster led by Dr Jonathan Read, predicting a pretty scary R0 of 3.8 that was higher than that predicted in other models. My colleague assured me they were good scientists and a respected group. This put us in quite a new position – the findings at the time sounded alarming and would have benefited from some third-party comments to help add context, but the SMC didn’t want to do anything to publicise a preprint – especially a scary one. We left the office on Friday night hoping no one except other scientists would see the preprint.

By Saturday morning I had 20 journalists asking for comments on the preprint which had gone viral. What to do? Hope that everyone would still wait for journal publication? Or help the journalists report the scary numbers with the aid of other experts in the field? It felt like a no-brainer, so I sent out a call for comments. The first response was from one of the authors who pointed out that they were working on revisions which would ultimately show that R0 was closer to 3.1. To their credit the authors had tweeted about the revisions, but my inbox was telling me that lots of the journalists hadn’t yet seen the correction so I ended up sending comments from third parties as well as forwarding a comment from the authors about the update. The preprint was revised but not until several days later – too late for some news articles and much too late for Twitter where people were either repeating the inflated number or pouring bile on the poor authors (one Harvard epidemiologist calling it “thermonuclear pandemic level bad”).

Several things became clear in the weeks after that. That many scientists were keen to post their preprint for discussion amongst academics, but also to announce it to the wider world on Twitter and through the media. That national news science journalists who, in the past, had had neither the urge nor the time to pick up stories from scientists’ Twitter pages or follow unpublished research were now doing so routinely on COVID. And that despite journalists often taking care to emphasise that such findings were not yet checked or published, the global thirst for information was so high that almost all new findings on COVID were getting huge media coverage.

Since that Saturday, the SMC has worked on many COVID preprints. We issue third-party comments on any we think will be picked up, either because they have been press released or because journalists have asked for comments. While our third-party comments often urge caution in interpreting such preliminary findings or occasionally point to major flaws, it was not lost on us that by issuing comments we were ourselves publicising the findings.

And to date we have run six press conferences on preprints. I should add that we emphasise in the subject line and invitation that the findings are only at the preprint stage, and we encourage the authors and any third-party scientists commenting to stress that too. The UK’s science journalists have gone out of their way to reflect this in their reporting and deserve plaudits for this and much else about their reporting of this pandemic.

So were we right to go against our own guidance so quickly in the face of the pandemic and, if we were, does it follow that the guidance needs to be revised? I don’t have the answer. But as we slowly emerge from the madness and chaos of 15 hour days and seven day weeks where it was sometimes impossible to find time to think, this blog post is my attempt to reflect with our friends on whether we need to reset.

Pandemic preprints: good and bad

Let’s remind ourselves why this matters. If it’s more important than ever to get new evidence out during a raging pandemic, it stands to reason that it may also be more problematic than ever if that evidence is flawed or wrong. There are more than a few examples of flaky preprints being covered by global news outlets. When schools should reopen and the role of children in the spread of COVID-19 have been amongst the most heated debates over the past few months. So, unsurprisingly, when Christian Drosten (head of the Institute of Virology at Germany’s largest university hospital) tweeted in April about his latest preprint study suggesting infected children had the same levels of virus in their bodies as adults, it was widely covered across Europe and was seized on by those cautioning against reopening schools.

Several of those we approached on this study raised alarm bells, including this excoriating comment from Dr Alasdair Munro, a clinical research fellow in paediatric infectious diseases from University Hospital Southampton: “This paper was released via social media in a format which has not yet been peer-reviewed… There are several methodological issues which compromise the ability of the paper to answer the question which it addresses, including some inappropriate statistical analyses which have resulted in an inaccurate conclusion.” 

Elsewhere, top statisticians weighed in with a blog post explaining why the preprint actually showed the opposite to what was claimed – that infected children have lower levels of virus than adults – and called for the paper to be withdrawn from circulation. Eventually, the authors released a new version of the preprint, over a month after the original was posted, but there was little media coverage of this second version.

Other examples of flaky preprints include one which suggested the virus was present in Wuhan as early as September 2019 based partly on satellite images of hospital traffic in the city; a modelling study that predicted overall deaths wildly different to other models, carried out by a scientist who works on the development of space instruments; and the now infamous Santa Clara preprint which suggested the number of people who’d been infected there was 50 to 85 times higher than thought.

So there is little doubt that some preprints have contributed to dangerous misinformation in this pandemic. But other preprints have played a far more honourable role, providing data that has allowed policy makers, the NHS and the public to better understand the virus and allowed other scientists to make remarkable progress in research on the virus in just six months.

The briefings we ran on preprints were all on large studies led by scientists we have worked with for years. We ran a briefing with Prof Liam Smeeth from LSHTM and Dr Ben Goldacre from Oxford on the world’s largest study on factors associated with death from COVID-19, looking at data from over 17 million patients. The authors wanted it to be publicised swiftly because it revealed which diseases and demographics were most strongly associated with death from COVID-19 and could help inform thinking about shielding, risk management, and further research on factors associated with COVID death.

Another briefing on a preprint was a rapid systematic review of contact tracing and population studies done globally on COVID-19, led by Professor Russell Viner at UCL, to address questions around susceptibility and transmission of the virus in children. Again, the reasons for speed seemed clear. The study was a systematic review rather than an individual study so it was a snapshot of everything that was known at that stage, and more critically what remained unknown.

What next for publicising preprints?

As we move slowly back to the ‘new normal’, I think the SMC and other science press officers should continue to be cautious around preprints. It’s reasonable to push back on those academics wanting press releases unless it’s very clear that the findings are significant and need to be in the public domain urgently.

Press officers asking their academics to wait a while for a press release should not be seen as unhelpful. There are reasons other than the potential for misleading coverage to wait. Most scientists still aim to get their findings published and want media coverage and impact at that stage. But journalists are unlikely to report similar findings twice. We’ve already had to break the news to scientists who ran press briefings at the preprint stage that we couldn’t run another one a few weeks later when it was published in a top journal. On other occasions, we’ve pushed back to scientists on the basis that we could not see the public interest in accessing the findings urgently, and said that we would be happy to run a briefing as soon as the paper was published.

I also think we need to remind scientists that one of the benefits of preprints is the opportunity for other scientists to spend some time critiquing and improving on raw research. Publicising research as it’s posted means the public is losing out on the chance to see their science at its very best.

Whilst there is some remarkable science appearing on preprint servers right now, there is also some junk. If press officers suspect that scientists are keen for publicity simply to raise their profile or get a slice of the action, then we should actively resist. It’s not easy for science press officers to tell academics that their research isn’t good enough, but we can and should flag any concerns about findings that may influence public behaviour but which are less than robust. There are various warning signs we can look out for – scientists who are working way outside their normal discipline, studies from a single author rather than a group, or findings which are dramatically different to similar work. The SMC’s mantra that ‘extraordinary claims need extraordinary evidence’ seems to hold even more true for preprints.

I also think that press officers who urge caution are doing their academics a favour. Many researchers are still naïve about the way the news media works, especially during a pandemic which is the only story in town. When interviewed for WIRED magazine, Dr Jonathan Read whose preprint had to be corrected said, “My suspicion is we don’t really know how people take this information and use it and work out what to make of it. We tend to just put these numbers out there. I’m not a Twitter user, so I didn’t really know what to expect. It’s been a steep learning curve.”

As we enter a calmer time where the UK appears to have come through the first wave of the infection, many in the scientific community will reflect on the lessons we need to learn. I hope this will include press officers and researchers reflecting on the way we approached preprints at the height of the epidemic.

I also hope we don’t walk into a situation where the need for speedy dissemination in a pandemic becomes the norm for all research. It’s great for open science if the pandemic has escalated the trend towards posting science on preprint servers, but it does not follow that it’s right to publicise new findings at preprint stage during peace time. Our guidance on publicising preprints did not apply to every situation the pandemic threw up, and I think we were right to adapt. But I see nothing that suggests that it is now redundant.

There will be many aspects of this virus that the wider public need to hear about quickly for some time to come, including which treatments do and do not work, whether children are as infectious as adults, and whether having had COVID-19 makes us immune to catching it again. There are lots of ways that data on these and other subjects will emerge outside of journal publication, and I have no doubt the SMC will continue to enthusiastically work with the best scientists to get this research to the public. But we and other science press officers do have to make judgements and provide advice and support to scientists. Suggesting scientists should wait until publication before press releasing their findings will often be the best media relations advice – for science and for the public interest.

With thanks to Hannah Taylor Lewis for her contributions to this blog.