Why is Covid modelling so controversial?

(Illustrations | Tracy Worrall)

If Sterling hadn’t got a penalty against Denmark, then the whole epidemic might have been different.”

Professor Graham Medley is using a little artistic licence to explain the uncertainties in statistically modelling Covid – after all, Raheem Sterling’s winner against Denmark came late on in the European Championships last summer, and needless to say the “whole” pandemic did not turn on events 18 months in.

But his basic point is valid and vivid. The social mixing that accompanied England’s run to the final was likely a factor in helping spread the virus, culminating in the “pingdemic” that saw widespread workplace absences. For statisticians trying to picture how the pandemic might develop over future weeks and months, the performance of the England football team at a major tournament counts as one of Donald Rumsfeld’s “known unknowns”.

“Picture,” not “predict”. As controversy continues on the political right over the accuracy of Covid modelling by the government’s scientific advisory committees, Medley is keen to stress that his job – as professor of infectious disease modelling at the London School of Hygiene and Tropical Medicine (LSHTM), member of SAGE and co-chair of the Scientific Pandemic Influenza Group on Modelling (SPI-M) – is not about making predictions.

“We cannot quantitatively predict what’s going to happen,” he says. “Our job is to provide the envelope of possibilities from the best to the worst, and then to give some indication to policymakers about that uncertainty, and about what factors drive that uncertainty.” Different scenarios are considered – different levels of transmissibility of a new variant, different levels of vaccine effectiveness, and so on.

SPI-M, effectively a subgroup of SAGE, doesn’t conduct modelling itself – instead it collates scenario modelling from independent teams at institutions such as Imperial College and the University of Warwick, reviewing and comparing these models to produce a consensus and draw conclusions from it.

When collating these models, SPI-M’s experts try to ensure the worst-case scenario they present is more pessimistic than what eventually happens. “The worst thing for me would be for the government to turn round to me and say, ‘you didn’t tell me it could be as bad as this’,” says Medley. “But the consequence of that is that the top level of whatever we do is always worse than what actually happens – by design. It has to be.”

This can spark accusations of alarmism against SAGE when newspapers focus on the worst-case scenario with little context, sometimes aided by quotes from individual scientists who want to see tougher measures.

But it’s the lower end of the “envelope of possibilities” that’s now inflaming lockdown-sceptics in the Tory Party and right-wing press. Discontent that has rumbled for more than a year has grown in response to the relatively low hospitalisation rates in the Omicron wave, compared to the high number of infections.

SPI-M’s “consensus statement” on 15 December, as Omicron took off in Britain, listed three scenarios and the modelled trajectories for each in terms of infections, hospitalisations and deaths, plus uncertainties surrounding each one. The SPI-M report stated explicitly “these are not forecasts or predictions”.

For scenarios in which no restrictions were introduced beyond the government’s already-adopted Plan B, the collated models showed hospitalisations peaking at 3,000 to 10,000 per day in January or February, with deaths peaking at between 600 and 6,000 a day sometime between mid-January and mid-March.

“No further restrictions [beyond Plan B] were introduced, yet cases are falling and it is clear that we will never reach anything near the number of deaths predicted by even SAGE’s most optimistic scenarios,” says Steve Baker, an arch-critic of lockdown measures and deputy chair of the Covid Recovery Group of Tory MPs.

“All of the evidence appearing from South Africa in December 2021 was that it was milder, but the modelling presented to ministers ignored this. We must question why.”

The SPI-M consensus statement warned it was too soon to drawn conclusions from early figures on hospital length of stay in South Africa, given differences with Britain in terms of demographics, vaccination levels, variant composition and other factors.

Nevertheless, more than a month later, it looks highly unlikely deaths will reach anywhere near 600 a day during the Omicron wave, let alone 6,000.

In a rare explicit prediction, SPI-M’s consensus statement said hospital admissions were “highly likely” to hit 1,000 to 2,000 per day in England by the end of 2021 – and so it proved, with figures dancing around the 2,000 mark going into 2022. But levels haven’t subsequently taken off in the way indicated by the models. The booster rollout, which didn’t have time to affect hospital admissions in December, may have kicked in by the New Year.

“Hospitalisations have turned out to be a lot better,” says Medley, “and there are a number of possible reasons for that which we don’t fully have data for. One is that boosters are working a lot better than they might have done, another is that the anti-virals being used are working in sufficient numbers. And then the other possibility is that the NHS has been setting up oximetry [which measures blood oxygen levels] at home and virtual wards to keep people out of hospital. And I don’t know how full they are.”

Another potential factor is behavioural change. Many people voluntarily restricted their socialising in December so as not to catch Omicron before visiting family at Christmas – an approach advised by England’s chief medical officer Professor Chris Whitty but not the government, and not required by the Plan B rules.

Hospitalisations have turned out to be a lot better and there are a number of possible reasons for that which we don’t fully have data for.

Medley says that as with England’s Euro 2020 run, this was hard to model in advance. “I saw a behavioural survey a couple of weeks before Christmas, trying to help us understand what people were going to do over Christmas, and the biggest response had been ‘undecided’. So two weeks before Christmas a lot of people didn’t know or weren’t prepared to say what it is they were going to do. And so we can’t include a prediction of that – all we can say is it might go down or it might go up.”

Michael Simmons, who monitors Covid data for the lockdown-sceptic Spectator magazine, wrote recently that two of the key models SAGE relied on made no attempt to predict how individuals would change their behaviour in response to perceived risk.

“It is not possible for models to exactly reflect how people will behave as this depends on a multitude of factors, both at population and individual levels,” the SAGE document said.

“Might this be the biggest single flaw in Sage modelling?” wrote Simmons. “A year of global lockdown studies have shown that people very much change their behaviour, often well in advance of lockdown rules.”

But this begs another question – if measures to stop people socialising can be averted because people voluntarily stop socialising for fear of catching the virus, where exactly is the gain? There is still a hit to the night-time economy; people are still left feeling isolated and alone.

Baker sees it as a civil liberties issue. “There is a huge difference between the public choosing to limit their contact and the government ordering them to. We seem to have adopted the idea that the public aren’t adults, can’t understand serious issues and will only make the wrong choices when the government allows them to.”

Aside from SAGE, Baker is critical of government departments for not publishing their own modelling of the impacts of lockdown measures, alleging a lack of publicly available cost-benefit analyses. “This is not only concerning as it likely means that government ministers often did not have the full facts when making decisions about lockdowns, but it also means that the public was not fully informed of the impact of the government’s policies.

“Many of the problems we are seeing now – the long NHS backlog, children falling off schools’ registers, the problems the current state of our public finances will cause in the future – are due to a lack of modelling of the non-Covid impacts of lockdown measures.”

Arguably it is much easier to quantify the economic cost of lockdown measures than mental health or children’s education – trying to model these long-term consequences of lockdowns could plausibly throw up the same kind of uncertainties and “envelopes of possibilities” as modelling the pandemic itself.

Medley says Covid modelling has advanced hugely over the last two years, with daily figures on the number of people being tested and those testing positive, broken down by age, sex and location. It’s now possible to model impacts in different local authorities, accounting for such factors as different school term times.

What’s missing is information on ethnicity, which people often don’t fill in and isn’t available from hospitals in real time.

“One of the major concerns we’ve always had is about inequity and the fact that the epidemic will have different impacts on different groups at different times. But that’s something we’ve never been able to capture in models,” says Medley.

“The fact we haven’t been able to do it in the past two years is not a good thing, because it is important, it’s critical to managing the epidemic. But it really is about the lack of data, to some extent the policymakers aren’t driving that. The policymakers’ focus has been on deaths, and then on hospitalisations, but not on inequity.”

It is not the job of SAGE or Covid modellers to decide policy – those decisions are for elected politicians. Epidemiological modellers do not, for example, have the mandate to make decisions on first principles, such as whether lockdown measures can be justified to manage NHS demand.

But the reports of modellers have inevitably become wrapped up in those political arguments, which has left Medley with mixed feelings – that transparency is good and necessary, but also makes his job much harder.

“This whole relationship between science, public, media, decision-makers, has really been challenged, stressed, in a way that I haven’t seen for any other policy situation.”

Related

Kim Leadbeater On Assisted Dying: "I Came Into Politics To Make A Difference"

By Sienna Rodgers

23 Nov

PoliticsHome Newsletters

Get the inside track on what MPs and Peers are talking about. Sign up to The House's morning email for the latest insight and reaction from Parliamentarians, policy-makers and organisations.