The professor will see you now - polling
4 min read
In an occasional series, Professor Philip Cowley offers a political science lesson for The House’s readers. This week: +/-3
With a general election due within the next 12 months, the optimist in me hopes that this will be a contest in which opinion polls are properly reported – while the pessimist in me knows that pollsters and journalists need the cash and clicks that come with over-hyping surprising poll findings.
Most informed observers – readers of this magazine, say – know that opinion polls come with uncertainty baked in. For most standard opinion polls in the United Kingdom, the margin of error (or MoE) is +/- three percentage points. In other words, if a poll puts a party on 40 per cent, it means their actual level of support is somewhere between 37 and 43 per cent.
The consequences of this are often less understood, though. When comparing change over time, the uncertainty applies to both the current and the past figure. To take a recent example, after last year’s Autumn Statement, The Times reported that Conservative support had risen by four points with YouGov, compared to the last poll by the same company. Pre-statement, the Conservatives were on 21 per cent; afterwards, they polled 25 per cent. That could indeed be a four-point jump in support as a favourable reaction to the policies announced.
But a four-point poll increase would also be perfectly consistent with actual support for the Conservatives having been stable at any of 22, 23, or 24 per cent – or even having fallen slightly (24 per cent before, 22 per cent after). And indeed, as soon became clear once a few more polls were published, this more sober assessment of the polls proved correct.
It is not just that this slightly hysterical reporting is misleading. It can also be consequential. A recent piece of research tested British voters’ reactions to these apparent changes in party support. It showed voters one set of poll figures and asked people how likely it was that the party in the lead would win. Another group were shown the same figures but including the degree of change since the last poll (+3, +6, or whatever). Do that, and you discover that voters are responsive to momentum. If a party’s support had increased since the last poll, it was seen as more likely to win, compared to when voters were shown the same poll figures without any indication of the direction of travel.
It is not just that this slightly hysterical reporting is misleading. It can also be consequential
Unsurprisingly, the larger the momentum, the bigger the effect, but the effect held even if the change made little difference to a party’s chance of victory, and crucially even if the change was within the margin of error. That is, even if the changes in the level of support could have been entirely due to noise and random polling error, they changed the way that people interpreted the poll – and how they saw the nature of the political race. Parties could appear to have the Big Mo, even when it was really just MoE.
A second study tested this directly, in a neat survey experiment in Germany that showed people opinion polls both with and without the margins of error and then compared their reactions. The point estimates seem definitive – more definitive than they actually are – but once you present voters with the uncertainty inherent in polling, it turns out it changes their tactical considerations. In particular, the study found that if the margin of error makes the race look close and uncertain, voters become more likely to vote for one of the major parties.
This is all why it is never sensible to get too excited about any individual poll results, but to look for wider trends. It is also why it is important that news outlets and others report polls properly. The German study found that how the polls were reported was a significant factor in voters’ interpretations. This means always mentioning the margin of error and not over-hyping tiny changes in party support. I won’t be holding my breath.
Further reading: W Krause and C Gahn, Should we include margins of error in public opinion polls, European Journal of Political Research (2023); M Barnfield, Momentum in the polls raises electoral expectations, Electoral Studies (2023)
PoliticsHome Newsletters
Get the inside track on what MPs and Peers are talking about. Sign up to The House's morning email for the latest insight and reaction from Parliamentarians, policy-makers and organisations.
