The professor will see you now - leadership elections
6 min read
In an occasional series, Professor Philip Cowley offers a political science lesson for The House’s readers. Here: leadership elections.
To misquote Sir Lewis Namier: party leadership elections are the locks on the canal of modern British history, regulating its flow. Of the 16 British prime ministers since the end of the Second World War, only two – Clement Attlee and Edward Heath – both initially entered and finally exited Downing Street at a general election. The rest came in or left No 10 mid-term. Of the last eight prime ministers, six came to power as a result of an internal party contest.
These are events at least as significant as general elections, but about which we know relatively little given their importance.
We know about the formal procedures, of course. One of the guiding rules of British political journalism is that the processes parties use to choose their leaders must be described as complicated and arcane (bonus journalese points for ‘Byzantine’), even when they are in fact relatively straightforward.
We also have some enjoyable narrative descriptions of different contests, albeit of varying quality. Of Randolph Churchill’s account of the events of 1963 Iain Macleod wrote that four-fifths “could have been compiled by anyone with a pair of scissors, a pot of paste, and a built-in prejudice against Mr Butler”. The same, mutatis mutandis, could be said of many of the accounts of the contests since.
Yet our understanding of general elections has advanced so much precisely because we’ve moved beyond an understanding of the formal rules and narrative accounts and tried – sometimes imperfectly – to measure systematically what actually happened.
There have been some attempts to do this to leadership elections. Over 30 years ago, using aggregate data Len Stark showed pretty convincingly that British leadership contests were driven by a hierarchy of needs; when choosing leaders, unity trumped electability, electability trumped policy preferences.
He also argued that the rules chosen didn’t make much difference to the outcome. For all that parties obsessed about procedures, it was likely that the same people would have been elected in most cases, although the possible exceptions to this claim — including both Margaret Thatcher and Jeremy Corbyn — strike me as important enough to treat that particular conclusion with some caution.
But there is a paradox here. Because when people have tried to study the way MPs vote during leadership contests, a desire for unity and electability don’t seem all that important; in study after study, the key divide seems to be more ideological.
Take, for example, the process by which the current incumbent reached No 10. An excellent piece of recently published research on the events of October 2022 finds clear differences between those who were known to have nominated Rishi Sunak, Boris Johnson or Penny Mordaunt. This is the latest in a line of studies tracking the way MPs have voted (or in this case, nominated) in leadership elections, and they all seem to show ideological divides are the key driver of the way MPs vote.
Sunak’s support came disproportionately from socially liberal MPs who were – in the words of the authors – not as “stridently pro-Brexit” as other parts of the Conservative parliamentary party. Johnson, by contrast, attracted support from “loyal hard-line Eurosceptics”. Other variables occasionally matter, but in different ways this European divide has been present in every leadership contest for which we have data since 1990.
There are two issues with these sorts of studies, though, which I suspect help explain this apparent paradox.
These are events at least as significant as general elections, but about which we know little
The first is perhaps obvious but relatively unimportant. Many of these contests – and all the ones that have changed the prime minister – have involved secret ballots. While many MPs declare publicly who they are supporting, not all do – and we know some fib. We are therefore working with incomplete data. We know Boris Johnson had at least 100 backers in late-2022; we know the identities of 65 of them. With a bit of detective work we can usually patch together enough information to work out what’s going on in broad terms, especially post hoc once we know how many votes each candidate received, but it’s still a partial picture.
The second is less obvious but probably more significant. We don’t know why MPs are voting that way. We can sometimes infer motives; if we find all the MPs of the right voting for a right-wing candidate, those on the left backing the left-wing candidate, we have a fairly good idea what’s going on. But, as always, it’s the ones in the middle, the waverers and the floaters, for whom we need to know what’s driving their vote. It would be plausible for the parliamentary swing voters to be the ones being driven by a desire for unity or electability, not ideology, and for them to be the ones making the difference in any contests.
To take one example, how do we test the importance of the (perceived) electability of the candidates? Usually, it is to see if MPs in marginal seats are more or less likely to back a certain candidate. This rarely shows any effect, but then perhaps we would be surprised if it did. Electability is a collective good as much as it is an individual one. MPs in all types of seats can be driven by a desire for an electorally successful candidate; those in marginal seats because they want to remain MPs, those in safer seats because they know that extended occupancy of the opposition benches is a miserable way to grind out your political career. Yet we have no way of detecting this. This isn’t, by the way, a criticism of these authors – I’ve done the same sort of analysis plenty of times. It’s more that we just lack the data to test it properly.
The same problems, with knobs on, apply if the wider membership is involved. What polling there is of the parties’ grassroots has a good track record at telling us who is going to win. It is much less good at telling us why.
The perennial cry of the academic is that more research is needed. But when it comes to how we choose our Prime Ministers, it is justified. We need the equivalent of a British Election Study programme but for leadership contests. If anyone reading this wants to fund such a project, do get in touch.
Your further reading for this week: L Stark, Choosing a Leader: Party Leadership Contests in Britain from Macmillan to Blair (1996); O Booth et al, Selecting Sunak: Conservative MPs’ Nomination Preferences in the (Second) British Conservative Party Leadership Election, Parliamentary Affairs, 2023
PoliticsHome Newsletters
Get the inside track on what MPs and Peers are talking about. Sign up to The House's morning email for the latest insight and reaction from Parliamentarians, policy-makers and organisations.
