Ken Livingstone’s defeat in the London mayoral election on Thursday was both shocking and unsurprising.

Shocking because, despite the presence of a large number of marginal seats, London remains a ‘Labour city’ as the party’s strong performance in the London assembly elections, and the narrowness of Livingstone’s eventual loss, demonstrated. Shocking, too, because, in Boris Johnson, Livingstone faced an opponent who, despite his undeniable popular appeal, has very little to show for his four years at City Hall.

Nonetheless, far beyond the ranks of those who, like this magazine, would have preferred that Labour had picked an alternative candidate in 2010, there was little surprise at the result.

To his credit, Livingstone’s frequent visits to the outer London suburbs during the campaign suggested an understanding that he could not, as he appeared to believe in 2008, win simply by turning out a huge vote in inner London. However, the former mayor’s offensive remarks about the Jewish community indicated a continuing attachment to the ‘divide and rule’ strategy – pitting one section of London society against another – which he employed so unsuccessfully four years ago.

Livingstone was, indeed, lucky to escape censure or punishment for remarks which, directed against any other race, may well have led to him being stripped of the party’s candidacy. And it was not just in this regard that Livingstone was treated in a way that no other Labour member – let alone candidate – would have been afforded. The former mayor’s appeals to party loyalty rang particularly hollow for those who recalled him campaigning in late 2010 for Lutfur Rahman, the independent mayor of Tower Hamlets, against the Labour candidate.

After Johnson’s victory in 2008, Livingstone’s supporters were quick to trumpet the ‘progressive premium’ – the degree to which he outperformed Labour’s dire vote nationally that year – which appeared to attach to his candidacy. It is undeniable that Livingstone had a broader appeal than Labour four years ago. But this was clearly not the case this year. In short, this was a defeat for Livingstone, not for Labour. Indeed, the party – from its grassroots to its leadership – fought a strong and energetic campaign in London, and, despite the result, will be all the stronger for it when it comes to the European and borough elections in 2014 and the general election the year after.

There are, though, wider lessons that Labour can learn from the former mayor’s defeat.

First, there were three vital tests in the campaign: on policy, competence and vision. On policy, Livingstone was a true reformer during his previous time in office: introducing the congestion charge, transforming planning policy, pioneering neighbourhood policing, and playing a critical role in securing the Olympics in 2005. But this time around, a strong reform message did not figure in Livingstone’s campaign. Yes, he had a perfectly credible pledge to cut fares. But Livingstone had little to say on how he might actually improve London’s transport services, perhaps a reflection of his unwillingness to line up with passengers against the Luddite leadership of the RMT. As in 2008, the former mayor appeared unwilling, beyond a promise to reverse cuts in police numbers, to give the critical issue of tackling crime the prominence it needed.

Johnson’s greatest weakness – his competence – was barely exploited by Livingstone, although the former mayor’s inability to explain or justify his tax affairs will not have commended him to many voters as a competent steward of the city’s finances. Perhaps most curiously for a man who throughout much of his previous time in office appeared able to articulate a sense of London’s place in the world, Livingstone was unable to offer any real positive vision for the future.

While we profoundly disagree with those in Labour’s ranks who, on the eve of the elections, encouraged people not to support the party’s candidate, it was Livingstone’s failure to pass these tests which was the true cause of his defeat.

Second, the process by which Livingstone was selected as Labour’s candidate was seriously flawed. A rushed selection during the leadership election prevented other, potentially stronger candidates, putting themselves forward. The two-college electoral system – representing only party members and the trade unions, and with no place, unlike the college which elects the leader of the UK, Scottish and Welsh parties, for Labour’s elected representatives – effectively put the choice of the candidate in the hands of eight trade union leaders.

Reverting to an electoral college with three equal sections would be preferable to the current system. However, our preference is for a more radical alternative: that the party’s candidate for 2016 should be picked by a primary, open to anyone willing to register as a Labour supporter. The strong performance of independent candidate Siobhan Benita in this year’s campaign shows again the waning utility of outdated appeals to voters’ tribal party loyalties. A more open selection process, which engages those who share Labour’s values, would graphically demonstrate the party’s realisation of this fact.

With some dignity, Livingstone has made clear that he will not seek the party’s selection in 2016. This is the right decision and offers the opportunity for a fresh start. Labour in London should seize it with both hands.

———————————

At Progress annual conference join the debate ‘After 3 May: Where next for Labour in London?’ with David Lammy MP and other speakers. Book your ticket now. Join Progress to attend for free.