Over the last century, successive progressive governments have wrestled with the challenge of renewal in office – some more successfully than others. The ‘progressive dawn’ of the 20th Century is commonly held to have begun a century ago this year, with the Liberal landslide of 1906. There would be a Liberal in Downing Street until Lloyd George’s ejection in 1922. But despite the significance of Lloyd George’s 1909 People’s Budget and the subsequent confrontation with the House of Lords, the Liberals failed the renewal challenge.

After the enactment of the initial Lloyd George national insurance schemes, there was no consensus amongst Liberal parliamentarians in favour of the more radical social reform necessary for a renewed Liberal party to remain relevant to a wider electorate on a future universal franchise. The Liberals were split between progressives and reactionaries on key issues of the age, such as votes for women, and they were fixated on outdated totemic issues such as temperance, church-disestablishment, and opposition to conscription. The Liberal governments proved unable to renew themselves for the simple reason that a critical mass of Liberals were not in fact progressive.

By 1916, many Liberal MPs were on the opposition benches. The last Liberal prime minister, Lloyd George, would find himself a prisoner of the Conservatives who made up the preponderance of MPs upon which his parliamentary majority depended. Faced with a straight choice between abandoning the social aspirations of his pledges and alienating his Conservative coalition partners, he chose the former. The ‘homes fit for heroes’ programme was abandoned. Progressives, like the health and housing minister, Christopher Addison, were dumped, and most subsequently joined Labour.

Ramsay MacDonald’s 1924 minority Labour government, in office for barely a year, had insufficient time to fulfil its initial aspirations, never mind run out of steam. For the second Macdonald minority government of 1929 to 1931, the central challenge was clear: in Lloyd George’s famous phrase, to ‘conquer unemployment’. But as Oswald Mosley pithily described in his resignation speech, Labour ministers failed to act cohesively and proved incapable of surmounting the obstructionism and small-mindedness of their civil servants, who succeeded in miring initiatives in turf-wars between departments and between national and local government. Without special advisers, they lacked allies in reconnoitring around Sir Humphrey’s flank. MacDonald’s government ended up ‘in office but not in power,’ until MacDonald infamously ditched his party and Labour lost office as well.

By 1945, Labour ministers had already held office through five gruelling years of war. But the opportunity to capture the commanding heights of the British state gave them renewed energy. The tragedy for Attlee’s 1945 to 1951 government was that it became distracted from the real challenge of improving the way nationalised industries were run by leftwingers who wanted to focus instead on a ‘shopping list’ of further industries to nationalise. Deputy leader Herbert Morrison warned against this in 1948: ‘A time comes, as we know, in war when headlong advance must be followed by detailed consolidation … That is the stage which we are now reaching, and if we go on always stretching our hands out for more and not making good the gains we have claimed, only disaster can follow. The test must now be the test of results. We have swept away the charity and poor law state and established the social security state, but [it] cannot endure unless it is also a state of social responsibility.’

But the British state was not built to achieve the objectives that Labour hoped it would attain. The civil service ethos was rooted in England’s class traditions and the culture of the public schools, which had for decades educated a classically trained caste of administrators to run the colonies. With colonial independence, the next generation of colonial governors went instead to Whitehall to tell the natives in the provinces of Britain what was good for them. According to a 1964 Fabian report, the civil service remained ‘as closed and protected as a monastic order’. The civil service class was temperamentally and culturally unsuited to the task of re-engining a flagging economy and working with a working class of whom they knew little and understood less, and a business community many had been brought up to despise.

The 1950 election saw Labour campaign on a platform of ‘fair shares for all’, and again secure more votes than the Conservatives. Nevertheless, the Conservative pledge to ‘set the people free’ resonated with many voters grazed by the rough edges of bureaucratic inflexibility and incompetence, producing a strong swing to the Conservatives in the suburbia and commuterland of London and the south-east. At the 1951 election, although Labour again got more votes, this time the Conservatives got more seats, due largely to the implosion of the Liberal party, who fielded candidates in barely 100 seats.

Labour’s next government came in 1964, when Harold Wilson successfully claimed to be more in touch with ordinary Britain than the Tories, whom he successfully caricatured as being more comfortable donning antique tweeds to shoot grouse than building a more meritocratic and egalitarian society for the ‘jet-plane age’. Wilson had only a wafer-thin parliamentary majority and began cautiously.

In 1966 he appealed to the country for a strong majority to enable him to govern decisively. He got it, but his government proved unable to deliver. The economic mess inherited from the Conservatives forced the government not only to ditch many of its progressive spending plans, but also to make slashing cuts. Meanwhile, British industry continued its relative decline and the government’s attempt to reform industrial relations ended in abortive humiliation.

Labour’s progressive Home Office reforms were not core issues to sufficient aspirational working-class voters. Their priority was social reform and improved public services to secure better life chances for themselves. It was this that didn’t sufficiently happen. Harold Wilson’s 1966 to 1970 government was a lesson, in Clive Ponting’s phrase, in ‘breach of promise,’ and ultimately Labour paid the electoral price.

Brilliant as he was, Wilson proved unable to renew Labour in government. Back in office in 1974, barely a year later he told his policy unit head, Bernard Donoughue: ‘I have been round this course so often that I am too bored to face jumping any more hurdles.’ On another occasion he said: ‘The trouble with me now is that I only have the same old solutions for the same old problems.’

Although an older man and a veteran senior minister, it was James Callaghan, prime minister from 1976 to 1979, who showed most nearly the vision to revive a tired government and engage with voters’ concerns. His success was reflected in Gallup approval ratings of at least 50 per cent for the 15 consecutive months preceding December 1978. This was a feat unequalled at the end of a full-term government between 1955 and 1997.

Helped by his policy unit special advisers, Callaghan identified several areas of public concern as themes for his government. These included school standards, highlighted in his famous Ruskin speech. A second theme was social housing. Callaghan declared that he wanted an end to the stories of bureaucratic ineptitude ‘every councillor and every canvasser hears [from] council tenants: “It’s taken them six months to come round and fix the tap.” … Let the individual tenant decide if he wants a new kitchen. Let him have a choice of colour for his front door. Let the tenants be involved in running their estates, so that they suit their needs. These are for them, for their lives. They are not just there for the convenience of town hall officials.’

A third theme was ‘crime, violence and vandalism’. A fourth was ‘to make the management of the health service more responsive to the patients and [employees].’ Callaghan sought clear positive dividing lines between Labour and Conservatives, to give Labour people something to vote for: ‘We do not want to win support on the basis of fear. We are the party of hope.’ It was his tragedy that the 1978 to 1979 Winter of Discontent torpedoed his government’s credibility.

In his last Labour conference speech as prime minister, Callaghan called for ‘a new basis for authority … The informed, active agreement of the individual citizen will certainly be sounder than the notions of the gentlemen in Whitehall.’ Tony Blair harnessed this ideal in his 1994 conference speech as part of the first dividing lines he sought to put down between New Labour and the Conservatives: ‘To the pensioners who fear to go out of their homes … Your concerns are our concerns. To the small businesses, pushed to the wall by greedy banks, to employers burdened by government failure, to employees living in fear of the P45 … let us say: “The Tories have forgotten you … Labour is on your side. Your aspirations are our aspirations”.’

With time, parts of Blair’s governments appear to have forgotten this. The challenge for Gordon Brown will be to ensure the government rediscovers it. If this proves too great a challenge, if too many ministers become drawn into defending the aspirations and limited horizons of Sir Humphrey, then the government will ossify and crumble. As Nye Bevan reminded Labour’s 1949 conference: ‘We have become so preoccupied with documents and papers that we sometimes fail to realise where we are going. These are merely the prosaic instruments of a masterly design.’ Governments that successfully renew are those that remember to be political.

Governments that forget to be political are reliant upon the bias of Whitehall to suggest policy that has already been watered down to the lowest common denominator of consensus in the departmental turf-wars that are the signature of the ‘independent’ British civil service. They are also vulnerable, as Attlee found over the unrealisable proposals for rearmament over Korea that prompted Bevan’s resignation, to being hoodwinked by ‘expert’ civil servants into taking unfortunate decisions. As Bevan said: ‘They know nothing about it. The great difficulty with the Treasury is that they think they move men about when they move pieces of paper about.’ Callaghan’s government was similarly burnt by flawed advice from Treasury officials which prompted the IMF crisis – the measures officials claimed were essential turned out to be unnecessary.

It has been said that politicians can campaign in poetry but must govern in prose. They must not let themselves become mere administrators, parroting the opaque phrases and meaningless jargon beloved of the civil service clerisy. It is this that sketch-writers like Simon Hoggart have parodied so effectively as ‘New Labour-speak’, when it is nothing of the kind. It is mandarin, with an infusion of management-speak. As Morrison told Labour’s 1948 conference, ‘the test’ for government would be ‘whether the ideals and purposes which were enshrined in legislation are to become a living reality, or whether human imperfections will convert the dream of the reformers into just another piece of bureaucratic routine.’

Morrison’s autobiography suggests: ‘When contemplating ventures and shaping policies, [a Labour politician] must ask himself the question: will it find an echo in the minds and hearts of the people?’ This is not a question a civil servant will ask. Indeed, many would think it morally wrong to do so, for that would be ‘political’. That is the job of politics and of political renewal. It is because the heavyweights of 1951 grasped this point, that, despite problems, Labour secured its highest ever share of the vote and more votes than the Conservatives in 1951. If Labour can achieve that at the next election, it will have renewed itself successfully.