While the eurosceptic elites wallow in the pleasure of the apparent unravelling of the 1985 Schengen agreement, they will be the first to complain as their cars are stopped and searched as they drive from one European country to another. Coaches coming into Britain are not required to disgorge their passengers at Dover and there would be July and August traffic queues back through Kent if the France decided to impose the kind of frontier controls that eurosceptics think are desirable. All that is being proposed are more spot checks, and an acknowledgement that with sudden surges of immigration more flexible procedures are needed.

But something deeper is going on. Firstly, the rise of immigrant populist identity politics is now beginning to alter European freedoms. Silvio Berlusconi had to respond to the anti-Tunisian xenophobia of the Italian Northern League – necessary allies in his shaky coalition – and hand out Schengen visas to 20,000 Tunisians fleeing the troubled north African region. This allowed the French-speaking Tunisians to go to France. There Nicolas Sarkozy panicked as the racist Front National accused him of allowing mass immigration into France. Sarkozy is terrified of Marine le Pen who has painted over the crude antisemitism of her father to make her party electable ahead of the spring 2012 elections.

Between the Northern League wanting to expel the Tunisian immigrants and the Front National saying Non to letting them into France there was no solution. ‘If we let them in they will steal our daily bread’ is how Auden described the 1930s response to immigrants. In 2011 there is a similar zero-sum approach to people movement from more and more European politicians. Maurice Glasman in Progress accused Labour ministers of telling ‘lies’ about immigration. This statement, of course, accords with the line from Iain Duncan Smith who today also blames Labour’s immigration policies for the failure of low-skilled British citizens to find work and, instead, turn to welfare. Others in UKIP, the BNP and rightwing press use even uglier language.

More considered statements, on display in this week’s Policy Network gathering of social democratic leaders in Oslo, argue that better worker and trade union rights in Britain would give confidence to native workers who would feel less threatened by foreigners if their unions were stronger. Yet those countries with the strongest social protection, full union and labour rights, such as Denmark, the Netherlands, Sweden, France and Germany are those where anti-immigrant passions are being incorporated into mainstream politics.

Denmark’s conservative government depends for its majority on an anti-immigrant rightist party. Denmark has announced it will bring back border controls as a gesture to the anti-immigrant right. Copenhagen has a bridge to Malmo which is like the Severn Bridge. Thousands of Swedes including immigrant workers willing to do all the low-pay work that sturdy Danes wouldn’t touch with a smelly herring cross every day. Copenhagen would stop working if real passport controls were imposed. Instead there will be ugly racial profiling. It won’t be much fun for British citizens whose skin colour might be closer to that of north Africans than to Marine le Pen as they try to travel and holiday in a post-Schengen Europe.

Will passport controls on people lead to calls for custom controls on goods? If low-wage workers are not wanted, how do small businesses stay alive? Of course in an ideal world every kitchen cleaner in an Asian restaurant or barista in Caffe Nero or Costa would be paid £12 an hour. But would we be willing to pay the price of a curry or coffee that would have to rise sharply to cover such wage costs?

The biggest single group of immigrants (in the sense of non-British) workers on the London Olympic Games site were from Ireland. France actually takes in more asylum seekers than any other EU nation. Yet French growth, as reported today, is much stronger than most of its neighbours.

Berlusconi emerges as the loser in the current row as the Northern League will now have to accept that the Tunisian refugees will not cross into France. The other big loser is the European Commission as EU national governments are setting up their own intergovernmental committee to decide a new stricter interpretation of the Schengen rules. These will have to be agreed by the European parliament so expect a ding-dong there as MEPs will not easily give up free movement of people as a core European gain. But this is part of the process of the slow renationalisation of the EU and part of the rise of xenophobic identity politics which are gradually moving into the mainstream of our political discourse.