He added David Cameron "knows perfectly well" that the free movement of people was a founding principle of the EU and that "free movement of labour is absolutely essential if you have a single market".
When asked about politicians' failure to restrict immigration he added "you can't strictly control net migration in and out from anywhere actually".
Clarke argued that in the modern world you would have more foreigners and more foreign companies moving in and out of the UK.
Clarke's claims on the EU migrants and benefits is supported by a study seen by the Economist showing immigrants contribute more in taxes than they receive benefits.
Christian Dustmann and Tommaso Frattini, of the Centre for Research and Analysis of Migration, found European immigrants paid a staggering £2,610 more in taxes than they received in benefits each year between 2007 and 2011.
Clarke's comments came the same day as German newspaper Der Spiegel reported that Angela Merkel is becoming increasingly irked by the Conservatives' moves toward an increasingly eurosceptic position.
With less than three weeks to go before the Rochester and Strood by-election, the Conservatives have been eager spell out a tougher approach to immigration. David Cameron has promised to make free movement a red-line in any future EU renegotiation and crack down on benefits EU migrants.
If the Prime Minister plans to alter the free movement of people it would mark a "point of no return", Der Spiegel said.
However, the Tories' escalating rhetoric on immigration appears to be doing them little good in the polls, with Ukip scoring 48 points in the latest Rochester opinion survey, leading the Conservatives by 13 points. Labour's ratings in the constituency have dived to a poultry 16 per cent.
Recent events in the EU have not endeared the institution to the British people. Last week, Brussels presented the UK with a £1.7bn bill following calculations by the EU'statistical body Eurostat. Cameron has refused to pay the bill and told the Treasury to examine the statistics in "exhaustive detail".