That's the verdict of a new briefing paper from the Adam Smith Institute that aims to separate fact from fiction when it comes to the political rhetoric surrounding non-doms.
In a speech at the University of Warwick on Wednesday, the Labour leader claimed there were around 116,000 non-doms in the UK. According to the ASI, that isn't the whole story.
The 116,000 figure accounts for those people who filed a self-assessment form and ticked the non-dom box. However, the ASI reckons there are around one million students and workers in the UK who don't have indefinite leave to remain in the country and are therefore, by definition, non-domiciled.
Labour have targeted the non-dom status as a 200-year old anachronism that sets the UK apart from most of the world. But again, this claim is questionable to say the least, according to the ASI. Japan, China, Belgium, Ireland and Australia all have systems that allow for workers who aren't permanent residents to be taxed only their local income for a period of time.
Far from being a radical reform of a controversial part of the tax system, Miliband's proposal is nothing new, according to the ASI. The Labour leader wants to crack down on those who are "permanently settled" in the UK but are not taxed in the same way as the rest of the population.
But if these people are actually settled here permanently then they are legally domiciled in the UK avoiding any issue of fairness and equal treatment. Rather, what Miliband is attempting to do is clarify and restrict the eligibility for non-dom status.
Perhaps the most contested claim of the policy is that it will raise "hundreds of millions" for the Treasury. Labour claim this figure stems from "independent" experts, however, one of these experts turned out to be a member of the Labour party.
International law firm Pinsent Masons has warned “if UK taxes on non-doms are increased then at the very least they are very likely to review their exposure to UK taxes and how their assets are held."
This warning is echoed by the ASI paper.
Citing evidence from Denmark as well as a larger literature the ASI argue that tax systems need to take into account the mobility of the country's top talent. In Denmark the top one per cent of inventors where were found to be "significantly affected" by top tax rates and more likely to relocate.
“Cracking down on non-doms may sound nice but proposals that sound nice aren't always good policy," said the ASI's head of the research Ben Southwood.
"Miliband’s scheme risks making both the UK and the Treasury poorer and less fair," he added.