And now the taxman's under pressure from MPs on taxing the super-rich. The Public Accounts Committee has said HMRC needed to crack down on the perception there was "one set of rules for the rich and another for everyone else" and needed to consider what it can do to deter the very wealthy from "bending or breaking" the law.
In 2009, HMRC set up a specialist unit to deal with high net worth individuals to try and better assess how they managed their affairs.
But since the unit was set up, the amount of income tax paid by high net worth individuals has fallen by £1bn - despite income tax receipts from the public as a whole rising by £23bn for the same period. And the number of individuals with more than £20m of assets has increased from 5,900 to 6,500 between 2009 and 2015.
And the committee questioned the development of appointing wealthy individuals with a personal customer relationship manager to make sure they paid the right amount of tax, saying that means "they get help with their tax affairs that is not available to other taxpayers".
The taxman says the rich get "additional scrutiny" as opposed to special treatment and it had secured "an adidtional £2.5bn from the very wealthiest since 2010".
The committee said it was "alarming" that one in three of the wealthiest people in the UK - with assets of more than £20m - were under investigation by the tax authority.
Chair of the cross-party committee Meg Hillier said: "HMRC's claims about the success of its strategy to deal with the very wealthy just don't stack up."
She added: "If the public are to have faith in the tax system then it must be seen to have fairness at its heart. It also needs to work properly. In our view, HMRC is failing on both counts."
The report also raised concern about tax evasion in the football industry and noted the "misuse" of image rights in order to reduce tax liabilities.
The committee said 43 players, 12 clubs and eight agents were currently the subjects of "open inquiries" by HMRC and in some cases, clubs were failing to cooperate fully with the tax inspectors.
Tax rules in place allow income from image rights to be treated as a separate revenue stream for tax purposes, so taxpayers who felt their image had a market value could create a firm to receive payments for those rights.
The committee said the government needed to take "urgent action" to address what it felt was the abuse of the system.