But the public don't seem to mind too much, according to a new poll.
YouGov found that British people tend to say Cameron has not really done anything wrong in terms of his tax and investments.
Some 45 per cent say Cameron didn't really do anything wrong, while 35 per cent say he has.
Labour supporters are, as you might expect, quicker to criticise the Prime Minister. But even 28 per cent of this group don't think he's done anything wrong.
That's despite calls to resign by some sections of the public, and a number of MPs.
Cameron had shares in his father's company, but sold them before he became Prime Minister, and paid full tax on what had been earned, he said earlier this month.
However, Cameron is less trusted on tax avoidance than Jeremy Corbyn, and Cameron's lead over Corbyn on who would make the best Prime Minister has narrowed from 26 points to seven points. Yet that's largely related to the EU referendum, YouGov says.
Meanwhile there is strong support for party leaders and ministers publishing their tax returns.