The company's shares plunged by up to 20 per cent in early trading.
The firm, which had seen $4.4bn of subscriptions in the first half of the year after two years of clients pulling out their money, said total assets under management fell to $65bn at end-September from $71bn at the end of June.
Analysts at Singer Capital Markets had been expecting net outflows of $200 million in the three months to end-September.
"Looking ahead, we are assuming that investor appetite will be generally suppressed for the remainder of the year," said chief executive Peter Clarke in a statement.
Man's total assets under management have rebounded from below $40bn last year, helped by last year's $1.6bn acquisition of GLG and a bumper Japan launch of a version of its flagship AHL fund.