More than 63 per cent of shareholders voted against the plan, which includes SFr6.68m (£4.7m) in 2012 compensation for chief executive Boris Collardi.
Because the vote is nonbinding, the pay plan will still go ahead.
“The board of directors will take the appropriate measures to work towards a positive vote at the next annual general meeting,” Julius Baer said in a statement.
Influential proxy advisory firm ISS had advised shareholders to vote down the pay plan, arguing it failed to give investors enough detail on bonuses, and did not defer shares awards for top managers, making them immediately available instead.
Baer’s biggest shareholders are mainly large institutional fund houses like MFS Investment Management, BlackRock and Harris Associates.
The vote comes a month after Swiss citizens voted in a referendum to introduce some of the world’s strictest controls on executive pay, including giving shareholders a binding vote on compensation at listed companies in future.
Rivals UBS and Credit Suisse, which both face shareholders votes in coming weeks, have drawn criticism in for their 2012 bonus plans.
Last year, a third of UBS investors voted against the bank’s pay scheme.