Instead, their top worry is the exemption of residential property from the new capital gains tax cuts, according to a survey by Amicus Property Finance.
Sixty-three per cent of landlord's said chancellor George Osborne's decision was their biggest challenge.
In second place is the changes to taxation on rental income, which abolished tax relief on mortgage interest. Alterations to taxes on maintenance and improvements came in third place; the changes mean landlords will only be able to claim for "wear and tear" costs for replacing furnishings when calculating taxable profits.
The impact of a possible Brexit on the UK economy came in tenth place.
John Jenkins, chief executive of Amicus, said: "The tax landscape has become a lot more hostile for landlords and it's no surprise that this dominates their list of concerns for the year ahead.
"In contrast, the prospect of interest rate rises, the threat of falling property prices and difficulties in accessing long term finance are less likely to be keeping landlords awake at night.
"Despite the new tax changes, we are seeing a sustained and growing appetite for property finance driven by the inability of some lenders to act sufficiently quickly to respond to demand. We're anticipating a strong 12 months for the professional buy to let market."