He will continue in his role until a replacement is found. Vindi Banga, senior independent director at M&S, will be leading the search for a new chairman.
Swannell said: “A year ago we chose Steve Rowe as our chief executive. Steve completed a thorough analysis of the business and developed a detailed plan to build a simpler and more relevant M&S.
“This plan is now underway and I feel that it is the right time for the business to look for a new chairman. It is a real privilege to chair this iconic company and I will continue to do so until my successor is in place.”
Swannell's departure comes months after senior executive Laura Wade-Gery, who oversaw the launch of M&S' website, announced she would not be returning from maternity leave.
M&S' share price was up 0.37 per cent at time of writing.
The retailer's new boss Steve Rowe has been shaking up the company's strategy this year, closing M&S stores internationally and planning to shut up shops across the UK.
It is expected that dozens of UK stores will shut of the next few years, and many of the remaining stores will be reorganised to focus more on M&S' food, as its clothing sales have been struggling.
The retailer has also changed the contracts for its staff, cutting benefits such as premium pay on Sundays and Bank Holidays. This, combined with pension changes for employees, landed M&S with a charge of at least £100m. MPs have been pressing Rowe to meet them to discuss the overhaul to staff benefits, but he refused the invitation.