A first-half goal from Jermaine Beckford was enough to inflict Ferguson's first FA Cup defeat to lower league opposition, and his maiden exit at this stage of the competition, in 23 years at the Old Trafford helm.
It earned Leeds a fourth-round tie against more Premier League heavyweights in Tottenham, and prompted Ferguson to threaten his flops with the axe ahead of Wednesday's Carling Cup semi-final with Manchester City.
But the Scot, who was fined and banned for two games in November for questioning referee Alan Wiley's fitness, also turned his anger on Foy, and what the United manager saw as insufficient injury time.
"The referee gave five minutes. That is an insult to the game and the players out there," said Ferguson.
"Leeds had far bigger appetite than us. Human beings can always surprise you but we didn't expect that
"We have to get this result out of our system as quickly as possible. We have a semi-final on Wednesday and a lot of these players won't be playing."
Leeds manager Simon Grayson hailed a memorable victory - their first at the home of their bitter rivals since 1981.
"I am very proud of my team and what they've done today," said Grayson. "We worked very hard and I feel we deserved this win. We knew we could hurt them today and this
signalled how well we've been playing all season."
Ferguson made seven changes but still fielded a strong side that included Wayne Rooney and Dimitar Berbatov in attack.
But it was the sought-after Beckford who demonstrated his marksmanship in the 19th minute, when he ran onto Johnny Howson's pass and slotted past Tomasz Kuszczak.
Rooney saw an effort cleared off the line before the break, but he and substitute Michael Owen both missed chances after the interval as the League One leaders clung on for a famous win.