The Dow Jones industrial average fell 50.57 points, or 0.28 per cent, to 17,867.58, the S&P 500 lost 7.48 points, or 0.35 per cent, to 2,102.31 and the Nasdaq Composite dropped 2.65 points, or 0.05 per cent, to 5,142.48.
The fall snapped a run of five straight days of gains for the index, with shares of Chevron down 1.4 per cent at $96.77 and Exxon Mobile down one per cent at $85.98.
Stocks added to losses after Yellen’s comments, which caused investors to reset their expectations of a December rate hike above 60 per cent.
Yellen said December remains a “live possibility” for a rate increase, and William Dudley, the president of the New York Fed and a permanent voting member of the Fed’s policy panel, said later that he would “completely agree” with Yellen.
Time Warner, down 6.6 per cent at $72.20, weighed on the S&P 500 the most after the company said ratings for its “key” domestic entertainment networks have dropped more than anticipated.
Shares of Twenty-First Century Fox dropped 5.2 per cent to $29.65 after it reported lower-than-expected quarterly revenue.
Other media stocks such as Disney, Viacom and Discovery also fell.
Health insurers also slid, with UnitedHealth, down 2.6 per cent at $114.64, the biggest drag on the Dow.
Groupon slumped 26.3 per cent to $2.97 after it forecast weak fourth-quarter and 2016 revenue.