The clash looked certain to end goalless and head into extra-time before substitute Dimitri Payet curled home a sumptuous 25-yard free-kick as West Ham set up a fourth round tussle with London rivals Chelsea.
“The most important thing is we won, and only with wins and clean sheets will we get the confidence back that we are missing,” said Bilic, whose side ended a four-match losing streak.
“I’m expecting is to make improvements. We have to earn a couple of good results and then the confidence will come back to us.”
Tottenham, meanwhile, cruised into round four where they will face Liverpool at Anfield after a crunching 5-0 victory over League One outfit Gillingham.
Playmaker Christian Eriksen opened the scoring with a 25-yard free-kick and added a second shortly after half time, while £18.5m striker Vincent Janssen notched his first for the club from the penalty spot and Josh Onomah and Erik Lamela also netted.
Holders Manchester City fielded a much-changed side but still overcame Swansea City 2-1 at the Liberty Stadium.
Crystal Palace crashed out of the competition after losing 2-0 at Southampton, while former Manchester United defender Paddy McNair netted twice to complete Sunderland’s comeback. The Black Cats beat QPR 2-1 at Loftus Road.
On-loan Chelsea striker Tammy Abraham scored a last-minute clincher as Bristol City dispatched Fulham 2-1 as another late goal, this time from Hull midfielder Markus Henriksen, settled the all-Premier League tie with Stoke.