The Labour MP had led the committee since 2007 and the group investigates areas such as crime, policing, drugs, immigration and extremism.
According to a report in the Sunday Mirror last weekend, the 59-year-old MP for Leicester East paid for the men to visit him at his flat in London just a week ago. It also claimed the MP offered to pay for drugs for the men.
"It is in the best interest of the home affairs Select committee that its important work can be conducted without any distractions whatsoever," Vaz said in a statement.
"I am genuinely sorry that recent events make it impossible for this to happen if I remain chair. I have always been passionate about select committees, having served as either chair or member for half of my time in parliament.
"The integrity of the select committee system matters to me. Those who hold others to account, must themselves be accountable."
He added: "I told the committee today of my decision to stand aside immediately from committee business, and my intention to resign. This is my decision, and mine alone, and my first consideration has been the effect of recent events on my family."
Vaz said he had recommended that Tim Loughton MP should chair proceedings "in the interim".