Boris Johnson spent more than 200,000 on the three water cannon, which were blocked from being used on the streets of London by home secretary Theresa May.
The former London mayor came under fire for his decision at the time, which he's not being made to forget now.
Launching her bid for the Conservative leadership, May said: "Boris negotiated in Europe. I seem to remember last time he did a deal with the Germans, he came back with three nearly-new water cannon."
Speaking at the State of London debate on Thursday night, Khan said: “One of the deals Boris Johnson managed to do with the Germans was to buy three water cannon. What I’m going to do is sell them and use the money for youth services.
“This shows the inability of Boris Johnson to get a good deal: second hand, paid almost a quarter of a million pounds. We want to get rid of them. We are paying for storage facilities for these water cannon – that beggars belief," the Guardian reported Khan to have said.
Water cannon have not been ruled out for use in Northern Ireland, and in fact were deployed in Belfast last July against loyalist demonstrators during the Orangemen's annual celebrations for the commemoration of the Battle of the Boyne.