The Government has admitted it rejected an offer for visa-free tours from the EU during Brexit negotiations, after days of insisting the offer was never made.