“I’ve always argued that the only way you can get frictionless trade is to actually be in the Single Market.”