After leaving the EU's single market with its free movement of people, goods, services and capital, he says: "We are fishing in a smaller pool."