A businessman has discussed how Brexit is increasing the price of food, calling for the UK to rejoin the single market to help the British economy.