The former chancellor George Osborne’s family wallpaper business has said that Brexit added £400,000 to its costs, as the company’s sales slid more than 16% during the Covid-19 pandemic.