The UK's finance minister has said he recognises that Brexit has brought trade barriers with the EU, but rejected calls for Britain to rejoin the single market, arguing that most obstacles can be removed over time.