Brexit has added to the UK's economic woes by lowering the value of the pound and contributing to price rises, an ex-Bank of England governor has said.