Former Bank of England governor Mark Carney has doubled down on his claims Brexit has taken a toll on the pound and sparked higher inflation.