The hits to Britain's economy from Brexit are probably appearing faster than expected, Bank of England Deputy Governor Ben Broadbent said on Thursday.