A former Bank of England policymaker suggested there may not be a need for an austerity budget had it not been for Brexit.