Britain has got poorer because of Brexit, former chancellor Philip Hammond has admitted.