Brexit preparations "sucked all the blood" out of planning for a severe disease outbreak - despite this being considered a serious threat for many years - a government adviser has claimed.