Brexit was prioritised over tackling Covid-19 in the early days of the outbreak, a former health minister has said, as the inquiry into the UK response to the pandemic gathers pace.