The consequences of leaving have long been clouded, including by the impact of Covid. Now the data is firming up—and it’s not good news.