Nearly half of UK farmers fear they will experience negative impacts on their business as a result of both Brexit and the pandemic, a new survey says.