More than 100 homes could replace a sprawling rural hostel after owners blamed Brexit for a dramatic fall-off in Eastern European workers coming to the Fens.