Brexit is being blamed for making the building of affordable homes on a Derbyshire housing site ‘unviable’.