ONLY a quarter of Scottish farmers are optimistic about the impact Brexit will have on the industry, a survey has found.