A major daffodil grower in the fens say they only have a fifth of their usual number of pickers this year because of Brexit.