Supermarket contracts and Brexit have been blamed for the lack of fruit and vegetables impacting British consumers.