Scottish fishermen who took their trucks on a protest convoy to London this week have said they did not back Brexit - but are now paying the price.