Confused truckers trying to find a huge Brexit lorry park are getting lost in a nearby village - making houses shake and churning up grass verges.