More than half of Brits think that Brexit is "going badly," according to new polling — and the number of people with that opinion is growing.