The hard-Brexit lobbying group, which produces research paid for by taxpayer-funded expenses, appears to be losing support.