Seven years after the EU Referendum, Brexit is finding it increasingly difficult to retain its 2016 supporters let alone recruit new ones