“What worries me is the casual political vandalism. They really don’t seem to care," Tony Blair's former chief negotiator on Northern Ireland has said.