A group which includes representatives of loyalist paramilitaries has written to the prime minister to withdraw its support for the Good Friday Agreement.