The Labour leader has been warned he risks tying his party to the continuing damage of a Brexit process that isn't working.