A majority of the British public would blame Boris Johnson if Brexit talks fail to reach a deal, a new poll has found.