A man was left bloodied with two black eyes after an attack he claims was sparked by him wearing an anti-Brexit T-shirt.