The Brexit brigade is still going on about bendy bananas and the return of imperial measures. But it is a strategy born of ignorance or – worse – panic.