Of all the stupid losses Britain faces from Brexit, few can match the departure of well-paid bankers from London.