Public concern about immigration to the UK may be down, but it has always been a surrogate question to living standards.