For all the reassurances about their status, the risk of being kicked out still haunts those without a British passport.