Just five people have been sent back this year, against 289 last year when UK was part of EU returns deal.