My grandfather, who died the same week as the Brexit vote, experienced the humiliation of crossing checkpoints just to see his family.