It is hard to dispute the idea that London did not prioritise finding a solution to the Brexit problems facing Northern Ireland, a former top Stormont civil servant has said.