Irish troops could return to the border in the event of a botched Brexit, the Irish prime minister has warned.