A family of three have been left £3,000 out of pocket due to a little known Brexit passport rule that resulted in them being unable to travel abroad.