Brexit secretary David Davis said this week that no one ever said leaving the EU would be "simple or easy". Except all these times that people did.