Keith Brown challenged pro-Union parties at Holyrood to explain how they would reverse "the dismal downwards trend" caused by the UK leaving the EU.