A soft Brexit scenario would minimise the damage, but would still result in job losses, according to forecasts.