Dave Ramsden said the 2016 referendum had contributed to a lower “speed limit” for the UK economy compared to other nations.