Dave Ramsden tells MPs referendum fallout also contributed to lower speed limit for growth of UK economy.