The Institute of Directors and the West Midlands’ Tory mayor Andy Street have cited the UK’s relationship with Europe as a major impediment to business confidence and performance.