The "Festival of Brexit" has attracted just 238,000 visitors in the last four months – a fraction of their expected target.