Many at the harbour in the Devon town are concerned that their industry has been sold short