The head of a major seafood company in the Scottish Borders says Brexit has "more or less finished" his business.