Jersey's "reputation is being tarnished" due to Brexit-related trade delays, the owner of an oyster company has said.