A Littlehampton company has had to let go dozens of employees after losing almost all of its business to Brexit.