Fashion retailer Joules has warned it is considering increasing prices for consumers as it revealed that Brexit had disrupted exports and could cost it an extra £1m per year.