MP Steve Baker has apologised for the "pain and difficulty" Brexit has caused to Wycombe residents.