Appearing on Good Morning Scotland, the party's Brexit secretary said it could not back something that is "the wrong thing for Scotland".