It is possible that this could be considered a conflict if any information was gained during your consultation that could potentially disadvantage you in any way during the divorce proceedings. Have you brought it to the attorney's attention? You can also file a motion to disqualify if they won't voluntarily remove themselves if you feel it is a conflict of interest.
If you would like to discuss this matter further, as well as your potential rights and options, please contact my office for a free initial telephone consultation.