You have good cause to be able to get an Order forbidding any contact between the boyfriend and your children. If your ex-Wife were to violate that Order once it was entered, then it might be possible for the children to live with you. However, right now, if she were to protect the children by keeping him away from the children and not allowing any contact between them, then it would not affect any custodial rights. Now, if they are living together, that would be a different situation.
If you would like to discuss this matter further, as well as your potential rights and options, my office offers free initial telephone consultations. During the consultation, we can discuss your matter in detail. If you would like to schedule a telephone consultation, please contact my office.