You said "getting a divorce" - so the custody hasn't been finally decided. It would matter a great deal as to "why" he was deemed unfit. But since he obviously still has some physical custody or he wouldn't be seeing them, it doesn't sound like he was deemed unfit. You should talk to your attorney and make sure he/she knows about this situation.
If you don't have an attorney, I would strongly urge you to get one, for the kids sake.