What are my rights if my ex is being charged with rape and 2 counts of unlawful sex with a minor but is demanding to see our children?

Since you mention that he is an ex-- that would imply that you were divorced or orders were entered at some point that gave him rights to visit the children.  If there is an order in place that provides him access to the children, then you need to do two things.  One is to file for a modification of your current orders to restrict his access while these charges are pending.  Second, you should also contact your state's child welfare agency to let them know that he is attempting access and you are concerned because of the nature of his charges.

If there are no orders in place because you and he were never married, then you do not have to allow him access to the children.  The only time that you get in trouble is when there is a court order and you violate that order.  The word of caution, however, is that if there is not an order in place and he is the father on their birth certificate, then he could potentially go to their school or daycare and check them out because there is not an order restricting his rights either.  The safest option would be to hire a family law attorney to file a custody suit for injunctive relief to prevent him from having any contact or unsupervised contact with your children until the issues associated with his rape charges are resolved.