Can we not rent to someone who has a child?

The housing discrimination laws do not, as you suggest, apply to buildings or homes with four or fewer units which the owner occupies--which would be your situation. So you should  not have to contend with the anti-discrimination laws and should be able to freely decide who to rent to.

That said, assuming you do not have small children of your own, an appropriate response would be to cite the risk of liability: explain that your home is not child-proofed or child-friendly, since you don't have children that age yourself, and you cannot take the chance that a small child would be injured by access to medication, tools, cleaning supplies, etc.