How do I go about getting visitation taken away from my child's father?

I am a lawyer in CT and practice in this area of the law.  The next time this man punches a hole in the wall you need to call the police to arrest him.  you should also make a motion in court for a protective order as a result of his conduct, which threatens your safety and son's safety.  You also need to file a motion in family court to order him to pay child support.  in most state's, judges will impose jail time on fathers that fail to pay child support.  You can go to the clerk's office to learn how to file this sort of motion unless you know what to do already.  I would also make a motion for sole custody and point out the violence and temper the father has.  however, unless you have a police report or record of this, the court is not likely going to terminate his custody rights.