If my son's dad has been in and out if jail and there is no custody agreement between us, am I breaking any laws by not allowing him to visit?

If there is no court order in place, you have no obligation to allow him to visit, but there is nothing stopping him from asking the court to grant visitation.  If you are thinking of allowing him access to the child but want to have visits supervised or limited, you can file for a parenting plan in Washington, probably called something different in other states.