Termination of parental rights

No.  He will have parental rights until your son turns 18, or you go to court and have his rights terminated, for example if you get married and your husband wants to adopt your son as his own.  At that point, the biological father's complete non-involvement will make termination of his rights (which has to happen to allow the adoption) a no-brainer for the court, as long as he gets notice.

That last part is important, and you might want to make some effort to keep track of the man.  Also, I'd recommend filing for child support, if you haven't already.  Your son has the right to the man's financial support, even if he isn't going to show his face;  if he doesn't pay, it's one more reason for the court to terminate his rights, when the time comes.