Is there a form my mother's legal husband can sign to waive his rights to her estate?

A qualified disclaimer is the required document.  It has the impact as if he predeceased the decedent.  The will and/or state law will be enforced and followed as if he in fact predeceased the decedent.

Yes.  It is called a Disclaimer.  By signing the disclaimer he will be treated as if he died before she did.  His share will then go to her other heirs, or the person the will states will take if he doesn't.