If a man gives up custody of a child at the time of divorce as part of the agreement, is that child still considered next of kin upon the man's death?

Unless the child was adopted by another man, he or she is still considered a descendant of the father.  Giving up custody is not the same as relinquishing or waiving parental rights.  Without some kind of unusual circumstance, the child is still a child of the birth father.