When you settle a case does PIP have to be paid back from the settlement money?

Yes, that is correct--the amount you collect under PIP is offset against your settlement (or, if it had gone to trial and you won, your judgment). The reason you need to do this is simple: it's the law. The reasoning behind it--say that you receive $20k in PIP medical benefits which pay for your medical costs. Say you win a judgment or get a settlement of $100k that includes the $20k medical. If the PIP benefits are not offset vs. the settlement's medical portion, you would be double dipping--receiving $40k of compensation ($20k from PIP, $20k from the settlement) for $20k of medical costs.