Sister and brother in law both passed. House in both their names only is children responsible for mortgage ? No will. House in forclouser

No, their children are NOT responsible for the mortgage unless they cosigned or guaranteed it: debts are not inherited. They have no responsibiity to pay for their parents' debts/mortage except if they agreed to be responible. If your nephew is sued improperly, he will still have to defend against the suit (failing to defend means you lose by default, or automatically, even if the case was not one that ever should have been brought) but can and should countersue the lender for bringing a frivolous or baseless case (the exact name of a suit for frivolous litigation varies by state). 
The above said:
1) The lender can foreclose on the house if the mortgage is not paid. They can and probably should name your nephew in the foreclosure suit NOT for money--he does not owe--but because as someone who stands to inherit the house, he is entitled to notice of the lawsuit and a chance to try to assert or protect his interest in the home (such as by voluntarily paying off the mortgage, if he chooses). So while he can't be sued for money, he should be name in the foreclosure action to give him the chance to do something if he wants. (If he doesn't care about or isn't willing to pay for the house, he can officially give up any claims to it and let the foreclosure case proceed.)
2) While the children are not responsible for the mortgage (or other debts of their parents), the "estate," or what the parents left behind is: so the estate can be sued, and money which the children would otherwise inherit can be used to pay the mortgage or other debts.