Can my mother or I have my sister's friends arrested for trespass when they sneak into the house at night?

I doubt it, at least at the beginning.  You'd have to catch them, and order them to leave and not come back, and then either they'd not leave, or come back, and be caught there again, at the very least, before they could be arrested as defiant trespassers.

There's another problem, of course, because they're being invited by your sister, and she lives there, too.  It sounds to me like the only real solution to this problem is kicking your sister out, at least until she's ready to live by the house rules.  This is really too dangerous to ignore.