Is it legal for them to do this?

If it's a private school, absolutely.  Students are not owners of the campus, they are invited guests, and they are subject to the school's rules.  They also have the right to discipline students who break the rules.  I'm sure that there's a rule against trashing the dorms.  If it's a chronic problem like this, then it's far too expensive to expect the school to investigate who's responsible for each empty pizza box in the lounge.  I think the college was entirely within its right to do this, especially if other dorms don't have this problem.

I hope the fine gets the responsible majority of the students in your dorm to get as annoyed about this as you.  The proper target of that annoyance is the minority causing the problem, not the school.