Can a landlord renting an illegal apartment sue for back rent?

No.  A contract for an illegal service or property can't be enforced in court.  It doesn't matter whether it's someone who's paid but didn't get what was paid for, or, as in this case, someone who provided something that didn't get paid for.

Find out what's what, before you get into this situation.  If the apartment isn't legal, it's probably because of a local law, an ordinance or regulation, and you should check with your town's housing official or department, if you think that you're living in an illegal apartment, or (better timing) just left one.