Some of the biggest concerns may be him changing the locks if you are still living together, cleaning out the bank account and gettting the kids back from him upon your return.  This is not desertion, however.  You may want to consult an attorney in your area.

If he's agreed to it, I wouldn't worry too much.  It's usually only considered desertion if you move out, or leave unexpectedly and stay away for a while.  A 5-day vacation isn't desertion.

I can't promise you that it won't come up. Sometimes, incredibly petty accusations get made in a divorce, and the courts are not impressed;  sometimes, people make up accusations in a divorce case, and most of the time the courts aren't fooled.  Your best defense is a good lawyer.