It would be really helpful if you could somehow record these statements.  You can't tape your telephone calls with him without his consent, because that's illegal under Florida law, but if he's stupid enough to say something like this in a voicemail, you should save it, and the same for anything in an email.  Sometimes you can get someone like this to give you the "smoking gun" you need by playing dumb, "What do you mean?," and seeing if you can get an angry-enough response.

You're going to have to go to court on a motion, to get sole legal custody;  that will be more complicated than usual without a good address, and I'd advise having a lawyer handle this in any event.