Your argument sounds reasonable to me, and I think it's worth getting into this with an attorney, if the bills are substantial enough and the doctor won't work something out with you.  This kind of case is usually difficult, but yours has the very helpful fact of the doctor chosing between two locations.  You should, of course, be prepared to have a new doctor take over your care if you do this, since you have no "right" to make him continue to treat you.