I think refusing the inspector is a very bad idea.  If the building department suspects unpermitted work, your refusal sends a signal that will be interpreted as an admission of guilt on your part, and it would be a good bet that the inspector would be back a day or two later, with a court order in hand and a police officer backing him up.

If all the work was done under the permits, and you don't know of any problems, allow the inspection.  There's probably no other way to get the certificate of occupancy anyway.  If you do know of problems, you need to talk to your attorney as soon as possible.