Is my company legally bound to remove references to a client from our website?

Assuming that he owns the intellectual property--e.g. copyright and trademark in his company's name, logo, etc., as well as the design of the website--then yes, he can order you to not use or reproduce any of those elements. The owner of intellectual property controls its reproduction and use.

However, what you can still do is, without using his logo or particular typography, to simply factually list his website and its address as being one of your clients and a sample of your work product: the right to control intellectual property does not prevent others from listing facts, such as that they had done work for him or created something for him. Potential clients can then, if they choose, go to his website to see for themself what it looks like.