If I quit or am fired from my job can I take my customers and manufacturers with me?

If you have a non-competition or non-solicitation agreement, you clearly cannot do this--such agreements are enforceable.

If you don't have such an agreement, it's more of grey area. You can contact customers you have personally worked with (so ones you know and have had contact with) and tell them you are leaving and striking off on their own; you can even ask them if they'd like to go with you.

What you can't do is: 1) take any customer or business files with you--not even a copy you made for yourself; 2) use proprietary or confidential business information to court them--you can't say that your father's company is charging them $X for the services but  you'll only charge $Y if that is not fully public knowledge (e.g available on their website); and 3) you can't use information about the customers you have only from having worked at your father's to try to get them to go with you (e.g. you can't say, "I know that you have an aggressive risk tolerance, but my father is too conservative" since you'd only know about their risk profile from having worked at your father's brokerage).

Basically, just tell them you're leaving, that you're starting your own company, how to contact you, and invite them to reach out to you--that's what you can do.

Note that even if you only do what you can, your father may try to sue you anyway, hoping to scare you off, make you waste money, distract you, etc.--you have to be prepared that you may need to defend yourself. And if your profession or industry has any ethical guidelines for this situation, be sure to follow them, too.