I'm in my first trimester of pregnancy, and my place of work wants me to get a TB test, can I refuse because of health concern and pregnancy?

It appears mandatory testing of pregnant women for syphillis is required.  There is compulsory testing for TB for the following individuals:

students, school personnel, as well as for those who work in hospitals, clinics and other medical associated facilities. If you are concerned about the well-being of your fetus, and your own well-being, contact your OBGYN and discuss the employer's request.

Most states have ant-discrimination laws, including at the federal level, regarding pregnant women.   Also consider talking with the EEOC (Equal Employment Opportunity Commission) regarding the Pregnancy Discrimination Act.