Should I be worried that my husband said this at work?

he should be "worshipping" you, his pregnant wife, not some dumb co worker. this is a problem. you need to talk to him.