Being around cats while pregnant: should I avoid them
Being around cats (or other animals) is perfectly safe while pregnant, but their litter boxes can present a problem. It's called toxoplasmosis and can also be contracted from handling or consuming raw or undercooked meat and also from handling soil without gloves. So the litter box is not the only place that the bacteria is present.
Best practice is to get somebody else to handle the litter box while you're pregnant. But if that's not possible, wear a mask and gloves while changing the box, wear gloves if you're handling soil in the garden, and wash and cook meat thoroughly while pregnant.
It can be done safely though, like you said.