The issue isn't the litter itself; it's the waste.
The Centers for Disease Control report that there is a risk of exposure to toxoplasmosis, an infection caused by a parasite. This parasite can be found in cat feces, though the CDC reports that exposure is more likely from undercooked meat or gardening. (They don't say, but I assume the gardening risk is through animal droppings in general, not just cats.)
The CDC gives the following advice to mitigate the risk:
If you own a cat, have a non-pregnant person change the litter box every day. If there is no one else to change the litter box, wear gloves and wash hands with soap and running water after changing the litter.
Risk may be reduced if the litter box is changed every day.
Keep cats indoors.
Avoid adopting or handling stray cats.
Feed cats only canned or dried commercial cat food, never undercooked or raw meat.
Do not bring a new cat into your house that might have been an outdoor cat or might have been fed raw meat.
From what the CDC says, I think avoiding being in the same building as a litterbox would be an over-reaction. Continue to enjoy your cats, but to reduce the risk as much as possible, have somebody else do the box maintenance or wear gloves and wash up afterwards.
For more information and citations, see the linked article.