Yes, there are general signs related to their stool as you noted, including diarrhea, blood, or constipation. For other signs, you should be looking for:
- Unexplained weight loss
- Vomiting
- Coughing and other breathing difficulties
In general, though, being able to see them poop (or the outcome of it) is the most effective way to detect the presence of certain worms as not all symptoms may apply to varieties of worms. For example, the stool is important for detecting roundworms and hookworms. The weight loss and vomiting would more likely signal a tapeworm and coughing/breathing issues for lungworm.
If you're really concerned, take him into the vet for an examination. The risk is higher for outdoor cats who are more likely to be exposed to fleas and other parasites.