In the case of the flu, it depends on the strain of it. The specific type of disease that transmits from humans to animals and the other way around is called a Zoonotic Disease. According to PetMD:
The potential for zoonosis depends on a variety of factors, including
climate, geography, population density, sanitary conditions (or lack
thereof), grooming habits, and other factors.
Something like Swine Flu is a good example and the linked article provides many more.
In the case of ferrets and the flu, its possible to transmit generally human-only strains to them. This is why ferrets are commonly used in studies because their immune system is similar to our own. From Wikipedia:
Because they share many anatomical and physiological features with humans, ferrets are extensively used as experimental subjects in biomedical research, in fields such as virology, reproductive physiology, anatomy, endocrinology, and neuroscience.
To really answer your question: In general, no. But there are other factors (like the particular strain of a disease) that can cause you to transmit your disease to your animal. If you're unsure, I think you should attempt to distance yourself from the animal or at the very least try to avoid contaminating their food.