I'm sure you probably wanted answers more geared towards cat/dog psychology, but the question is open-ended and I think this is an important addition.
I certainly wouldn't recommend using this method with small furries. We'll go with rats for the example, since I have the most experience with them, but this should be applicable to most small prey mammals in packs.
The primary form of misbehavior in rats is biting. The logical retaliation then is to 'bite' back, or pinch the rat sharply with your fingers. Depending on why the animal was aggressive in the first place, this could have several outcomes.
There are two primary reasons why a rat will bite. Firstly, dominance aggression; either the result of hormonal imbalance, injury, lack of socialization, or just plain old personality. The animal has to be in a pretty bad mindset to bite you, and when you 'bite' back? You're starting a fight. Small rodents live in a world of shifting social dynamics that oscillate between 'dominant' and 'submissive'. A dominant rat will regularly remind submissive ones of its strength, usually through forceful grooming, but occasionally through biting. Any retaliation is seen as a challenge and the challenger becomes a threat. Continuous retaliation will result in a fight. You absolutely do not want to get in a fight with a rat, the odds are never in your favor.
The secondary reason a rat will bite is fear, usually a result of lack of trust and handling, former abuse from humans, or psychological damage from previous fights with other rats. The rat fears you are a threat and will do its utmost to avoid confrontation, but will bite if cornered. And if you retaliate? You're attacking, and validating the animal's fear, making it harder to undo this damage later.
Now, all of this becomes moot if the animal is simply being naughty. Rats, being social animals, accept their human in to their pack, and expect you to behave as such. For example, when I am going through my rat cages to see who has eaten what food, I am often pushed or gently bitten as a way of saying 'hey, that's my food, hands off'. I immediately gently pinch the rat on the butt to remind him that he's out of line and that I can go through his food stash whenever I want. It works: the rat realizes he's in the wrong, and I can keep track of everybody's diet without getting bitten. None of this is aggressive or worrying behavior - the rats do it to each other 24/7, reminding each other who is more dominant, who was sleeping here first, who got hold of this banana chip first, etc. They push and shove and nip, and they aren't shocked or upset if their human reacts the same way - it's very hard to get through to small animals without using their own language.
In short, retaliation is not recommended as a method to discourage aggression. In other cases, you have to consider what message you are sending to the animal in question - what your actions translate to in the animal's language.