I'm going to speak from a perspective about CATS because that's what I know. Some of this may apply to dogs, but I'm not nearly as familiar with them.

Both of my girls are super anxious, and their anxieties have totally different sources and manifest in totally different ways. So we've tried a wide range of products aimed to reduce anxiety.

I wasn't able to find the ingredients list on the Royal Canin website (got a maintenance message), but Petfooddirect shows the following:

Chicken meal, corn, brewer’s rice, wheat gluten, corn gluten meal, wheat, natural flavors, powdered cellulose, dried chicory root, chicken fat, fish oil, calcium sulfate, salt, potassium chloride, , DL-methionine, vegetable oil, fructooligosaccharides, taurine, psyllium seed husk, choline chloride, vitamins [DL-alpha tocopherol acetate (source of vitamin E), inositol, , niacin supplement, L-ascorbyl-2-polyphosphate (source of vitamin C), D-calcium pantothenate, tripolyphosphate biotin, pyridoxine hydrochloride (vitamin B6), riboflavin supplement (vitamin B2), thiamine mononitrate (vitamin B1), vitamin A acetate, folic acid, vitamin B12 supplement, vitamin D3 supplement], L-lysine, dried hydrolyzed casein, L-tryptophan, trace minerals (zinc proteinate, zinc oxide, ferrous sulfate, manganese proteinate, copper proteinate, copper sulfate, manganous oxide, calcium iodate, sodium selenite), L-carnitine, rosemary extract, preserved with natural mixed tocopherols and citric acid. source

My first thought when reading this question was that this would be an herb-based anxiety reduction (there are a lot of those on the market). I pulled out Dr. Kidd's Guide to Herbal Cat Care (2000), and he suggests Oat, Catnip, Valerian, Chamomile, Lavender, Skullcap, St. John's Wort, or Kava Kava for anxiety. I don't see any of those (or their scientific names) in the ingredients list. The only herbs I see are chicory (my reference says that's for liver function) and rosemary extract, which he says

contains bioactive ingredients that help prevent the breakdown of the chemical acetylcholine in the brain. A deficiency in acetylcholine is believed to be a contributing factor in senility in general and Alzheimer's disease in particular. (p 47)

The most interesting thing in rereading the chapter on treating stress is the following:

Also, and perhaps most important, when Pet needs on of the herbs, take the same herb yourself. When you are calm and relaxed, your cat will follow suit. Just be sure to consult your physician or herbalist before taking a new herb, especially if you are already taking medication. (p 104)

This reminds us how important our own feelings are to our pets! There are a lot of anecdotal stories online of many remedies working for anxious pets. In some cases, it may be that the owner felt they found a solution and so they felt better, thus allowing the pet to feel more relaxed.

BUT, we still haven't really figured out the RC Calm food! One thing I noticed when looking at the ingredients is that it has a lot of protein for a grain-based commercial dry food (34%! by comparison the standard RC Indoor dry food is 27% protein). They claim that hydrolyzed milk protein and tryptophan (amino acid) help maintain your cat's emotional balance.

I was able to find a bunch of related studies performed on dogs. In the first study, selegiline is a prescription medication used to treat anxiety in dogs. Zylkene is a hydrolyzed milk protein dietary supplement. They appear to be roughly equivalent in effectiveness.

• Effects of alpha-casozepine (Zylkene) versus selegiline hydrochloride (Selgian, Anipryl) on anxiety disorders in dogs. Journal of Veterinary Behavior: Clinical Applications and Research Volume 2, Issue 5 , Pages 175-183, September 2007 abstract

The second study used measured cortisol values as well as behavioral observations, but noted a high level of variation before concluding that the hydrolyzed milk protein may be useful.

• Efficacy of a diet containing caseinate hydrolysate on signs of stress in dogs. Journal of Veterinary Behavior: Clinical Applications and Research Volume 5, Issue 6, November–December 2010, Pages 309–317 abstract

But, finally, I found a study with a negative result (this is actually uncommon with publication bias) about the effects of serotonin precursor tryptophan.

• Dietary tryptophan supplementation in privately owned mildly anxious dogs. Applied Animal Behaviour Science Volume 121, Issues 3–4, December 2009, Pages 197–205 abstract

It appears that there is some evidence that hydrolyzed milk protein MAY help calm anxiety when fed several weeks ahead of the anxiety-producing event.

To address the second part of your question about animals getting afraid of the "keep calm" stuff - I would be surprised to see this reaction to food or a particular supplement/medication (the process of BEING medicated, certainly - we switched our human-anxious cat to transdermal anti-anxiety medication that we could just rub on her ears).

It could happen if you were trying a devices like a tight wrapped shirt that are supposed to calm anxiety. I did buy one of those for the cats, and even when I put it on the cats that are much easier to "read" (i.e. figure out how they're feeling) than the anxious cat, they didn't seem scared of it, but more confused by the feeling of something on their back. Every animal is different, but mine weren't scared of it, but I could see it potentially happening.