This is not cruel. Unfortunately, you aren't going to find a consensus on when it's right to blanket a horse, because it depends so much on your horse's situation.
Personally, I have 5 horses in their prime years of life, that have been on the property for years and are allowed to grow a full natural coat. I tend to blanket mine below 40deg Fahrenheit when it's going to rain and below 20deg Fahrenheit if it's going to be dry.
People from different areas will tell you different things. For instance, my family is from South Carolina, but my uncle was dating a girl that had a place in New Orleans. He flew down to ride horses with her. It was about 70deg and the horse had worked up a little bit of sweat and was dirty, so he went to hose it off and bathe it. He was half way through when the owner showed up and started flipping out about how he was going to kill her horse, bathing it in the freezing cold like that.
One thing to consider is that a horse is not a human, so when a person is cold, doesn't mean a horse is cold. The have a large body mass designed to help keep them warm in the winter. A horse stands the hair on their body up when they're cold, just like when you get goose bumps. This creates an insulation layer and traps air, so it's like putting on a blanket for them. Their body also has mechanisms for pumping blood to their extremities to keep them from being cold. When you put on a blanket, you flatten the hair on their body, you can also cause their core to think it's warm and stop it from warming the legs like it should. Also, when it's too warm and moist under the blanket, you get rain rot. My mother had a horse that she insisted on blanketing when I thought it was too warm. He ended up getting rain rot over most of his torso, causing his hair to fall out and him to be colder in the long run, not to mention having to treat it.
So there are a few things to keep in mind. One is, has your horse been on the property since last fall? This would allow his body to accept the changes and grow a winter coat. Another is, is your horse clipped? If you remove his winter coat, you have to provide one. Is it going to rain? Rain flattens the hair and removes the insulation quality like a blanket does as well as causing evaporative cooling. I've been told it's colder than snow sitting on them as snow has it's own insulating properties and the hair keeps it off their skin, but I haven't research that at all. Is your horse really old or young? Older horses have it harder than younger horses, but both lack the mechanisms a horse in his prime have to stay warm.
As a side note, I have had two horses since the day they were born. Their first winter, they grew coats so thick I could put my hand on their butts and have them almost disappear. Now they don't grow much of a winter coat at all. So, they have the ability to grow it, but their body doesn't, because it doesn't feel it's necessary.
Does your horse have some kind of shelter that blocks rain from above and wind from the sides? We have a tight copes of cedars that do that for our. That could cause you to need to blanket more frequently. Also, consider if you're having particularly nasty weather. A horse may become adapted to nasty weather if it's nasty all the time, but we recently had it go from the mid-20's on a consistent basis to dropping down to 4deg for a couple of nights. If the weather goes well outside your normal range, then the horse may need a blanket as he won't spontaneously grow his coat in time to adjust for this.
Finally, it boils down to observing your horse. From the signs that I know, if your horse's ears feel cold or you physically see him shivering, then he needs a blanket. The ears are part of his extremities and also don't have a large blood flow, so they'd get cold before the rest of the body. Keep in mind that hay will play a much larger role in keeping your horse warm than most other things. The act of digestion causes them to produce heat from the inside out. If the weather is supper nasty, I provide hay 24/7, usually in the form of a round bale. Grain doesn't do a lot. People who say grain makes a horse 'hot' are talking about energy, not physical warmth. Grains role is two fold at any time of the year. One, it provides minerals and nutrients (if fed in the proper amounts) that they horse may not be getting in their normal environment) and two, it's providing calories. In the winter, the calories put on fat that is an insulating layer as well as something their body can burn to stay warm. My horse are fat already, so I don't provide that much grain.
In the end, putting your horse in the barn and trying to baby them can cause more harm than good. I tend to ere on the side of letting nature handle it, but keep an eye on the situation. Good luck.