There are a lot of factors that affect the answer here, and each will affect an individual cat/human pair in different amounts.
Harness Training
The first step to being able to exercise your cat outside responsibly is to train him to accept the harness and leash. Often when a cat feels an unfamiliar weight on his back, he will roll over to escape it, and when the harness does not come off he will get confused and distressed. Properly training the cat to accept a harness (instead of just sticking it on him and seeing what happens) is a lengthy process that most cat owners are not familiar with and do not take the time to do properly.
Territorial Familiarity
Cats are territorial, and an indoor cat will view your home as his territory. He's comfortable there because he knows all of the hiding spots, the food/water sources, and the safe places to eliminate, etc.
When you take your cat to a new place (outdoors) you are forcing your cat out of his territory so he's already slightly uncomfortable. Additionally, if the area has any feral/outdoor cats, the area may be scent marked as someone else's territory, further putting him on edge.
You see this behavior some with dogs as they scent mark the route of their walk, but when two dogs meet they may be friends or competition, and they also have a much longer history of looking to humans for socialization cues. Cats are more likely to view an intruder as competition.
Territory Structure
Cats view their surroundings in 3 dimensions, where people (and dogs) generally just look in 2 dimensions. Climbing helps cats feel safe (they can observe their surroundings without beeing seen), but a walk with a person forces them to only use the more exposed ground level. This can be pretty uncomfortable for a cat.
Elimination
Cats are often trained to use a litter box (as predators they will bury their waste to avoid alerting prey to their presence) and so can eliminate inside. Dogs, on the other hand, are rarely trained to eliminate inside and so must have regular outside time.
Owner Selection
Cats are generally seen as more independant and requiring less intensive care than dogs, so they tend to attract owners who are looking for a pet that does not require regular walks.