All cats have an instinct to hide their waste from prey and other predators. Domestic cats are (mostly) descended from African wildcats, which further refined that instinct to burying their waste in sand. Kitty litter is specifically designed to mimic sand and thus trigger those instincts.
Note that if there is something wrong with the litter box, pain from a UTI or other stress in the cat's life, they may find other places to go, but those places will still usually be fairly soft and loose (e.g. clothing or bedding) to fulfill the core hiding instinct. Unfortunately, those are usually also the places that are most difficult to clean and most upsetting to humans.
Other species of cats (including some used in "hybrid" breeds) prefer to hide their waste in water, dirt, leaves or other materials based on their habitat, though some can be trained to use litter. Likewise, some housecats can be trained to use a toilet. However, in both cases, the training is difficult and fragile because you're working against their instincts rather than with them.