The stuff in the gravel can be a source of anaerobic bacteria and ammonia/nitrogen if you let it build up too high, so it's definitely a good idea to get rid of. It'll break down somewhat on its own, but nearly all tanks with any animals will produce waste faster than they eliminate it.
I would recommend using a gravel siphon rather than just randomly stirring though: that way you're removing the waste immediately instead of mixing it into the water column. (And doing your water change.) You'd need to siphon out any of the denser solids/particulates anyway, since they're likely to settle out before any mechanical filtration can grab them.
Cleaning the substrate is especially important if you have an undergravel filter. Those will pull waste deep into the gravel, and that waste will start to block the flow of water as it builds up. If it gets too bad, the denitrifying bacteria in those spots will die back due to a lack of food (the ammonia and nitrites that you want them to consume) and oxygen.