Why is he doing this?
It is hard to prevent a dog from rummaging at food scraps. Most dogs love their food, much of the training advice you'll see around recommends using food treats as rewards. Dog's also love to scavenge, their sense of smell is far superior to our and poking their noses into new places is like sightseeing for us. The problem is one day he may eat something poisonous.
What can I do to break this habit?
Your dog is older, I don't know his breed, but it is always harder to break an older dog of bad habits, particularly in this type of case; as it's not really bad habit, as perfectly natural. A bit like teaching a toddler not to take candy left on the kitchen counter.
So, you've got a battle on your hands.
(1) I really don't like taking my dog out
Trying to train your dog to achieve this will take a great deal of effort and time; including the need for some negative reinforcement, which is not the best thing for a ten year old dog, who is otherwise, basically well behaved. Such training might confuse him, as to why he's suddenly being scolded for something that has been OK for how long? Possibly years. You need to ask yourself; are you up for this?
Given your motivation to stop this behaviour is to enjoy the time spent with your dog, I would be inclined to suggest that training him out of eating (what he consider treats) could be more of a struggle than the one you're having now and the idea is to help you and your dog enjoy your walks. It might be easier to drive and take him to an area where there isn't food scraps, but I understand this may not be possible.
So, this brings us to...
(2) Positive reinforcement
Given you are finding your walks stressful, I strongly recommend you don't correct him for his unwanted behavior, rather just use reward, as it sounds like the situation between you both might be a little tense and it's best to improve the fun between you and your dog. Down the track, some minor scolding using voice control can be used to correct the occasional slip, but from the outset, keep it all positive.
This is going to require finding the treats your dog goes crazy for. You have the competition of fish and rice with all sorts of flavors, so it's got to be treats that your dog really loves.
To begin with just go for short walks, even limit is to 100 metres if your dog's behavior is too severe. It's vital to set your dog up to succeed. As he improves increase your walking distance.
Start by showing him you have a bag of his favorite treats, give him a sample when you put him on the lead. Carry the treats in a
bum bag or
fanny pack facing your front, make sure he knows where they are. Also carry a treat in your hand. As you give him one replace it with another.
As you go out, you're are going to have to encourage him vocally with happy
hey look at this! type voice calls whenever he goes to sniff and show him his treat. Immediately reward him with his favorite treat, with lots of verbal praise, so he associates the treat with verbal praise. Repeat. Use a catchphrase; I wouldn't use the
leave command, as I am basing this solely on positive reinforcement. Pick a word you are happy with, even his name and stick to it
Rover!, happy and excited..(look at this Rover!)
This process has several benefits, it can become a game, so you and your dog can have fun, and it's serves to distract him, plus the reinforcement.
When your dog has totally grasped the notion that he will receive the treat whenever you call
Rover! in this voice, you don't reward with him a treat every time, but it is vital to keep up that verbal praise and replace the treat with a quick pat on the head. I haven't included pats in the previous step, as they are no competition for fish scraps.
This process will take some time, and I would suggest always having some treats on hand when you're out together, as it's important to maintain your dog's good behavior.
This post also gives some details with assistance with leash training.
There are other things you can do to discourage him from taking food from the ground. These techniques are frequently used to train guard dogs to only accept food from certain people, so as to prevent being baited. If these softer methods do not work, we can review the behavior and discuss some more severe training techniques, that will not damage your dogs trust in you or spoil your fun walking.