To start the process, get the horse familiar with its muzzle being handled. Start by handling its head and running your hand over its muzzle; start sliding your hand over it quickly and getting out before they start fussing, and move to longer approaches as they become more confident. Pet them all over their body; chances are they will attempt to become the boss in the relationship and nip at you. Don't strike them when they do that, but lightly tap them on the nose with your hand; they will likely whip their head out of the way right afterward, as if another horse in the herd was biting back to establish dominance.
This is a fine line to draw between too much discipline and not being firm enough to get the message across. If you aren't being firm enough, they will come right back and try to bite you again--and it won't be as light either. If it's just right, they'll likely whip their head back away and then lick their lips. A pure relaxed wrist movement is usually enough; don't delay at all--try to meet them as they are trying to get you (it will also help prevent them from actually getting you), and if they whip their head out of the way after the bite/attempt and hold their head high and away from you (they will--just like if they bit another horse), make them step back away from you instead--basically, an immediate dominating move to let them know they do not get to boss you.
Be sure and pet them immediately afterward (and after any discipline) so they don't get jumpy and think you only touch an area of their body to discipline them. Don't try to feed them until they are good with their head being handled. They are allowed to nuzzle you, but don't let them take your hand, clothes, or anything you are holding inside their lips--that, in my experience, always progresses to a nip/bite with the teeth on your body.
When they are safe to handle and know to not use their teeth, do not "hold" the treat. Rest it on your hand, so they can lip it off. At this point, they may start getting excited and grabbing at the treat. If they make a grab, I usually snatch the treat back (to prevent an accidental bite), get them to stand quietly, and offer it again. Be very careful doing this (or anything described in this answer)--horses are big and dangerous, even if they aren't trying to be. Always use your personal discretion when working with a horse; if you feel unsafe, get an experienced friend or professional to help. If something isn't working, stop and try something else; each horse is different--the end goal is for them to learn successfully, not learn according to some pattern someone recommends. Continue the process described above until they are good. Some horses may never be able to control their excitement, just like some people, but most will quickly learn.
Personal anecdote: my horse (who looks very much like a 4yo version of that picture) was very good for the first couple baby carrots, but then started grabbing the carrots. When I stopped giving the carrots to her (whether I had more left or not), she started trying to shove me with her shoulder--i.e. run me over. I switched to feeding her carrots through the opening in a stall over the course of a few days until she learned to always take them gently.