If stray or neighborhood cats are getting into your yard, your cat will be able to get out of it, and make no mistake--your cat is faster and more agile than you. You will not be able to stop him from leaving the yard if something catches his attention or frightens him. Outdoor cats face a number of risks, including diseases and parasites, poisoned prey, competing predators/wildlife, prey that fights back, motor vehicles, and cruel humans--and these risks apply to your cat immediately once you allow it to roam freely even in your yard, because your fence does nothing to keep him in.
For the safety of your cat and your own peace of mind, consider other options to simply roaming free. The least expensive and simplest option is to get a harness and leash and leash train your cat. It will take a little time for him to get used to it, but you'll be able to walk with him while he's outside and keep him safe in case of something spooking him, other animals in your yard, or something of interest on the other side of the fence, as well as protecting your local wildlife from him. Pros of this option include that it's inexpensive and allows for a wider ranging exploration of the yard; cons include the time it will take to train your cat to wear the harness and a risk of slipping out of it if not adjusted properly.
A higher investment is to construct a "catio," an outdoor enclosure designed for cats to give them some fresh air and safe outdoor time. These are typically a screened porch-type structure, equipped with various perches and "caves" for additional enrichment. These can be custom-built or purchased ready to assemble. A properly constructed catio will prevent the cat from escaping, and in turn, prevent any other animals from entering and harming your cat, or from becoming your cat's prey, reducing the risk of injury and illness. Pros of this option include a lower need to monitor the cat as closely and no training (the cat will figure it out instantly), and cons include the cost and setup involved.