There's some good advice on this website.
Cats given free access to the outside world tend to have other things to occupy their minds than sampling unfamiliar vegetation. But even free-roaming adult cats may accidentally ingest needles or seeds that have become entangled in their coat during grooming.
So it's unlikely that this will cause a problem for your cat. Many people have free-roaming cats, many people grow lilies and other poisonous plants, and not many cats suffer as a consequence.
However:
Free-roaming cats have access to many gardens so it will be impossible to prevent all possible contact with potentially harmful plants. You can, however, remove the most toxic plants from your garden and make a note of any in your neighbours’ gardens that are potentially dangerous. List common and Latin names. This list may help your vet if poisoning is suspected.
So, take a note of what the variant of lily is and in the unlikely event that your cat shows signs of illness, you have the right information to hand over to the vet (of course, not all illness will be lily-related, and there may be other plants around that you don't know about).
Basically, don't worry about it too much.
