You think that the cat spends 95% of the time in your garden, but cats are extremely efficient with schedules. There is a good chance he knows approximately when you check the garden and tries to be there when you do. It might be because you feed him or it might be because he likes you and wants to be petted by you. Whatever it is, there is still some chance he has a family but prefers to spend some time with you.
For this purpose, and to prevent an accidental catnapping, you can attach a collar to the animal. You can either attach a paper collar, i.e., just a regular paper with your details on it and tape it around his neck, or buy an easy break collar and attach your note on the collar. Make sure that the paper collar is not too thick otherwise the animal might strangle himself if it get caught by a fence or a branch. Easy break collars are sold for £2-3 in UK. The prices should be similar in France.
In UK, many wandering cats are microchipped so you can contact a vet. If there the cat is microchipped, the vet can contact the last owners on your behalf and start working out how to take care of the cat.
If you cannot contact an owner even after attaching a collar and checking for a microchip, you can still take care of the cat in your garden. Stray animals are well adjusted to the elements and they can survive in your garden, depending on the winter temperatures. You can attach a catflap on a garden shed door if you have one. If you want this to be a long term investment, you can get a microchip activated cat flaps, which should be around £50 in UK. This will make sure that the cat flap will only be activated for your cat and he will be perfectly safe from other cats and other similar sized predators.
Even if you don't have a shed, you can buy a small cat house or turn a simple cardboard box to a cat house. Make sure that it is isolated from the floor with a rug or styrofoam blocks and it has a metal or wooden roof so that it won't get wet if it rains. Adult cats are surprisingly durable against cold as long as they are not wet and they are fed well. Put an old sweater in it for your cat to cuddle for warmth. If you want your cat to be extra cosy, you can get a USB heating pad. They are relatively cheap and since they operate at 5V, there is very small chance of an incidence.
I am not knowledgeable on how cats might attack/hurt a newborn, so you might want to ask this as a separate question if you decide to keep him in your house.
I hope this helps.