I don't really think that neutering really is the issue here. It sounds like he is completely alone for the largest part of the day, which is a huge red flag to me. Additionally, if he's gone through a significant change in character within a very distinct time frame, it might be something that physical (other than neutering) - this is something you should discuss with your vet.
When I was a kid, we usually had two dogs, a male one (German Shepherd) and a neutered female (other breeds). I believe the male wasn't neutered, as our elderly female dog died when he was about six or so, and when we got a new female dog and that dog matured, he became interested in her, and we ushered her to the vet immediately. So, he at least attempted to mate, but afterwards never got the chance again, and it didn't seem to affect him negatively (only that ever since then our female dog, who was about half his size, attempted to jump on him, usually sideways - we wondered if she thought that it had been just play :) ). Neutering a dog can have some effect on a dog's character and also on it's health (though the latter doesn't seem to be 100% clear - see the links provided above) - but I doubt that neutering will have a large effect on his wellbeing if the rest is left unchanged.
This is were being alone comes into play. I'd be gloomy as well if I'd be locked up all alone all day long (if I'm reading you correctly). Dogs are social animals, they need company. If you cannot arrange for anyone to stay a home with the dog (and interact with him!), you might want to consider getting another dog (male, possibly neutered, or a neutered female). Yes, your dog won't like it in the beginning and will be aggressive towards the other dog, but he'll get used to the other dog and gradually learn to accept him.
Last, phone your vet (if you haven't done so already) and ask them if they think you should bring him in. Also watch out for other signals: does he eat and drink as much and as happily as before? How does he react when out for a walk? What happens when you spend the whole day with him?
And with regards to what the vet said over contact with female dogs: clarify with him if he meant that it would be better for your dog's wellbeing not to mate, or if he was thinking of the amount of dogs in this world and wants to prevent unneccessary litters. Obviously, for this reason alone allowing your dog to mate regularly to insure his mental wellbeing should be out of the question, except if you manage to equip him with a condom every time :) But as I said above, I doubt that this is actually the problem.