In my experience, most cats do not adjust well to being moved. Taking your cat out of the environment he knows -- and with his people not around, either! -- is likely to be stressful. And if there are pets or children at the other house, that might well make it worse.
I've left my people-focused cats for a week or so with just daily or twice-daily visits and not had problems, but perhaps your cat is more needy. Would the neighbors be willing to spend more time at your house? If they were going to watch TV or read or surf at home in the evenings, perhaps they'd be willing to do some of that at your house every now and then?
All that said, there are cats who adjust well to being moved. Think back to when you got your cat; how did that go? Have you tried shorter visits elsewhere? I had a cat once with some special medical issues that needed some extra attention, so I tried taking him to a friend's house for a short trip (a few days) to see how that went. She gave him his own room and he did fine, so we continued that for longer vacations (but not much more than a week at a time). Similarly, one of her cats required extra attention and I was able to bring him to my house for a week. Both of these cats were very calm and mellow to begin with, and we treated it like adoption in terms of giving the cat a limited, safe environment to start with. (Over time both cats got the run of more of the house.) But don't do this for the first time for a three-week vacation; if things don't go well, you aren't in a position to adjust. (Well, unless your neighbor can bring the cat back to your house while you're gone, and do the regular visits, and you're both ok with that.)