Like human beings, cats are different. Some outdoor cats are perfectly fine and willing with a schedule (be it 'only during the day/night' or 'every evening for a few hours'), however, yours demonstrably is not which can be one of the problems with 'going for walks' with cats - they demand it on their own schedule and time.
I know, not the answer you wanted probably... Sorry.
So, given that he won't be fine with a schedule, your options are:
1) A cat 'doorbell'. (f.e. https://www.amazon.com/Newcomdigi-Doorbell-Water-Proof-Dust-Proof-Wireless/dp/B01CY0UQWU/ref=sr_1_3?ie=UTF8&qid=1483180530&sr=8-3&keywords=cat+doorbell - not endorsed by me, just the first one I found). This is activated by the animal and notifies you wirelessly that the cat wants to be let in.
Only an option if you are home at all times, of course. Also you should look into clicker training for this (which is also great for other things).
2) A cat flap/door. Might need a custom one for a Bengal, depending on the F generation. The cat can go outside and inside as they please and they won't annoy you anymore. If you don't own the place you live in, cat flaps can be out into windows as well (you do have to replace the glass of course), so its not like you have to open up the walls or anything :)
3) An outside fun-house time in combination with the cat flap. This is an awesome but also very expensive thing to do and only an option if you have a garden and the permission from the landlord or lady. Basically a metal square cage with trees / wooden beams, hiding spots and other 'outside stuff' to entertain cats with.
...Well, okay, there is one last thing you could try. Again with clicker training, you could teach your cat a specific sign - that is only used for this and not ever in jest - that means 'we will go outside now'.
Given the cats previous behavior, this is unlikely to work as the problem is not specifically the irregularity of taking them outside but that they want to explore on their own.