Your cat isn't weird, a lot of cats display this behaviour. It could just be because he goes to check out what you are doing, and then boom! What a good idea, partner, let's use the litter box.
When my cat was young, she used to do this every time I cleaned the litter box, and I clean it once daily. However she stopped doing it after I started telling her "No" every time she tried to do so.
She still likes to come see me clean it every time, but she doesn't go in. She waits until I'm done then gets in. It could be due to the fact that cats know the litter box gets clean and fancy after we are finished cleaning it, and so try to make things faster by just using it already even before we are done cleaning.
I read that when a cat sees someone removing its feces, it seems weird to a cat, and the cat starts getting a little stressed or confused about it and so uses the box again to keep their feces there, I'm just not sure on which site I read that and so I'm sorry to not be able to set you a link here.
It could also be because your cat prefers an open box like @Stig Tore said. And by the way, you should consider cleaning the litter box once if not twice daily, except if it is a huge box.