3 weeks or so should be enough to get some of the bacteria to transfer.
Note that only so much will grow in the NEW pump until the load is greater. so you will not get a full transfer from existing media regardless of how long you wait as the bacteria will stay alive in the media until it is removed.
So, if you have already had the new pump connected and already cycled for 3 months it would be a non-issue to remove the old sponge, but you may still get a spike regardless depending on how much the bacteria in that sponge is actually contributing... but I wouldn't worry too much about it. IT should be a very minor change at this stage, and since bacteria will have already transferred into the new pump, it will only take a couple days at most for more to grow to fill in for whatever is lost when you remove that sponge...
Wait... just realized you are going in opposite direction? Removing HOT for the sponge? Why? No sponge will ever provide the depth of filtration you would get from a HOT type filter, regardless of size of sponge... unless you are attempting to allow the fish to spawn, and even then, best I would suggest would be to leave the HOT but place the sponger over the intake to reduce the chance of eggs/spawn from being sucked into the HOT.