The peak occurs because there are not sufficient bacteria to counter the effect. Once the peak starts, the bacteria start to form and break it back down.
These bacteria form in your gravel, your filter, anywhere. If you change the gravel completely you will naturally lose some of your bacteria, but if you keep your filter intact then you shouldn't worry all that much. Keep an eye on the levels and do water changes if it gets to dangerous levels.
Make sure you always leave plenty of time between changing the gravel and filter media so the bacteria have time to form in the new gravel / filter before you take the only ones you have left back out.
To quote from this page:
During the cycling process, ammonia levels will go up and then
suddenly plummet as the nitrite-forming bacteria take hold. Because
nitrate-forming bacteria don't even begin to appear until nitrite is
present in significant quantities, nitrite levels skyrocket (as the
built-up ammonia is converted), continuing to rise as the
continually-produced ammonia is converted to nitrite. Once the
nitrate-forming bacteria take hold, nitrite levels fall, nitrate
levels rise, and the tank is fully cycled.
If the established tank uses an undergravel filter, nitrifying
bacteria will be attached to the gravel. Take some of the gravel (a
cup or more) and hang it in a mesh bag in your filter (if you can), or
lay it over the top of the gravel in the new tank (if it has an UGF).