Also, I usually keep some air in my calf area for trim and the ascent came immediately after I went vertical to move the bubble to my chest again.
In addition to what's already been said here, when you are horizontal, the gas in your suit is fairly evenly distributed over a wide area, and you're presenting a larger profile against the water column that helps to buffer against quick buoyancy changes.
When you go vertical, it's not just the gas by your calves that goes to your head, it's pretty much all free the gas in the suit (you immediately feel the squeeze increase in your legs and lower torso as the bubble migrates to your chest and shoulders). That, in addition to the fact that you're suddenly presenting a smaller vertical profile to the water (and probably subconsciously breathing a bit more and finning a bit as well to maintain position and orientation) makes it very common for newer drysuit divers to start ascending, and they often don't notice until they're well on their way up. The changes we're talking about here are small and can be compensated for with breathing, but if you're neutral, even small changes are enough to get you started on your way up.