I am not surprised at all - pardon the play on words.
It certainly makes a difference for me.And, that (not removing the mask) is a key point (see below)As a general observation, no mask breathing u/w is one of the primary difficulties I see for OW students in general. Skills such as performing the CESA, completing an underwater navigation exercise, taking gear off and putting it back on underwater, and - of course - developing good buoyancy control and trim, all of these may challenge a student's skill and require repetition to acheive mastery. But, they generally don't induce panic. Sucking water (warm or cold) up the nose underwater can turn an otherwise comfortable, rational diver into a 'bolter' in an instant. I have seen that happen, more than once.
I am convinced, from personal experience, that the shock of facial skin / soft tissue being suddenly immersed in very cold water can 'take your breath away' to borrow a phrase, and cause you to inhale abruptly - forgetting to practice airway control (and only inhale through the mouth).
No mask breathing is, for me as a diver and as an instructor, one of the most critical skills. It should be practiced regularly after certification. During tech training, it was probably the single skill that almost caused me to say, 'To H... with it!' Today, I regularly go to local quarry, to what we call the 'Deep Hole' - 93 feet - and flood and remove my mask and sit there and breath for several minutes, just to be sure I can still do it. In warm water in the pool or in the Caribbean, it is a piece of cake. In 43 degree quarry water, it is a piece of something else.
You are not alone, it is not just your physiology. You may not have 'forgotten' the skill, but it sounds like you had not practiced it. If you are going to dive cold water, you should practice it, regularly, and in cold water. But, frankly, I think it is a skill that every diver should regularly practice.