I had the same problem as an open water diver. It was my first time wearing a hood and diving with gloves etc. only my problem was the actual removal and replace. The water was 32F and I was frozen before I even hit the bottom I got the mask off and started to panic because of the shock/cold and then really panicked when trying to put it back on. My snorkel got stuck and when I fixed that I had to deal with the hood issue. With the three fingered mittens that wasn't the easiest thing I have done diving. I have to say if it wasn't for my instructor holding me down I probably would have hit the surface really really really fast! My instructor managed to fix my hood and... sure enough... I got to redo the skill. Ask me if I was impressed! The second time around went much better and after the evaluation left the skill for a while.
When I did my advanced course we had near 0 vis and a few of the other divers had some really big problems with their dry suits. (we were doing dry suit cert. at the same time, did OW in 7mm wetsuits... was very cold!!!!) One diver had her feet completely come out of her fins because she was quite small and had rolled the legs up a bit on the suit to accommodate for her lack of height. When her feet popped out of her fins she panicked and grabbed on to me and in the process knocked off my mask at which point I almost panicked as well... Thankfully my brain took over and decided that I had to be this lady's 'rock' and help hold her down (air in her feet at this point). Our instructor found us, yay for lights, and helped her ... and I'm still sitting there, no mask, no idea what is going on and I wasn't planning on opening my eyes in the mud! It felt like forever before I someone was around me again and it was my instructor searching for my mask. He found it, I cleared it and went on with the dive. Ever since that experience I remove my mask every dive! Lucky for me too... a month later in South America (at least the water was warm this time) I was doing a safety stop and got nailed by another divers fin (they were also at the safety stop and had just started to leave) which knocked my mask off... it sunk! I finished my safety stop and counted on my buddy’s’ really big signals (and a tap/tug) to make it to the surface…
In cold water I find that splashing your face with water before you dive is a good thing, as already mentioned!!! It takes a bit of the shock away when you actually have to remove your mask later in a dive whether you choose to or not.
Practice, practice, practice, that's all you can do to become comfortable with a skill.
If you are diving with a group of people (I go diving with 4 or 5 other people on a regular basis) you can make a game of removing your mask. So long as you are in an area where you won't harm anything on the bottom, form a circle and play a mask passing game. Take yours off, pass it to the person on your right and put on the mask being passed to you then clear it ... and continue until you get yours back.
Another thing, if you find you can't do something like this diving or don't get to dive a whole lot... check with your dive shop (if they have a pool) of a local swimming pool and see if they have a night or some time when you can take scuba gear in the pool. It makes for a great place to practice (and its a little warmer too... and a hot shower after getting out doesn’t hurt any either!). Many dive shops will let you use the pool and some have special nights where anyone can come, use the pool and gear (the usual pool stuff) and just practice their skills.
Good luck to you and congrats on the way you handled the situation. Many a new diver in the same situation would have panicked, stopped breathing and darted for the surface! You are also right about continuing with your dive. If you hadn’t, your probably would have over thought the situation and would have had more anxiety going into future dives/training.
Happy diving and all the best
AP