OP, I somewhat do feel your pain. I pulled my mask off in the pool for the first time and I freaked out. However, I just sat breathing for awhile and allowed myself to calm down. Then I put my mask back on and cleared it. when we did the swimming around with the mask off, I had to work to keep my breathing under control, but I did it. That night I jumped in my hot tub and practiced until I could take off my mask, look around, breathe thru my snorkel, put my mask back on, clear it, etc. I'd say just practice. It really does help.
Thanks for the support WreckDiver - I should have been practicing in my hot tub tonight. I guess it is a mind job too as I was never able to aquire that skill as a child, should have been easy then!!!