If you can't get over feeling panic without holding your nose, then be sure you have a reliable way to remove and replace your mask AND hold your nose. Like everyone said, practice is the key. I would think you want to practice with a reg under the surface rather than in a tub, or with a snorkel, but I don't know this from experience.
Can you find a pool to use and maybe take an experienced friend or instructor? Get comfy on the bottom and remove and replace a bunch of times?
If you can't find a way to handle this, be very careful in future diving. At a minimum, stay close to a buddy and have a plan. If you do overcome it, and you probably will, I suggest you practice this somewhat often while diving so you remain comfortable with the feeling.
Don't feel bad, I think this is very common. Recently on a dive my nose had gotten runny and goobered up the inside of my mask and I couldn't clear it out enough. I pulled off the mask to "rinse" it out better and replace it. The nearby divemaster leading the dive swam over to me as fast as he could, grabbed my BC straps and held onto me. Startled the heck out of me. (I couldn't see this, it was described to me later). I guess I should have warned him, but it didn't seem like a big deal at the time.
He later told me that so many people panic and bolt if they have a flood or lose a mask, that he didn't know what I might do.
Good luck and hang in there.