I'd get back in there and try it again in safe, stress free conditions. The ocean is enormous and humbling. No one is going to argue that with you.
Since you're relatively close, I would fly or drive out to San Diego or Long Beach for a few days and go to Catalina to learn to feel safe in the ocean. The Dive Park in Avalon is super protected and calm, has great visibility, and you're never more than 50 yards from the shore where you walk down the steps into the water. You can dive in any depth you want. It's 15 feet deep next to the steps and drops down to 90 at the edge of the reef. There's as much life in the shallows as their is down deep. I'd actually say there's more. And after you're comfortable in the shallows next to the steps, slowly move deeper at your own pace.
I personally don't get scared getting into the big ocean, but I have anxiety about going too deep, especially in the middle of the ocean. So no, you're not alone in having anxiety/panic attacks in the ocean. And to make you feel even more normal, I grew up in San Diego and surfed a few times a week from the time I was like 10. But everytime we go under 85 feet deep, I get nervous and will only do it in great conditions.
I also find that anxiety (for me and a lot of other people) happens way more on charter boats where you're rushed and can't go at your pace. It's a lot different when you can suit up at your own pace, do the dive however you want, then sit on land relaxing and reminiscing about what you saw for as long as you please before your next dive.
Good luck! Let us know how you progress.