I had that same experience several times early in my diving creeer. I had it happen in a quarry, when I had about 30 dives. I had it happen in the ocean, off Bonaire when I had ~100 dives. When I was doing tech training, with several hundred dives under my belt, the first time I went to 145 feet I fought the urge to 'cut and run' for almost the entire dive. We were diving in a FL sinkhole, and there was a boat sitting on a rock pile at 145 feet, and I hovered at the stern staring at the transom for most of the dive, so much so that afterward the Instructor asked me what was so incredibly fascinating about the boat. The fact is there was not a d**n thing fascinating about it! But, it allowed me to focus on something and not think about how anxious I was long enouigh to complete the dive and the required skills.
I don't know that there is one and only one 'cure'. For me, it was a matter of continuing to dive, to push depths when I felt comfortable, to not push them when I didn't. Part of it was doing what some people would call bounce dives. I would be diving with a group, on Bonaire or Roatan for example, and I would drop to 100 feet for a few minutes, then come back up to the 70 foot range with the others. I continued that practice over time as I felt increasingly comfortable. Now, I do solo dives to 200+ feet (with the 'appropriate planning and preparation' caveat, of course), without any particular anxiety at depth. It was a matter of acclimatization. For me, part of my increased comfort was merely trusting my equipment. Early on, I kept thinking about everything that could go wrong. Fortunately, I have had enough minor things go wrong, over the years, that allowed me to 'work the problem' successfully, that I am no longer as paranoid as I once was.
Yes, dark / murky water can enhance anxiety. Yes, cold water can enhance anxiety. Yes, a hood - that fits too tight around the neck (and compresses the carotids just a bit) - can definitely enhance anxiety. Don't be embarrassed by it, and don't give up on yourself, many of us have gone through it. Work yourself deeper at your own pace, and don't let anyone else press you into hurrying, or into going deeper than you want, on a particular day.
A story: years ago, when I was fit enough to do reasonably serious technical rock climbing, I was at a climbing site, waiting for a pair to climb a route ahead of my buddy and me. The climber leading the pitch seemed to be futzing around at the base of the climb quite a bit, and I initially thought he was probably just a new, inexperienced climber. But he turned around at one point to face his belayer and I quickly recognized him, as the author of a number of well-known climbing guides, and a guy who had put up a number of impressive new routes. I heard him mention to his belayer, 'I just get butterflies every time at the start of a climb.' That was an eye-opener for me, and very reassuring. If truth be told, I don't get butterflies before every dive. But, the first time I stand at the stern of a boat each new season, ready to jump off in 160+ feet of open ocean, I can tell you that my heart rate is way up there.