My guess would be that you have a bit of claustrophobia. My wife was admittedly claustrophobic, she could dive in clear water but no way murky, and for that reason. She knew it.
Go into a pool at night, work on getting used to being in dark water but in a completely controlled environment.
I don't think I'm claustrophobic. I've been spelunking when I've had to squeeze through some really tight spaces for fairly long distances and it never bothered me. Being on the bottom of a big heaping pile of cousins playing football never bothered me, heck wearing a wetsuit doesn't bother me. Being in really tight crawl spaces running gas lines never bothered me.
I really think it's just a fear of the unknown. I have a healthy fear of heights, but never stopped me from working on the roof of a building on a chilling tower. I do creep to the edge and look over the sides, no issues.
It's almost like being in a new environment on top of the wall, being cold, and in the dark all by myself ticked all the boxes at the same time. Again though, not a panic fear. More like a "I'll try this another day" type of decision based on fear.
I like the idea of going into a dark pool, but I don't think it'll be anywhere close to the quarry. I need to find a local indoor pool so I can practice with some new gear that I got anyway.