Not a cave diver here, so I apologize if anything here is off base. Safety seriously is my biggest concern in diving. Don't dive beyond your level.
First, OP, I can kinda see your view. It does seem like you're shielded from the currents/storms/unreliable visibility/etc when you're blocked off from the open water where people do all sorts of crap. I personally love small spaces versus wide open, but make no mistake, cave is more dangerous.
Here's something from my best interpretation of things I've read/watched:
Caves seem easy to silt out if you're doing a flutter kick and using longer fins. Lots of times, OW divers go into caves and silt the whole damn thing out by using improper fins and technique, or simply not watching where they're going. God-awful trim can make this worse cause you'll constantly kick the bottom. (maybe the fin part is wrong, I just figured you'd hit stuff easier with long fins)
Also, some caves have multiple little openings at the top. Some are air pockets, some are dead ends, and maybe one or two are true exits to the surface. If a diver isn't familiar with the cave, or doesn't tie a guide line (cause they're not trained...), they can
easily die.
There was an accident at a place called "the shaft" in Australia. Apparently, to exit the cave tunnels, divers had to go back into the cavern and ascend. But the tunnel they used was silted out so bad, when they turned around to leave, it was pitch black. A couple of them, probably from narcosis, ascended up thinking they were surfacing- it was a false opening in the roof of the tunnel.
This isn't where I first heard the story, but here's a link. If it actually has no relation to cave diving and I'm just dumb, disregard
1973 Mount Gambier cave diving accident - Wikipedia
There was a story right here on Scubaboard in the cave diving forum where someone went in, the current changed, then part of the cave collapsed and he couldn't get out. If I remember right, he died.
Because there is no direct ascent to the surface, if you have an emergency, you are absolutely screwed without a line. If the emergency is running out of air, well, Godspeed. People have said to follow the 1/3 rule of gas consumption in caves. Guess what's not taught in OW? That. And running lines. And several other things.
Plus, maybe you're proficient, but your dive buddy isn't. You want to help them, but no dice. Gas planning may not account for them totally running out of air and you getting close to it. Say they're tangled up in a line? Oh yeah, they'll use air
fast. Just like you, trying to get them out.
Pretty soon, it's 2 emergencies. Not good in open water, very bad in open water technical, probably lethal in a cave. Especially if you have gas switches to make- panic will mess that all up. Or, cause desperation. You're nearly out of trimix at 200 feet-but wait, sure as hell can't save yourself with your Nitrox mix or deco gas. Probably a better death than drowning, I'd imagine. Maybe.
Oh, speaking of, use the right gas mix to not get narcosis at the depth you'll stay at. That got someone so lost in a cave that he couldn't find his way out. Pretty sure he had a line, couldn't use it. Guess who it was? An open water instructor. But he wouldn't have known about trimix, just nitrox- lethal at 32%, max ppO2 at 1.4- past 110ft. Maybe he'd be ok if he did. He'll never have another chance to learn and retry.
Oooh, nearly forgot about this one. Some passageways get so narrow you have to push your gear through before you get through. There's probably a certain way you have to do this (maybe not).
It seems that you're thinking of two different types of divers- those who are recreationally certified, just having fun, and those who are technically certified, who
perfected every skill
well before trying for cave.
Some of these factors cannot be eliminated through skill/experience. Just like the guy in the cave that collapsed. Highly trained and experienced divers have a better shot at making it out alive, but still have a higher chance of dying in a cave than in even a technical OW dive.
Wow, holy crap that was a big word salad. Maybe I should write an essay lol...sorry about that.