I'm not a cave diver. In fact, I'm not even OW certified yet - that process starts tomorrow!
Nevertheless, I always find these types of discussions fascinating.
Despite my lack of experience in diving, I have engaged in quite a few high risk sports - alpine mountaineering, paragliding, rock climbing, back country skiing, high speed mountain biking. My observation is that everyone I've encountered in a sport that most people would classify as "risky" justify the risk to themselves with the old saw about "proper training" and "not pushing your limits." I think that's basically playing with statistics - whether you choose to believe it or not, the chances of getting chopped due to errors in flying a paraglider are increased over your getting run down by a beer truck on the way to the launch. Risk is determined by the chance of an error multiplied by the consequences of that error. You might play soccer a thousand games, have a near injury a dozen times, and sprain an ankle twice. At those odds, soccer would be pretty low risk. In a high risk sport you have many opportunities to make a bad decision, and the consequences of error are high. Place a shaky belay anchor on a run-out lead, your partner peels, you're both dead or seriously injured. Further, in high risk sportsk, it's more likely that it's your decisions that directly affect your outcomes, in statistics like traffic accidents, often you are a victim simply for being on the road. Hence, your skill in risk management becomes highly important.
Take a look at a sport that's been around for awhile - say Himalayan mountain exploration. How many big name mountain explorers have survived over the years? Answer - not that many. Famed explorer after explorer have died in the mountains, often from "freak accidents" - but more likely from cascading bad decisions mixed with a bit of bad luck. Although these guys were experienced and followed the rules, their repeated exposure to a harsh and unforgiving environment eventually caught up with them.
Same with deaths in paragliding/hang gliding. All beginners are drilled that as long as they follow the rules, they'll be perfectly safe. The only people who die are those who push the limits or break the rules. Problem is, conditions can change, along with the rules. It's not the beginners who die - it's the intermediates and experts, many of whom don't feel they are pushing the limits until the time of impact with an unyielding planet earth.
I fully don't intend this to be a troll, or a flame of cave diving. Heck, I applaud those who push the boundaries of exploration in all things. But folks who engage in high risk avocations should accept that and practice their own personal risk management assessments on a daily, hourly, or even more frequent basis. Having PADI tell me that diving is safer than flying on a helicopter (or whatever the euphanism du jour is) is essentially meaningless. Statistics are great until you're caught in that fatal minority. If you're drinking the kool aid of a particular sport, don't rely on the other kool aid drinkers for assurances of safety - the risk is there, you just have to deal with it on your own terms.
What it boils down to is your personal risk aversion factor: Do you derive enough pleasure from (insert sport here) to risk not coming home to your family? If so, cool. Get trained, follow the rules, but be aware that at some time in your career you may to have to make a decision, probably what seems like an insignificant one at the time, that may affect the outcome of your life. If you screw that up, folks are going to point to that decision and say that either you had insufficient training, or didn't follow the rules to justify their continued participation. But you might still be dead.