So be really conservative and build in contingencies. Everyone does that. But, you can't create a contingency for every thing under the sun or you'll end up wearing a complete dive store.
How conservative is absolutely safe?
The answer, of course, involves classical technical dive planning.
1) Properly evaluate the site (cave, wreck, surface conditions) for feasibility.
2) Plan with software, and execute with an HE dive computer, on conservative settings.
3) Minimize all risks, and eliminate any unnecessary risks.
4) Ensure you have an equally competent buddy for the dive.
5) Ensure you and your buddy are each independently self-sufficient.
6) Redundancy for all critical items of gear.
7) Overlapping capabilities for deco gasses (I like 20/40, 30/30, EAN 50, and O2).
8) Rule of thirds for bottom mix.
9) Always use DPVs.
10) Avoid CCRs and SCRs.
11) Abort the dive the moment conditions become dice-y.
This has kept me alive. It should work for everybody. Certain people, like GI3, I would not dive with, under the category of "competent buddy." Certain pieces of gear, like CCRs and SCRs, I would not touch.
There are a lot of technical divers who regularly violate several of the above. That is why there will always be diving deaths, and not due to anything "unavoidable."
There is no "accident" that is "unavoidable" in scuba.
There is foolishness, lots of that, however.