One rule of thumb might be, when you get to the point (a depth, a temperature, whatever) that it scares you, you have exceeded your limits.
Well THAT might be a problem! LOL Remember, ignorance is bliss!
This does make me think of a semi-related question:
Imagine that you're a newb (say, like me) and you dive to 130'. Because you're a newb and haven't been educated properly on gas consumption, you stupidly run completely out of air while you're at 130'. And, also because you're an inexperienced newb (possibly diving with a newb buddy), when it happens you don't see your buddy anywhere close by. Though you are a newb, you keep calm and cool and make the decision to make a controlled emergency ascent, swimming/floating up at 60'/min up to the 60' level and then 30'/min to the surface, skipping the safety stop since it's optional anyway and you're out of air.
By my calculations, the ascent would take 3:10. That is longer than most people can hold their breath. But, with the partial pressure of O2 in your lungs at 130, even though you're NOT holding your breath, you should be able to "last" a lot longer on your lungful of air than you could by holding your breath on the surface, right? Plus, as you ascend and the ambient pressure drops, some of the air in your "empty" tank will become available, right? So you might get another half or full breath out of your tank on the way up?
Or, to summarize all these questions: If you dive to the Rec limit and you have an almost-worst case scenario, is it reasonable to think that a calm, cool, and collected diver could still get themselves to the surface without getting hurt or drowning?