I think anxiety is the number one issue. Then there is a physical fitness aspect, especially in dives against currents.
I just found out that a tight suit on my chest and abdomen increased my air consumption, and this was as clear as day and night when I took it off.
Overtime, your muscles develop and you will dive much better. But if you are like me, diving only 10 or 15 times a year, we will just follow the pack.
The error in judgement of some folks to try to stay down too long so that they are not the first one up is a big contributor of the injuries we see in scuba diving. If you need to come up first, be the first to surface, just let your dive master know what your tank reads, and if others get upset, tell them to jump in the lake.