I sometimes dive with a really big guy that is always the last to ascend. He just doesn't work. He is constantly drifting along and watching others. He never chases anything and will often just hang vertically watching, other times in perfect trim and using little fin kicks to adjust.
I'm 6'2 and 180-185. I have always been in the water and comfortable. I am very fit with good lean muscle mass. (cyclist) I have a very high VO2 max and that helps with my efficiency. If you do heavy workouts, your metabolism works in the following days to repair and build. That takes energy and oxygen. Take it easy in the days before diving if you want to minimize this.
We move more than we think we do. Video will show you what you can't self monitor in this regard.
Comfort counts. Stay warm. If you feel a bit chilly at the end of the dive, your metabolism is working to keep you warm. Let your exposure gear do that. Add a hood or a vest or more neoprene.
My wife and I are usually the last back on the boat. On the 2nd or 3rd day of the trip we will be in the RMV range of .45 if we aren't focused on it and if we work at positioning ourselves or fighting current sometimes. If I know I have been efficient, I may work during parts of the dive to position myself to get a shot. I may work harder to get myself into a spot where a subject will be passing by. I may see an eagle ray or large barracuda or something. I never chase but I may swim hard to get on the other side of a large coral formation and position myself to be at complete rest when they then swim by calmly. I have to decide if that expenditure is worth it and if I have the gas relative to others I'm with. Usually I can get away with a couple of expenditures like that though most dives I don't have any reason to. I'm almost always just being as meditative as I can be. Yoga can really help big guys learn to control their breathing.
Finally, if you try to limit your breathing you may build up CO2 which is what causes us to have to breathe. Our instinct to breath isn't because we need oxygen, it's because the build up of CO2 causes us to hurt from a build up of acids in our blood. It is the PH that makes us breathe. Limiting our breathing builds up CO2 and getting rid of the build up causes us to breathe more and the gain is lost. This is why it is suggested that we breathe slowly and deeply. Your goal is to efficiently collect CO2 out of your blood and exhale it. Shallow breathing is fine on shore but inefficient while diving as Scubadada eluded to in his excellent post.
@Compressor @rsingler did I get my description of breathing mechanisms correct? I always like to confirm my understanding with experts and these are two of the most knowledgeable on the board, being anesthesiologists by trade and being highly experienced divers.