People react very differently under that type of stress. With a redundant air supply my sac is not going to double. It will go up, but won't come close to double.
I was once abandoned by a boat buddy and ended up entangled in kelp (shouldn't have tried to surface in a kelp forest to check on lost buddy) and without redundant air my sac barely moved. I figured I had enough air to last another 45 minutes, so someone would eventually come looking for me if I couldn't work myself lose.
I'm sure if you go to a buddy, then your heart rate will go up. Many unknowns involved. Knowing that I can turn a valve and have air really helps keep me calm, even if I swim over to a buddy in an OOA situation.
I have yet to hear a good argument against taking a pony other than people may push the gas planning too hard because of the pony. Outside of that, what drawback is there? It's a piece of gear you may never use? I may never use my octo, or my airbags in my car. If something goes wrong with a pony, I'm no worse off than if I didn't have it to begin with.
As long as the pony is completly out of your thoughts when planning the dive, I see no drawbacks.