I've never dived a pony bottle and said, "Damn, I wish I didn't bring this with me", nor have I heard anyone say something similar.
I *have* heard divers say that, or to be more forthright, I have heard divers say they said that and therefore no longer dive/own a pony. I've heard them say that it was inconvenient to dive the pony. It's purely anecdotal, but in every case among those I know personally, every single one said they didn't like how the pony adversely affected their stability and made them "turtle". Of course, that is a direct result of mounting additional weight above you (behind, if you're vertical); if they had their pony sling-style, it would actually *increase* dynamic stability, but none of them tried that, as they'd already decided to scrap the pony.
As for me, I have plenty of dives with my 19cf pony (including flying with it, as I will do again in less than two weeks), and I have never had a dive where I wished I didn't bring it. If I'm in very shallow water, such as a 17-foot-max (barring a shovel) shark tooth dive, I don't wear it, but outside those rare exceptions, it's always on me. I may never need to use it -- gear and other external problems are very rare, and I'm *NOT* going to just "run out of air" -- but to me in my personal experience, it's utterly trivial to carry along on the dives. Sure, it's a bit more weight in the luggage or being carried to and from boats, but that's utterly irrelevant.
It seems that solo dives and low-quality buddies get all the press when it comes to ponies (and I've certainly logged both), but I threw in another reason for *me* to dive a pony. When I'm leading divers (or even diving with students), should I have an unlikely problem, I have no doubts whatsoever that I could swim up to any of them and share air for the ascent. On the other hand, if that were to happen, my ability to handle any additional situations would be very significantly reduced. If I have an issue but have my (sufficiently-sized) pony, although it's certainly thumb-the-dive time, my capabilities during the return and ascent are effectively preserved.
Now, anyone is free to believe that I have seen one too many "Engineering Disasters" episodes of Modern Marvels, but the way *I* choose to see it is that any significant event underwater is going to involve a chain of failures. A single problem in isolation is not sufficient to cause an accident (note: forgotten or never-had training can be one of the chain of failures, of course). If adding minimal inconvenience to my diving is all it takes to add one more level to the required chain of failures that would have to be present in order to be in deep, it doesn't bother me at all.
I carry a lift bag for light salvage and backup buoyancy, even though I've never had a buoyancy failure (although I have come up with more than a few lost anchors, hehe). I carry multiple cutting tools in case of entanglements, even though I've never been held fast. I carry two or more lights, even though I've never had a flood during a dive. I carry a backup mask, even though I've never lost mine. I carry a big surface marker, even though I've never been adrift at sea. I carry a pony even though I've never had a gas supply failure on a dive. Every single piece of this is "unnecessary" redundancy, and it may not make sense to someone else, but none of it bothers *me*. Can I tell someone else that any of it is "worth it"? No, I really can't. I can only say why *I* dive with it.