For context: I actually own all of the following sizes: 1.7, 3, 6, and 19cu. I consider 1.7 and 3cu to be more far dangerous, than no pony bottle at all. They give a false sense of security, while providing almost zero security or redundancy.
While diving, try using one of those pony bottles. You'll be surprised; you might get a few breaths before suddenly it's impossible to breathe and near empty in a shockingly short amount of time.
The bare minimum size I can possibly recommend is 6cu. At 6cu, you're still looking at needing to surface IMMEDIATELY. When I tested my 6cu in a pool, each breath at about 7.5ft deep, used about 100psi. At 30ft, you're looking at maybe 15 breaths? The 6cu could save your life, but you need to not delay.
My actual ideal size is 19cu for recreational diving. Aside from extreme emergencies, entanglement, or panic, you should be able to safely surface from any no-deco dive, including taking a safety stop. If you travel a lot, perhaps go 13cu, but be prepared for am abbreviated safety stop.
30 or 40cu is more air, but they're big enough I've noticed a lot of people who have this size just leave it at home. A 13 or 19cu on your person, is 10000x more useful than the 40cu you left at home. The 40s are good for technical diving though, so if you plan to do that you could future proof with a 40cu.