I did exactly that when I was doing my solo checkout dive off the Australia liveaboard last month. Under the watchful eye of the DM I calmly and slowly removed my primary reg, laid it in my shoulder, reached down, took the pony reg and placed it in my mouth, then casually took a breath and got nothing. He probably noticed that my next move was to open the valve. Wasn't a big deal. Except for the mouthful of sea water as I neglected to purge or breathe out first.
This was my first open water dive with the bottle slung vs back mounted. In that orientation I always had the valve open.