I have no problem with jumping off the boat and immediately descending, if conditions require it. We did this in some of the current diving in Cozumel.
But my diving at home is almost always in flat water with very poor visibility, and making sure the team is close together and prepared to descend at the same moment is often critical. I suspect, as Soggy says, that I'd have enough air trapped in various places to be able to stay on the surface by swimming, but I'd simply rather not. Being relaxed and comfortable and not at all winded is a very good place to start a dive for me, and I can see no detriment whatsoever to having my wing inflated when I get in the water.
Maybe not until the overpressure valve pops, but I'd be one of those people with a full wing that you guys would be regarding with suspicion.