I was diving with a buddy a couple of weeks ago who forgot his wing. He did have his drysuit, though. He swapped his STA for his doubles and completely forgot to put his doubles wing on, leaving it on the kitchen table. We decided to let him do the dive (there were three of us watching him, so he was in good hands). We did a quick bouyancy check on the surface, and down he went. He completed the dive with only his drysuit as bouyancy. He added a couple of pounds of weight for the second dive, and everything was uneventful. After talking to him about it, he felt that he could have just swam the rig up.
If you have some experience, and have some form of bouyancy, it's not that big of a deal. I've done dives without using the BC inflator, just for practice.