I think Genisis is right (Oh my god did I just say that, heh). I am a swimmer and it took me a long time to not use my hands for minor corrections. I still catch myself sometimes.
That being said, on the surface is when most people have problems I think. Yeah, you have problems under, but it's at the surface that you see the people freaking out...caught in kelp, big wves, can't get back to the boat, etc. I think someone that that is comfortable swimming would have the presence of mind to just shrug it off and deal with it. I know I've been caught on the sruface with a long swim back to the boat. Yeah it sucks, but I "know" that as long as I can see it, I can swim to it...in fact if I can see it, I know I can pull you to it.
I guess it comes down to the question: Would you jump off the boat here or could you get to the bouy WITHOUT your gear? No, then stay on the couch. The less you can rely on your gear, the better and the better I'll feel about diving with you in case I need you to get me back.