Everyone brings up great points and some great advice.
I am a novice at diving and UW photography, but I have done every dive except for my 2 night dives with my camera. I haven't taken my strobe yet but plan to on my upcoming trip. I always kept my camera on a cord around my neck like a pendant and held it close to my chest while I swam, and since it is almost neutrally bouyant it never really got in the way. I am going to use another SB members trick now that I am using a handle and strobe and will zip tie a d-ring to the handle and clip it to my BC. (sorry can't remember who deserves the credit, but thanks whoever posted this a couple of weeks ago)
I haven't found it to be a big distraction since I am always going over a checklist in my head when diving. First, keep an eye on my gauges every couple of minutes. Second, make sure my buddy or group is close by and they are ok. Third, be aware of what is around me, how close I am to the reef or ground. I try to keep the focus of the dive and group first and when I see something I want to shoot I take a couple of shots and keep going. The beauty of a point and shoot is having an lcd screen so I don't have to have the camera directly up to my face to see what I am shooting.
I never really had a situation where I felt the camera impeaded my ability or the dive. I left it on the boat on the night dives because I knew it would be a bit much to juggle and I have never dove at night before.
I think each person has to make this decision on their own. I wouldn't suggest a DSLR with dual strobes and trying macro your first dive out but a small P&S clipped to your BC and a decent head on your shoulders should get you through a dive without too much trouble.
Thanks cfelliot for the D-ring idea for the camera with the handle.