The only thing that really bugs me is the buddy who insists on going his or her own way on a dive, leaving all the buddying effort to me. I should not have to spend my entire dive trying to figure out where my buddy just surprisingly disappeared to. Here is a story about the worst one, a diver who said he had over 600 dives (and I believe that to be true).
It was in Cozumel, and our group paired off in buddy teams while being led by a DM, as is required by law there. On the first dive, he kept dropping behind me and then darting off to check out one thing or another. I had to keep turning around and looking for him, and the DM had to do the same thing, often stopping the whole group while waiting for him to come back to the pack. For the second dive, we agreed to do a site called Paso del Cedral, and we specifically decided to do it because there is a complex Swiss cheese-like swim through that can be fun. As we went through the first part of the dive, he kept hanging back and going his own way, like before, and I kept hanging back to stay with him. As the group headed for that swim through, he darted off in a different direction, well behind the group. The visibility was not great, so I tried to stay half way between him and the group so I could see both. The DM turned and looked back. Seeing me hanging so far back and not moving to him, he gave a WTF? gesture, and I pointed to where my buddy was swimming. Even in the distance I could see how exasperated the DM was--that diver was now too far along in the current, and he would never be able to join them. The DM led the entire group to join my buddy, and so everyone missed the focal point of the dive. Back on the boat, my buddy asked the DM what had happened to that great swim through that was supposed to be on that site, and one of the other divers told him that we had missed it because we had to go chase after him. He thought that was pretty funny, and he didn't seem to be bothered by the angry glares of everyone else on the boat.
It is just selfishness to put your own interests and desires ahead of your buddy and anyone else on the dive.