If you are exploring/sight-seeing, then try not to buddy with a spear fisherman or photographer unless you want to observe them.
For the original OP, I do two kinds of diving.
One with the group where I will buddy up and make sure the group has their sh*t together.
Two, Spear Guns and/or cameras. When either of these are in my hands, I am solo diving. I will not do this with a group, and I will typically look for the other 1 or 2 divers during the dive as a courtesy. Notice I stopped at 2. I personally like to dive with no more than 2 others max. I typically limit my night diving to 2 divers also.
The important thing to learn here is to communicate each divers expectations before you are limited to your version of hand signals (under water) and never let your buddy put you in a situation that you havent been trained to handle or you arent comfortable with.