If you are not able to sort out significant mismatch on diving goals with your insta-buddy within first dive, you are doing something wrong. If my buddy is swimming off and ignoring my signals, so what, I will tail another buddy team and try to stay close to them also signaling them that I am joining in. At the surface, I will ask to join to another person or team. If there is a guide, I will ask for a different team assignment. It is then the dive guides problem to sort out the crappy dive buddy.
If there is a huge mismatch in diving skills that you feel other diver is a total beginner, same as above applies. You signal the guide and join to another team, you make this the guides problem. This is so simple, cheap and effective.
If you are failing all of the above, you are actually accepting the deviation; you are carrying a pony is just a defense of you failing to be a safe diver and ignoring the risks that these divers may cause to you or to others and themselves.