On the rec side here, but the answer is it depends. I take my responsibility as a buddy seriously. I dive frequently with my son, and now occasionally with my wife. The difference in that situation is I feel responsible FOR my buddy, whereas in an instabuddy or dive buddy I feel responsible TO my buddy. I will attempt to articulate the difference in my mind.
Responsible FOR, means I will die before I would give up and leave my son or wife. I knowingly accept that responsibility and try to mitigate risk as much as possible, but if push comes to shove there isn't really a choice to be made for me. They as divers are responsible for themselves, and they are responsible divers and attentive buddies. My son especially is trained to dive off to my right side about a foot or 2 in front of me, if he is out I of position I give him and openhand right arm signal and he gets in position. This came into play after a few dives of him being close to me but all over the place. My neck hurt and I was twisting and rolling to keep track of him. We established our routine and it works for us. My wife is newly certified and I keep close proximity to her, and if my son is in a trio with us I have explained that it is even more important that I don't have to search for him as my main focus is on my wife due to inexperience. Just as when I take people caving, hiking, white water rafting, sky diving or any other activity that has inherent danger, if I brought you I have a responsibility to bring you back. When that person is family the pressure is even heavier.
When I am partnered with an instabuddy or an experienced diver I know, I feel a responsibility TO my buddy. This means that I will do all that I can to be a quality buddy that stays close, is attentive, and will attempt to help my buddy in distress in any way that I SAFELY can. I will not die because of another divers stupidity or bad decisions though. The buddy is responsible FOR himself, I am responsible FOR myself, and we have a responsibility TO each other to help each other safely complete the dive.
So to abandon my buddy would take them being irresponsible, unresponsive to redirection, and a threat to my safety to continue with them. Anything else is me being irresponsible and dangerous in my opinion.