Imho, we can't blame the other side only. When you signalled him, did you sure that he understand what were you trying to say?
I've seen my friend and his buddy, when one signalled that he was low on air, the other gave OK sign and kicked off to chase after fish. I know, the one who chased the fish was a terrible and ignorant diver (proven over time), but that time, it was obvious that he didn't understand. It's very debateable that a divers should know signaling language, but there are also minor possibilities that maybe you have different sign language and you didn't have enough briefing each other.
I'd say, in emergency moment, you did it right, you were low on air and you needed to get up. If he didn't understand or didn't want to go up, save yourself, but don't forget to think back why did it happen. If you didn't have enough briefing or familiarize with each other sign language, it can be worked out. Ask yourself whether you have been a good buddy before you dump someone else without really dig out the problem source. If he is proved to be ignorant, you'd know what to do.
I've seen and experienced that it's not easy to be happy buddy all the time. I have constant buddy but sometimes we still have problem. Such as, we've mad with each other when we tried to call each other underwater. Either one was too occupied with pictures or enjoy the dive too much that we didn't check out on buddy for some time. Although the distance between buddy is not too much apart, problem can occur when there was current, emergency situation if you can not communicate within a minute.
I've once did my safety stop with other divers after I tried to inform him I was LOA for around five minutes. I was swept by current up while he was busy with his picture. But we always have spare buddy as we dive in the group, so it was not that bad and we improved over time. There was time when his tank o ring burst and he went up because the tank emptied like crazy. When I saw him, he was on the surface. But I still blame myself for not realizing it on time, had it been deeper dive (we were starting the dive around 6 meter deep), I might not be able to assist him when he needed me the most.
Great buddy habbit can be improved overtime and can be done with stranger with good preparation.