Number one rule of diving.... well, there's a lot of number one rules.
One of them is that anyone can call a dive at any time for any reason. No apologies necessary, no repercussions. After-action review is good for learning how to avoid the problem situation next time.
I've called dives at depth, on the surface, and before getting in the water. Last Saturday I called a shore dive (for me, others still dove) before even taking my gear out of the trunk due to weather conditions I wasn't comfortable with.
No worries, there's always another day.
And now, a partially off-topic rant...
The only real issue I've ever had with was in an unnamed tropical location with a fresh-faced DM/guide. I gave the "thumbs-up" signal at the pressure he laid out in his dive plan, he responded back with OK and turned the group INTO the current at 90fsw and started working very slowly up the wall. Too slowly, IMO, for the depth, current, and gas reserves. We didn't have to turn into the current, it was a live boat operation, and we were the only group. I asked him about that when we got back to the boat, and he said he turned that way because "that part of the wall is more interesting". Umm, dude, we're thumbing the dive, not sightseeing.
Sometimes I wish there was a standard hand signal for "F*** you, we're going up NOW." when situations like this come up.