I doubt any experienced divers on DB or elsewhere have gone through their diving careers without calling off more than one dive.
I've been on trips where we've called off the dive at a specific site due to weather conditions, or called off the dive altogether. I've had guides call off dives and I have called off dives myself, working as a guide. I've also called off dives just diving with my buddy. Most common reason has been weather. Too high winds, too much ice in the water, too strong currents, or hopeless visibility. Other reasons have included illness on the part of one or more of the divers, or just not feeling up for the dive. I called off one of my first post-OWD dives for this very reason. The vast majority of the places that I have missed out on, I've had a chance to come back to.
The most recent dive I've called off was, like you, in the Red Sea. My buddy and I were scheduled to head out to Elphinstone, but he had never been diving in the Red Sea before, and so didn't feel comfortable with doing off-shore reef diving. We talked it over, and agreed that we'd call it off and stick to shore reefs. I didn't go back to Elphinstone on that trip, but fully plan to go there next chance I get.
A very talented diver once told me that calling a dive for the right reasons is the first indicator that you're on your way to becoming a great diver.