Diving nitrox is not safer than air if both are dived to NDL.
Yes and No. No in that what matters most is your NDL and your ascent practices. Yes in that if you do the same dive on Nitrox vs Air your nitrogen loading will be less and thus, at least in theory, the dive will be safer. This is magnified if doing multiple dives. With that said that is really a hypothetical situation because rarely are you doing the exact same dive if using Nitrox. You are staying longer and may be coming just as close or closer even to your NDL than if you'd used air. In that instance the Nitrox dive may be "less safe" than the Air dive.
The primary reason for using recreational Nitrox is to increase your NDL. That is seen most clearly on repetitive dives in the 40-100' range using something like EAN36.
But with the hypothetical situation aside I tend to agree with you. The whole theory of NDL supports your view. Dive within them and pay attention to your ascent practices and the gas you are using is irrelevant to the level of safety of the dive.