Spectre once bubbled...
The catch is that it is believed O2 is narcotic as well, so you're replacing a narcotic gas with a narcotic gas.
You are dead on Spectre. In fact, some people believe that O2 is MORE narcotic than N2. The added complication is that we metabolize O2, so the narcotic effect may be lessened.
One thing is for sure, if O2 is narcotic, and I'm not sure how narcotic it is, it has the added effect of making you feel like you are not impaired. This may well be even more dangerous.
The TDI advanced nitrox book says that this is an advantage, but you'd have to be and idiot to believe that. If you're impaired, you might as well be aware of it!
The only safe and sure way to make your mix less narcotic is to add Helium to it. The other problem with trying to use O2 to make a mix less narcotic is that you need this advantage when going deep, and that's exactly where you don't want a lot of O2 due to the dangers of O2 toxicity.
I think if two people dive together they simply need to take into account the limitations of the mixes they dive. For more serious dives, it makes sense to all have the same mix. But for a 40 ft reef dive it hardly matters what's in your tank.
Well, ok, maybe Argox would not be good ;-)