Fortunately, I do not wear contacts, BUT....I did get certified in Cuba and logged 60+ dives there, some with contact wearers.
I was informed by my instructors that contacts are pool safe only(even that is iffy) and should never be worn in open water. Salt water is a corrosive and can damage the surface of the eye when it gets between the contact lens and the eye. If you let the salt water dry at all once it's trapped there, the salt will crystalize and can easily scratch and even cut the surface of the eye which can lead to nasty infections or worse. Chlorine in the pool can and most often will act much in the same manner as salt in salt water when it dries.
In fresh water, you have the issue of fresh water bacterias and contaminants getting between the lens and the eye. Granted there is less of a risk to eye damage in fresh water, but the problem still exists to a point.
These instructors always educated students to the issue, but also stated that if they were sure to rinse with fresh water and follow it by salene solutions (or simply take out the contacts) once out of the water, it shouldn't be a problem. They, as well as myself, realize that prescription lenses for masks aren't cheap and for some simply aren't practical.