jiveturkey once bubbled...
I wear contacts too and I've never had a problem. I considered getting the prescription lense in my mask. I have a single lense mask so I'd have to but a double for the prescription lenses which adds more expense. No thanks. I'll stick with my contacts.
So have I & never regretted it. No problem with lost/swapping mask drills or swimming with open eyes in mask hunting drills. That said, wearers of Conventional contact lens (i.e. replace annually, < 45% H20 content) or RGP (Rigid Gas Permeable) hard lenses should be more careful as these would be more prone to accidental loss. I wear mid-water (55% H20) monthly disposable Toric lens and clean them daily.
You could potentially get bacteria but I'd say the salt water is probably not too bad on your eyes. Saline solution may be sterile but it's really only salt water.
One of my past careers was as a business mgr. for a leading contact lens & solutions manufacturer so I can speak to the above. Hopefully a diving optometrist or diving doc can jump in too.
The infections you can obtain from pool, lake or ocean waters could be a serious concern so removal/cleaning of lenses after diving would be a prudent step to take. If practical, even between dives would be advisable. Why? Because the potentially infectious organisms can infiltrate into the lens matrix and thus aren't rinsed from the eye. Remember that all contact lenses have a percentage of water in them and will draw in moisture from your tear film to stay lubricated and comfortable on your eye. Even if you have not had a mask clearing/loss episode, a small portion of moisture in your mask - including residual mask cleaning solution - can infiltrate into and irritate your eyes and stay there in the lens.
Don't think you are any safer in Lake or Pool water. Chances are the level of sewage treatment into your favourite lake is not what you think. Your local swimming pool may be chlorinated but has numerous containments - is there any children or Seniors' water classes at your pool? I'll let your imagination work on that one.
The level of risk is low but as in your diving, you want to minimize your risk. If you are wearing extended wear lenses (ones that you leave in at night), you are at higher risk, since you would not have the habit of taking out your lenses daily to clean/disinfect. This is a very popular modality in Quebec and may become more common as new materials are being introduced that make this more feasible.
I always have a backup pair in my Save A Dive kit along with my spectacles and carry cleaning solutions with me.
If in doubt about your lenses or diving/swimming with them, talk to your Optometrist!