Gently....
And remember, many things work well so long as conditions remain favorable and events proceed as anticipated.
But occasionally conditions do not remain favorable, and events - sometimes tragically - do not proceed as anticipated.
I recall such experiences as clinging to a wall during a downwelling current, and being thrown and rolled across a reef on Okinawa trying to exit the ocean after have misjudged the falling tide. Other anecdotes I've read involve divers who spent the night clinging to rocks in the Gulf of California, their wetsuits were in tatters after having been scraped across the sharp rocks all night.
Gloves are equipment, they can be safety equipment.
You can do stupid things without gloves, and stupid things wearing gloves.
I carry a signal mirror because things may not go as I plan. I carry a SMB because I cannot control currents, nor swim against them for very long. I wear gloves because I've had my hands torn up on sharp rocks.
Clearly, stupid divers harming undersea life is a problem. But banning gloves (a) does not always solve that problem, and (b) makes a presumption that things will always go the way divers anticipate they will go.
And then things don't.
Gloves are safety gear when things go wrong and you need your hands.
YMMV.
