The minute part of the undersea world that recreational divers see is the exact opposite of the vast majority of the largely lifeless mud bottom of the ocean. The continental shelf is only 8% of the ocean and runs between 100–200 m or 330–660'. Divers routinely visit less than 1% of that. I'm not arguing that we can't visibly impact some of those tiny areas, but is is far from the big picture.
The ocean barely notices a few clumsy divers that disturb the bottom. The same divers that endlessly fuss about other divers conveniently forget about what their boat anchor does, let alone what drag nets do that supply their fish mongers.