I don't think permanence matters, IMO. Where do you draw the line?
My bellwether is whether or not my 'touch' might at all influence something organic. If it will, then I'd best not. It's too easy to start backing off of different gray lines from there.
(Well it might wipe off a slime layer, but I'm sure the organism replentishes that constantly...well, it might potentially lead to a bit of an infection, but I'm sure it won't lead to any permanent damage...well, it might leave a little mark/scar, but I'm sure it won't be fatal....well, it might kill that little spot, but not the whole colony organism....well, it's only one fish/shark/whatever, there are thousands...well, it's only one species, there are others much like it....hey, why's this dive so damn empty and boring???)
Of course all divers, no matter how careful with their trim and buoyancy, do end up intermittently 'touching' reefs with fintips, perhaps fingertips, or at the very least explosions of bubbles, which (especially for Nitrox) can't exactly be the reef's expected environment. You can't help impacting simply by being there. I just don't see any reason not to minimize my impact as much as I reasonably can, regardless if the site is a ship that's just been sunk, a rig that might only be there for 20 years, or a reef that's been there for thousands.
So the structure isn't permanent - do you think the next diver wants to see your sitz marks on the metal? May as well get your dive knife out and carve "I wuz heer" if "permanence" is your only benchmark. (Blue Grotto spring dive in Florida comes to mind. Would be a beautiful dive if any portion of the limestone walls wasn't covered with graffiti.)
Not to single out any particular type of diver, but it always amazes me to see certain semi-pro divers that supplement their writing/photo income by being 'naturalists' for one preserve or another (e.g. the Flower Gardens) brace and touch for pictures when they're in the water. Sometimes they're just doing it in sand...sometimes not. Kind of goes counter to their whole 'conservation' message they're supposedly proselytizing at the same time.