It depends entirely on the critter involved. Of course I'll touch seaweeds (I'm a phycologist by training). There are many invertebrates that are quite touchabnle (some even solicit it) and as long as you know how to handle them it is not damaging.
The best advice is don't touch unless you have reasonably good knowledge of an organism's natural history and sensitivities. If you are reasonably knowledgeable, and your actions do not damage or stress the animal, I see no problem with it.
I watched a giant kelpfish male guarding its nest for about 2 weeks. At first it attacked me if I came close, so I backed off. After a few dives he became quite accustomed to me and would even swim over my outstretched hand so it rubbed his body. I could touch him and he appeared to solicit a gentle stroking whenever I returned to the nest.
Anyone who has seen fish rub their bodies against rocks or the gravelly bottom realizes that they can tolerate a lot of contact. However, I don't want to touch them unless it is obvious they are interested in it and solicit it themselves. I'm usually too busy taking pictures to touch anything anyway.
Another case. I went down to Magdalena Bay as a naturalist to help interpret grey whale natural history and behavior in their calving grounds. You've all heard the stories about how visitors pet and touch the whales. I was determined not to myself. However, it became so obvious that one female in particular was soliciting human contact that I finally did stroke her on the offered belly.
This whale came up to our Zodiac and turned over to allow people to touch it. When we tried to leave after the maximum time allowed, the whale came over to us, partially lifted the Zodiac up and "towed" us back to te original site. Then the next day it came right up to us again. No question that this whale enjoyed human contact.
Again, in general it is best NOT to touch unless you have good knowledge of the species, and sense that it is not frightened (or damaged) by any contact.
