You may not be hurting it but it was clearly agitated.
Imagine if every diver that day did the same thing to this octopus. They don't live that long to begin with. I don't think they need to be bothered so much.
Actually the octopus was not very upset.
And as I said earlier, people need to know what to look out for, to make sure the animal (and your fingers) are safe.
This particular octopus, a Two-Spot Octopus, will have a lot more intensive color changes, with warm colors (red, orange) when it gets upset/angry. It will also change the shape of it's skin, creating "spikes" all over it's body, to scare of predators. Look for that next time you see one..
This one was more interested in catchin the weird creature in front of it (my hand). With any sign of serious discomfort for the animal, I would of course aborted the "playtime"... The reason he went back into it's hole the first time, was most likely because of fear for other animals around us. As soon as I came back to him, he came out again. If he was afraid or felt threatened, he would have stayed. Octopus are smart creatures with an awesome ability to learn by their mistakes.
The reason I posted this video, was because of the surprising tolerance it showed, for such a long time.
The Two-Spot octopus is the octopus you will see in most home aquariums, just because of it's "nice personality".
I am not a marine life expert, but I have a lot of interest and some knowledge about certain creatures we bump into, on land and under water. And as I said earlier, if you don't have the knowledge, don't touch.