Should you have said something to him during the SI? Of course you should have!
You should have chatted with him and got to know him. That's what people do on charter boats. They be friendly and enjoy each other's company.
The guy didn't put any of you in danger. Not one bit. The only thing him steering with his hands does is make HIM go through HIS air faster. You're not obligated to go up with him....
I normally dive my own boat so am only on charters on vacation. But I just got back from the Keys/Jupiter and did 3 charter boats, everyone was super friendly. It made the trip so much better. I don't care that a photographer sunk down and kind of landed on me when we were at 90 feet on a drift dive. We quickly moved apart. I don't care that someone else got in my photo at the Chris of the Abyss statue. I'm sure I made a bunch of mistakes and got in people's way too. Enjoyable people won't get pissed off over it.
Everyone is just there to have fun. Don't take it so personally....