The only answer is "it all depends." Really, spearfishing dives should be an entirely separate category because shark encounters and not unexpected, and the way they approach can be aggressive, but spearos generally know how to handle themselves and do not have the same emotional reactions other divers might have.
If I am on a non-spearing sightseeing or photography dive, a shark swim-by would not faze me as that is pretty normal behaviour. Usually, you see them only at a distance and are lucky to even get a curiousity but "not really interested" swim-by. I am thrilled by these encounters and try to stay with the sharks, but that is not usually possible.
On the same dive, if I am approached quickly by a shark clearly interested in "me" and showing speed, quick movement, or "fins down" posture, I would be more concerned, and the closer the approach the more concern I would have. A "bump" would cause the most concern. But, I am not concerned that the shark is intent on "attacking" me at random. Rather, the concern is because I understand that sharks approaching this way have been artificially stimulated into feeding behavior, which puts me at higher risk.
I have experienced all of these behaviors at shark feeding/scenting sites, and if that is what I signed up for, I have no issues.
I would be upset if I encountered this behavior on a regular dive site not dedicated to regular shark feeds.
