I thought diving was a lazy-man's sport!?! (or lazy woman's).
I've done 30+ dives at the jetties, and have yet to find it necessary to circle the point.
I usually dive to the area around the sunken boat, maybe 200ft further, turn, and work my way back to the entry point. You'll see 10,000 more things just taking your time and looking for the small critters.
There's nothing illegal about diving the jetties as long as you have a flag, and don't have a speargun. I suspect the fisherman was just protecting his spot by feeding you some misinformation.
This happened in 2008: