I was out there just over a week ago (on a Tuesday) and there WERE a couple of fishermen trolling around on a boat, casting all over the place, even amongst the two swimmers that were there.
That was probably the owner. I've been there once when the owner and a buddy were out fishing in a boat. I believe generally he does not let others fish there. Of course I found it funny because he's got almost no fish in there!
I've never seen swimmers there before. Man I hope they don't open it to swimmers. If that happens I won't be diving there from late June through mid Sept. Other times of the year would be fine since it'll be too cold to swim.