As the operator of a 37 foot power boat lets look at the other side of this.
From more than 100 yards away in the open ocean you are invisible as a diver - even with a dive flag. If I am driving directly toward you odds are I won't see you until you are 20 - 30 yards away (depends on the boat but I don't have a flying bridge so even closer for me) - and that is if I am paying very close attention. Rough water or waves of any kind make it even worse. I have managed to drive over a 20 foot log 16" through that was floating just barely at the surface in relatively calm water. This in an area where floating debris is a constant hazard so I was paying extremely close attention. Didn't see it until it was 10 feet from the boat - too late to do anything but pray to the gods of bent props.
You are very hard to see swimming on the surface. Screaming will have absolutely no effect at all - none - I can't hear you. There is a reason boat horns are very loud. Waving a dive flag - good idea - making white water (swimming hard enough to splash etc.) good idea, unless there is a lot of white water around you. Anything that breaks the pattern in the water. All of this depending on where the sun is - if I am driving toward the sun then you are essentially invisible no matter what you do.
Descending is by far the best option. If he was coming directly for you from that far away just as likely that was his course and he never saw you as that he was aiming for you and was trying to scare you. A boat is not like a car it wanders on course so the amount that you can swim perpendicular to a boat's course from 100 yards away is well within the amount that a boat will track so the fact that it seemed to change course toward you while it indicates that he might have been aiming for you it is not conclusive - could have been just bad luck that the boat tracked that way.
If you are in a place where there is likely to be, or even the possibility of boat traffic - you MUST be able to descend at any time. Otherwise don't dive there. Depending on a boat operator seeing you is a bad idea. It might happen, but it might not, and while it would likely be the operators fault in a boat/diver collision the boat wins every time.
Glad this worked out - as you say it could have been much worse.
I appreciate what you are saying and know what you mean when you say the boat always wins, this is undeniable.
However the fact remains that it is the boat captains responsibility to watch for hazards, particularly divers in the water close to shore in a well known dive area. This boat is local to the area and has been plying these waters for 40 years. And close in at Rockport Harbor is hardly the open ocean.
Where we dive is not a particularly high boat traffic area but it is a popular dive spot and all divers are required to tow flags so a captain who is experienced in these waters should know what to look for. And a captain who doesn't should be steaming slowly enough to watch for divers when traveling that close to shore. It may not be easy to watch for divers but that doesn't relieve you of the responsibility of doing so, even if it might be only their survivors who come after you.
Finally, as was already mentioned, there are plenty of circumstances that can arise unexpectedly forcing a diver to the surface. It seems to me that by your logic we should not be diving in the ocean at all as boats can come by at any time. For that matter most lakes around here have higher boat traffic than this beach so perhaps we shouldn't dive there either? Your reasons for not seeing a flag have nothing to do with the size of the boat, in fact a smaller boat lower to the water should have an even more difficult time. If boats can't be expected to act responsibly and respect a flag...why do we bother to tow one?