If there was no wind, and the engine was off, then people on the boat should be able to hear people in the water yelling a few hundred yards away. If the wind is blowing it makes it so much harder to hear due to waves slapping the hull and the wind noise itself and even whitecaps, so if they could not communicate by yelling it was a good distance and it was windy. This sort of situation would lead me to suspect that swimming for the boat is not a good, practical option since the boat could easily be moving at 2 mph.
It might make sense for one person to drop gear and make a sprint after the drifting boat, but it is by no means a safe thing to do and risks permanent separation from the group, loss of buoyancy devices, flashlights etc.
So if we are talking about what to do in these sort of situations, I think it is probably safer to stay together, especially if you have working flashlights and you are reasonably sure that you will be reported lost and there will be aircraft and boats deployed for your rescue (eventually).
Also if the water were frigid and you didn't think you would survive a 8-11 hr soak, then that would push me harder in the direction of "swim for it".
I don't see how any armchair quarterbacks can reasonably know if the swimmer could have made the swim to the boat.