Twelve feet is a nice depth for snorkeling. If you were snorkeling, everyone could meet on the surface every couple of minutes, and stay together that way.
Forgetting about depth, it is not easy to keep a team of more than three divers together. In over 40 years of diving, I have only been a part of a team of more than three divers on three occasions: the first two were on a "resort SCUBA course," the first day I ever used SCUBA, and the group was led by the instructor. On the other occasion, I dove with a couple and another guy who was a friend of the husband. As soon as we got underwater, the woman had some trouble clearing her ears, and the husband and his friend swam off for parts unknown. I had a nice dive with the woman, who just needed a couple of minutes to get her eustachian tubes to work. Fortunately, we were in a quarry where no flag was required.
My recommendation would be to buy another flag, and dive as two buddy teams. Then none of these issues will come up.
If you MUST dive as a team of four, then the flagman is the leader, and should either swim backwards or turn around frequently to look at the rest of the team, so that he can maintain a pace that everyone can keep up with. Everyone else must follow the flagman wherever he goes. The team must all stay together, and if anyone gets separated, they must all surface - because, if anyone is separated, there is at least one who does not have a flag. And if a flag were not necessary, you would not have had a flagman in the first place.