If you are talking about how many people I stay with while underwater, I prefer only 1 buddy because the vis is often bad enough where I dive that it becomes really hard to keep more than one other peron in sight. With more than one buddy, it becomes way too easy to get separated from one of them. (However, in good vis, it's nice to sometimes dive in a group of 3 or 4.)
On the other hand, if you are talking about the number of people going diving at the site altogether, I prefer more than 1. It is fun to go diving with a group of people; we just split off into 2-person buddy-teams when we are in the water. I feel safer that way, because with a group, there is usually someone on the surface, and if something happens to me or my buddy, I know there will be other people nearby who can help deal with the situation. If just me and one other person dive at a site, I worry about not having anyone else around who would miss us if we didn't return to the surface in a reasonable amount of time, or help us out in the event of an emergency.