I think it depends on the dive op. Some may feel cramped in anything more than a 6-pack while others can be happy with perhaps twice that many. In my opinion, what makes the difference is when you are diving with a smaller boutique dive op you are generally diving with an entirely different caliber group of divers than those (not all) who generally dive with the AI in-house dive ops. It is human nature for new divers to think they are safer on a boat with lots of other divers (safety in numbers right?) but nothing could be further from the truth in my opinion.
In my opinion it's so much better (and safer) to the newbie diver on a smaller boat of experienced divers who all know how to watch out for themselves, each other and the group as a whole than to be a cattle boat of 15-20 divers who plan to dive a day during their trip with their other days spent renting a dune buggy, parasailing, visiting Mayan ruins, going on a tequila tour, swimming with captive dolphins and closing out their trip with day at the spa and a massage.
I will use 3P's as an example but I am sure other boutique dive ops deliver similar experiences. 3P's is known for the freedom it provides experienced divers... If you want to hang with the DM you are welcome to and if you want to venture out on your own keeping the group in sight you are welcome to. You dive your own profile with 3P's and no one is forced to maintain some silly underwater train of divers with locomotive and caboose DM's. As such, if a 3P's boat is at it max capacity of 10 divers, you can rest assured that at least 6 of those 10 are going to go do their own thing venturing off and rejoining the DM from time to time which leaves those who want to stick close to the DM with basically a private DM experience.
I well remember a newbie diver we met on a 3P's boat who was diving solo. He was an offshore oil rig driller from Canada and his wife didn't dive so he left her at the AI and went diving with us during the days. He wasn't shy and announced that he was new to diving to all on the boat and we all welcomed him and assured him he'd have an absolute blast. He hung close to the DM his first day (didn't hire a private DM). His first day he was the first to surface with the DM sending up his SMB and seeing him to the boat. Those 2 short dives that day lasted all of one day as the other divers (including myself) started explaining breathing technique, energy conservation, buoyancy, don't fight the current let it take you on a ride.. Etc Etc Etc. By day 3 he had gone from an initial 35 min bottom time to over an hour. On day 3 I surfaced before he did and met him at the ladder as I was on the boat and asked WTF are you doing surfacing after me? Great times. Great times. Great guy we hope to dive with again but our visits haven't matched up with his since. Some day.