Yeah, one. Earlier is good... to a point. The wall and reefs run basically north and south, and when you dive your first dive, you are down deep with the wall or reef east of you. If the angle of the sun is low (in the east, of course), then you are in shadow, and it's dark. Also, the lower the angle of the sun, the more of it gets reflected away by the surface. It's the reason I don't do afternoon boat dives very much; by time for the second dive, the angle of the sun is low enough that things at depth begin to gray out.
Also keep in mind that the reefs are BIG. I have been to Palancar when there was a veritable traffic jam on the surface, and I didn't see a single diver other than our group at depth.
Of course, there is always the counterexample. I once had this $%@#!! numbnuts idiot diver from another group pass under me in a swimthough out on the Santa Rosa Wall and pin me to the overhead. He actually grabbed my foot to pull himself past me! It's a good thing for him that they don't allow firearms in the water! ;^)
Oh, and one more thing, just getting out there early is no guarantee that some other dive op won't dump another load of divers right on top of you halfway through your dive.