The diver needs to float with any attached ballast and misc stuff in pockets without the BC.
The amount of "float" is personal preference.
If you anticipate taking gear off underwater (for emergencies or another reason), then your ballast is dictated by the ability to be neutral underwater sans equipment (ala freediving). Such weighting usually doesn't provide much float on the surface - slightly above eye level with a full breath.
Of course, if the diver has ballasts in such a configuration, then it needs to be ditchable.