Well, if everything possible goes sideways, the one thing you would like to be able to do is float. If your dive gear is all of your ballast, the only way you can make yourself irrevocably buoyant is to ditch it -- which means ditching whatever is left of your gas supply.
If you are sufficiently experienced to remain calm in the event of everything going pear-shaped, you may have enough poise to tread water and inflate a lift bag. Or you may have an equally experienced and attentive buddy, who can assist you with this, or with remaining buoyant. But you have added stress to a situation which, with newer or less frequent divers, really needs to be defused.
I think the decision to dive with no ditchable weight at all should not be taken lightly. That said, I do it -- but I also do 180 or so dives a year.