I dive non-ditchable weight all the time . . . when I'm on a surface supply umbilical with hard line coms, a safety diver geared up ready to come get me, and something like six people supporting me from topside.
If you're doing basic recreational scuba and you need lead, and virtually everyone does, making it impossible to get it off you is crazy. What you may or may not be able to "swim up" is irrelevant. First of all, it assumes you are uninjured, non-panicked, wearing functioning gear, and otherwise able to swim.
In reality it is what I can swim up, since I'll be rescuing you. Don't make me swim it up. It also assumes that once on the surface all the gear will work, textbook rescue techniques will work, and you will be non-panicked.
In reality, the sea will be pounding us, you'll be unconscious or panicked, and I'll be wishing for a way to instantly make you float much, much higher in the water. I won't have it.
When you get to the boat, the people pulling you in will be wishing that lead wasn't on you. They might not be able to get you in without figuring out where the hell you put your lead and maybe giving up and stripping all your gear in the water (I know the theoretical way to do it, but real world and theory do not always meet).