Clearly you have never used a rubber belt with a wire bail buckle. The only one of the things you list that such a belt does is , "slide up or down during a dive," and then only when you want it to in order to adjust pitch. I'd say the same for the SeaQuest, but it does reguire tightening on occasion, which is incredibly easy and safe to do.
I agree, when only a few lbs are in play, who cares where it goes.
What is it about cave diving that makes is, "not one of those circumstances?" And remember we've already dealt with the BS about loss and entanglement.
I've had the chance to examine, in detail, over 2000 diving fatality cases and there's one thing that blasts through all the confusion and clutter: rarely does a diving fatality stem from a single cause, it's almost always the result of a multiple series of cascading failures. And there's one really important lesson that I've learned that I credit with having saved my bacon a few times: the effective ability to deal with any one of the items will stop the chain reaction. Being able to remove and replace ones rig is just one of those items, not some end all/be all that defines you as a "good diver" or a "bad diver."
We all evaluate what we need for personal and team safety and then we and our team live and dive by those choices. We choose to be diver neutral/rig neutral, it is one of our core values, it costs us nothing, it does not obviate a DIR style balanced rig, and with the proper selection of belt (or harness if you'd rather) it does not create any risk. I can not, for the life of me, see why anyone would do otherwise.
The entire objection to being diver neutral/rig neutral seems to stem from the very reasonable rejection (a rejection that we made almost 50 years ago) of an unsafe type of weight belt buckle without any consideration of the rather simple alternatives available. Get a rubber belt and try it, you may be surprised, and the life you save might be your own, might be your team mates'. Or it might never be an issue, but, if nothing else, it will help your trim.