Speaking only for myself, I prefer to dive only with the weight I need. For doubles that means that I dive without weights.
If I did have to remove my gear uw, I´d make sure to hold onto my harness to keep from floating away, likely this would be inside a wreck or somewhere like that where...