It depends, I have and will again. If the dive is within swimming distance of shore, there are no currents or wind, relatively shallow and clear water. I keep my dive to the anchor and if I want to move, pick the anchor up. I would more often leave my wife up top who is fully capable of operating the boat.
Then there is my kayak, I do not anchor it per say, if I set the anchor down for a moment I do not leave it. When ready to move, pick it up again.
But, I have made swimming shore dives up to two miles from shore (or entry point), with no boat and no BC either, just an inner tube float and flag and had a blast.
I do not know what other people can do, only what I can do, so take nothing I do as a recommendation for what you should do.
N