My husband and I used to always anchor the boat and both go down as you describe, did that for few years.
Then one day at the end of a dive, as we were getting back to the anchor we noticed the line had a lot of slack and was perfectly vertical. Everything about the situation looked wrong I was trailing way behind but could see my husband started to swim as fast as he could to the anchor, and so did I. Before getting to the anchor it looked like someone/something was now taking the slack of the line and it was being pulled in the opposite direction. With one huge stretch my husband got hold of the anchor and shoved it as hard as he could down again in this new direction it was been pulled. It did grab and things got sort of quiet. We went up and encounter a really mean thunderstorm.
Things were fine when we went in, but sometime after we jumped the wind die signaling the proverbial "calm before the storm" with no wind or current at the moment the boat was not pulling the line, hence the slack in the line. The wind then suddenly change to the opposite direction turning the boat around and making it pull the anchor off.
That day we used every possible "lucky point" available to any person.
1- On a deeper dive with less than great viz, we would've lost the boat.
2- If we would've turn around a few seconds later, we would've lost the boat.
3- If the storm started a few seconds later we would've lost the boat.
then comes the part of "what if we lose the boat?"
For that if you are close to the coast (SE Florida) you swim but your boat crashes and probably takes someone/something in the process.
If your are far from the coast (Panhandle Florida) your boat may eventually be recovered unharmed but how about the divers?
Since then we try to bring a third person with us but that isn't easy so mostly we make solo dives. One person topside while the other person dives, most of the times we don't anchor now, follow the float with the flag and bring the boat to the diver at the end of the dive.
I used to have a 3rd person for the Panhandle dives, but now the boat is in Pompano and my search for a local person (or couple) is on. You'll think that would be easy, but that is an issue on itself.