I dived solo off my own 18' center console many times over 3 summers mostly on an artificial reef less than a mile from shore. I made a few attempts to find a watcher by advertising on local fishing forums and diving FB pages but didn't get much interest.
I always made sure to put out plenty of anchor line and tried to avoid rough conditions where the boat might pull the anchor or I'd have trouble getting back to it working against a current. I'd thoroughly check the placement of the anchor, sometimes removing it from the sand and securely attaching it to a wreck, often tying my reel directly to the anchor line despite the potential drawbacks of doing so.
I used an anchor that was larger than necessary, understanding that if current picked up I might not be able to unhook it without surface support so it might be necessary to cut the line to release it. I always carried a spare but I recovered several anchors from wrecks so there was no shortage of anchors and I never did have to cut it free.
One time I approached the bottom and found the anchor hadn't grabbed and it was briskly moving above the sand. After unsuccessful attempts to try to get it to grab hold, I tucked it under my arm and was pulled along the bottom by a strong current. It was a thrill ride. I asked myself if I remembered any anchored boats in the vicinity whose captains might not appreciate an unattended boat bearing down on them, and what might happen if I accidentally let go of the anchor.
I would lock the boat ignition key, car keys and cellphone in the glove box and stash the key somewhere just in case someone came across the boat either with good or mal intent but it was a relatively desolate area with few boats around and I wasn't down all that long, maybe 30-40 minutes.
Never had an issue however in the back of my mind there was always a bit of irrational fear that as I ascended the line something would go wrong and the boat would be coming down to meet me or I'd find it on the bottom. Always double check the bilge pump is turned on and operating before you get ready to splash. There were occasions where I was enjoying a nice dive yet there was that nagging thought in the back of my head "did I turn on the switch?"