I see no real problem with leaving the dive boat unattended as long as certain conditions are met. Those who say it should never be done are probably right....for their areas and level of experience. In other areas, it's perfectly fine. In some areas, it's easy to see the boat from the bottom. Honestly, nowadays I would worry to much about theft and vandalism to leave the boat alone but each to his own.
Yes, it’s more about the boat and your level of seamanship. Sailboat owners (such as me) regularly leave their boats at anchor, unattended. Some do it for days on end. With the right anchor set-up it’s perfectly normal behaviour - not at all deviant.
On the other hand, a small inflatable would be best tied to a mooring ball.
I’m moving to the Caribbean soon, and can’t wait to dive at the locations of my choice, in weather and sea conditions that I’ve assessed with all the nav kit on board, at my own pace, and only when I feel in the mood. Not having to follow some sketchy dive guide, being able to potter about on the surface and descend when I’m ready, not when I’m told to, are further benefits.
If I think the on-board oxygen facility, is a backup worth having, then I’ll buy a set. It costs buttons compared with boat ownership.
The OP’s precautions are sensible. To that I would add a hand-held VHF radio, and a bungee cord to attach your BCD to the boat if you’re tired after surfacing. Finally, if you’re worried about retrieving your anchor, use a tripping line with buoy.