I was in St. Croix earlier this year and I thought the Frederiksted Pier was sensational. Southern rays, eagle rays, turtles, and great macro stuff on just about every dive. Best of all, you can dive the pier on your own schedule (provided there is no cruise ship at the pier, which you can check online since the cruise ships have a fixed schedule). So if you wanted to dive early in the morning, you could get easily two dives in at the pier and be back by noon, before your wife knew you were gone.

I stayed at an Airbnb in Frederiksted within sight of the pier, and dove the pier at night and at sunrise a bunch of times. I did a shore dive at Cane Bay which was good for variety's sake, and spotted a reef shark in the distance, but Cane Bay didn't hold a candle to the pier, IMHO. Having said all that, Frederiksted is not really upscale. So you might want to drive to the pier from wherever you're staying.