I only had 2 dives off Cayman Brac (18 off Little Cayman) aboard the
Cayman Aggressor IV in 2016; it was back in 2017 I visited St. Croix.
St. Croix Research Report Aug. 2017 -
Trip Report - St. Croix Research Report Aug. 2017 Note: a lot changed since my report. N2TheBlue got changed around (including the name), I think Cane Bay Dive Shop went through changes, someone told me a zip line operation opened, so if you take an interest in St. Croix, you'll need more up to date info. You may still get a feel for what a week on the island might be like.
If you're just going for diving, I think the Caymans are going to win. I liked St. Croix, and you can look at the photos in my old thread, but if it's just about diving, I'd head to the Caymans.
St. Croix is a really neat destination, if you factor in topside enjoyments but are a taste for the quaint and charming as opposed to a lot've whiz bang cruise ship excursion-style offerings.
You mention the wall being close to shore. I'm guessing that might be important to you for one of a couple of reasons; either you get sea sick easily and like short boat rides, or you're interested in shore diving.
To me, St. Croix is a boat diving destination with supplemental shore diving; it's not in the same niche with Bonaire or Curacao. The swim out from Davis Bay was long; the swim out from Cane Bay was way long. The Frederiksted Pier was really neat, but I wouldn't base a trip on just diving it exclusively.