Not an easy call. It's hard to fight geography and weather. If you are doing some diving in other ports, maybe do the stingrays for something difference. I agree it's something you should do once. If you think you will get back to Cayman for a land based visit, which I would recommend, maybe save Stingray City for that trip.
Possibly you could do Stingray City and also do a shore dive at one of the shops with on site shore diving, depending on the schedules. It wouldn't be the same as doing a couple boat dives on one of the walls, but sort of gets you both. Eden Rock is often suggested as it's close to the cruise pier, but it's also often described as well worn. The shore dive at Turtle Reef is very good, used to have the Divetech shop there and is now Sun Divers which gets good reviews too. That would be a longish taxi ride and there can be traffic, so that would take longer, possibly too long. A compromise would be Sunset House, a short taxi south of town (you could walk, but I wouldn't with gear.) Some decent shore diving and a great outdoor restaurant/bar if you have some extra time.
Or you could just go to Sunset House or Turtle Reef and spend your whole time there. Sunset House is a good dive hangout, you could dive, eat, look at the pictures at Cathy Church's shop, even walk to town for a bit. From Turtle Reef, you could walk to the actual Turtle Farm or get a bite at the Cracked Conch.