We often hear that. Inevitably, the person saying so has confined his/her Grand Cayman diving to the West Side, where most of the dive ops and hotels are located and where the vast majority of the diving is done. If you only dive on the West Side you will not be impressed. The coral there, particularly on the shallow sites, has suffered a lot.
The best Grand Cayman dive sites are on the Eastern side of the Island and on the North Wall. The East is serviced only by Ocean Frontiers and by Tortuga Divers. Experienced divers will prefer OF.
The North Wall diving is very weather dependent, one reason divers can come here for an entire week and never see it. Many divers seem unaware of its charms and make little effort to get there. When the weather is right, you can get to the N Wall with operators such as DiveTech, Neptune, Wall to Wall, Indigo, & Over the Wall.
IMHO, a week on Grand Cayman diving the N Wall and the East End with 2 experienced dive ops is at least equal to, and arguably better than, a week at one of the all-inclusives on the Brac or Little Cayman. I have done all 3 more than once.
I will refrain from comparing GC to the rest of the Caribbean - it's a large area and I have encountered all types of diving - good and not so good - in other countries. However, if the weather cooperates, you will not be disappointed with GC.