I have dove both several times and here is my take.
The diving in Cozumel is very good and one might say if you really like drift diving, then it's great. You don't have to worry about making your way back to the boat. In my experiences the vis has been very good as well as the marine life. All in all the diving is really really good. And of course you have lots of places to stay at all different price ranges as well as all kinds of eating options. Really a lot of variety topside for just about everything. I've always enjoyed Cozumel.
I have dove all three Cayman Islands. Grand Cayman was good, Cayman Brac was very good to excellent, but Little Cayman outshines them all including Cozumel. We were there in May and I have no doubt the vis on most dives exceeded 100 ft easily. The reefs were very healthy with all kinds of sponges, corals, and covered with fish. I was able to get great video of sharks, turtles, eagle rays, southern stingrays, big free-swimming green morays, tarpon, all kinds of schooling fish, etc. You just can't name them all. But I know you can see that lots of places.
What I think makes Little Cayman different is the fish almost seem tame. They don't shy away easily. If I approached pretty much anything slowly, I could get right up to it, film it, and whatever it was didn't seem to mind at all. There are even grouper there that will swim after you and want you to pet them. If you don't pet them they will follow you until you do.
Now getting to Little Cayman takes a few layovers but it's worth it. It is more expensive to stay there as many places are AI. But as in many things, you get what you pay for. We stayed at the Little Cayman Beach Resort. It cost twice as much to stay there for a week as say Scuba Club Cozumel which I like. But it's like comparing apples to oranges. The rooms are Scuba Club are clean and comfortable and that's really all I need. But the rooms at LCBR will blow you away. And the food at LCBR is almost worth the price of going. Unbelievable! And of course you get 17 dives for the week at LCBR while many operations on Cozumel are 5 days of 2 tanks diving or 10 dives for the week.
I would say you need to give Cayman Brac or Little Cayman a go and see what you think. Going there won't ever make you not want to go back to Cozumel but it will make you want to keep going back to the Caymans!