My wife and I went to Isla Holbox last August to swim with Whalesharks. We loved it. Granted, it is more difficult to get to than Cancun or Isla Mujeres, but I think it is worth the effort. Holbox is like Cozumel or Cancun was long before they became developed as they are now, especially Cancun. It is simple, laid back and the people on the island are very genuine, kind and honest. No rat race there. The island itself is beautiful. I thought it was a very endearing experience that anyone that has been to Cozumel or Cancun should do at least once. Besides the whale sharks, the thing that impressed me the most was the food. I had two of the best meals of my life on that island. One was at Edelins pizza where they have the best lobster pizza ever. Then another place called Zapata's had the best grilled fish ever.
Granted though.....there's no night life and not much to do besides the whale sharks, maybe some fishing and eating. Every night pretty much everyone on the island goes to the beach to watch the sunset. That was always an impromptu party and alot of fun.
I didn't find that it was very expensive there. We were there during the peak season and I found everything to be quite reasonable. Food was downright cheap.
With all that said...... I would recommend a trip to Holbox.