I wouldn't waste my time on anything south of Islamorada unless you would be interested in Diving the Vandenburg off of Key West. Just my two dollars worth.
I've dived a fair bit west (south) of Islamorada and I have found it quite enjoyable. I've also dived the upper keys, and liked it, so I'm not saying not to go there too. Here are some things I like in the lower keys:
1) I often dive out of Marathon, and we're often the only boat on the reef (six pack). It's nice not to be following or running into a pack. I like being able to come around a corner on a reef and have it be "undisturbed territory."
2) The Thunderbolt and Flagler's Barge are both fun and interesting to me (the Barge is very shallow; the Thunderbolt is a ~100' dive).
3) Sombrero Reef is a really nice spur and groove reef, with lots of relief, and can provide two dives back-to-back (one east, one west) (note that there are usually other boats at Sombrero).
4) Looe Key is like Sombrero, but "more." Really nice diving, I think.
5) Some of the other reefs out of Marathon are not as spectacular, structurally, but I still find them very enjoyable. No crowds (as mentioned above), plus plenty of life. I've had a few "personal first" dive finds there, such as octopus and (diving with) dolphins.
So in my opinion, definitely not a waste of time. If I were going tomorrow I would probably dive some in each place.
Blue Sparkle