Two summers ago my wife and I planned 3 days in the keys, with 2 in Key Largo and 1 in Key West. Too bad tropical storm Faye kept us in a FLL hotel for the first 2 days. We still drove all the way down to KW for 1 day.
KW is a hip town, especially the sunset party at Mallory Square. Definitely more night life. Duval street is interesting to walk. Diving there was so so. More like Veradero, Cuba. (Well, Americans don't know Cuba, they are banned. But Bay of Pigs was good diving.)
This March I'm taking the family, so including the 3 kids. We'll be diving/snorkeling in Key Largo. It's supposed to have great diving.
Since diving is your priority, you might want to spend 2 days in KL and one day in KW. Even though others might say to stay in KL the whole time, if you are in KL, might as well do a day in KW. So you can say you drove thru the many islands of the keys, and get to experience a bit of KW.