I mostly with the above with the caveat that I although I grew up on the island it has been a 3 years since I was last there. I thought that the diving off the dock at Castle Comfort was amazing. I saw lots of flying gunards, a short nosed batfish and tons of octopi on night dives.
Fort Young is definitely a better place if you want to experience a little more of the local culture and people and are willing to leave the "resort" (neither one really is

) Roseau is a fun little town and you could always stop ny my alma mater St. Mary's Academy and tell Brother Wakeham Josh Dobrow says hello. Of course I was such a holy terror that might not be a good idea. However, the Botanical gardens are kind of neat and there are some neat little stores and stuff. I hear the waterfront has beccome very touristy to service the cruise ships and that is a shame but I am sure you can find some nice handmade stuff that is not total junk. There is not alot of nightlife during the week but there will be things going on on Friday and Sat.
If you stay in Castle Comfort you are not that far, but you do NOT want to drive at night, especially of you have had ANYTHING to drink, and the cab situation can get sketchy late at night. It is not a bad base of ops if all of your non diving time will be spent doing hikes etc but you will be a little more isolated.
Feel free to PM me or ask any other questions about the island.