I have a lifetime of going to Peanut Island on my grandparent's houseboat and then my aunt and uncle's boat. (They used to live on Lake Worth.) I would say snorkel and swimming is best for Peanut as it's mostly sand around the island. (Which, by the way, was mostly built from what I understand by the Lake Worth Inlet being cut in around the 1920s or teens.) I've been diving with family off the Palm Beach side of that inlet for decades. Poking through rocks down to around 30 feet, looking for cool, old bottles, etc. Obviously anchor near the island, in about seven feet of water or so, and use a flag! Roll off the boat and get to the bottom and stay there until your dive is done.
One of our favorite childhood memories was sneaking into JFK's old bomb shelter on Peanut, on the backside of the old Coast Guard station. In the early 70s, that was an adventure. Now, they've cleaned it all up and made it part of a museum, although I don't know how often it's actually open.
Good old family memories...