I am going to throw out a suggestion that is not in the Lesser Antilles but it does hit some of your requirements and it provides an opportunity to enjoy multiple Caribbean experiences.
Cayman Brac is a sister island to Grand Cayman. It is small and not very developed. The word Brac means "Bluff" in Scottish and hiking, rock climbing, and caving are popular activities. There are also nature preserves and a bird sancturary. I don't think there is any rafting but you can do ocean kayaking and the diving is truly excellent. There are only 2 dive operations on CB (Reef Divers and Brac Scuba Shack) but they both have very good reputations.
There aren't many places to stay and there aren't many restaurants or grocery stores. We like to stay at the Cayman Brac Beach Resort which is a small, diver-dedicated, all-inclusive resort and Reef Divers in the onsite dive op.
One of the nice things about visiting any of the Cayman islands is that it gives you an easy opportunity to enjoy a multi-island vacation because each Cayman island has unique attractions. You would have to fly into Grand Cayman anyway and it is a very developed tourist destination, but it offers great dining and a world class beach. And there are undeveloped, beautiful areas on GC to explore. There is a hiking trail (the Mastic Trail) which is lovely but consists of about 2 miles of meandering through the local flora, and I don't think that is what you are looking for. There are also botanical gardens and a Blue Iguana preserve; as well as nice diving options, and beautiful, remote snorkeling sites.
Little Cayman is the smallest sister island to GC and it offers the very best diving in the Caribbean IMO. Gorgeous, dramatic walls and abundant sea life and beautiful colors. It is a very tiny island but it is more developed and upscale than Cayman Brac because of the spectacular diving - yet it is still a small, quaint place. It is home to a Booby sanctuary and a marine research station. You can also kayak to Owens Island which is a small deserted island. We like to stay at the all-inclusive Little Cayman Beach Resort with Reef Divers as the onsite dive operation, but there are several options for accommodations and dive operators on LC. You can do a lot of walking on LC but no hiking or climbing.
The Caymans are expensive, but you can control the costs with planning; like renting an apartment or condo on GC and cooking some meals, or staying at a reasonably-priced, all-inclusive dive resort on CB or LC.
For your interest, below is an article I posted comparing the 3 Cayman islands, in case you would like more information. It is very easy and inexpensive to travel between the 3 Cayman islands. Good luck!
The differences (for a diver) between the 3 Cayman islands - Cayman Islands Forum - TripAdvisor