I did my last cave diving trip to Mexico in a drysuit (75 to 77 degree water). I wore a base layer and a layer of polarfleece sweats from Lands End, and I was almost too warm (lovely feeling!) I do have a couple of things to share:
1. I agree that the suit will feel strange and it will take a dive or so to get used to managing the air in it with so little undergarment. My buoyancy was a little off at first.
2. When a drysuit has a failure, you are much more uncomfortable in wet undergarments than you are in a wetsuit.
3. From my experiences travelling with a dry suit, they choose the times when you are further from home and most want to do the dives to cause trouble. Take your wetsuit as a backup.
But, even with all that, I was so comfortable diving dry in that warm water that I'm seriously considering taking my Fusion with me EVERYWHERE from now on.