I think family and cost can have a lot to do with it. My mom doesn't like me diving, and she HATES the thought of cave diving....we were in Ginnie springs for around an hour, and she didn't like that one bit. You've got to remember that someone who hasn't gone through the training doesn't understand the difference between trained and untrained divers all that well....they understand it's safer, but until you go through the training there's just no way to understand it IMO. I bought my can light from a guy who's wife didn't want him cave diving now that they had kids. He had over 500+ cave dives, most involving scooters and stages, but both of them decided that it was still too much risk with young children.
Also cost. Even once you front the money, drysuit seals need to be replaced about once a year ($200), Nitrox fills (Usually $20-$30 for a set of doubles), gas to get to and from dive sites (usually runs me at least $50 a weekend), entry fees (can be as high as $40 at places like blue grotto), and it just keeps adding up.