The main things holding back the amount of diving I do are:
-Work obligations: I have to work Mon - Fri so that rules out five days of diving
-Cost - I can only afford one boat trip a week, but have discovered some great shore diving spots so that is less of an issue these days. The cold has been an issue as well and I have only just been able to afford a drysuit, so could only do two dives a day over the last few months, will be aiming for three a day when my drysuit arrives
-Lack of buddies: This was a real issue for me when I first got my OW. I was hanging to get back in the water but it wasn't til a month later than the club ran another DM lead trip. Then I showed up on boats a few times for instabuddies but just going on boat trips is REAL expensive but it is difficult to organise shore diving without a prearranged buddy. Anyway, I met my regular buddy on a drift course in late April and I have been diving every weekend except one since then. But yea, I imagine if I did not have a regular buddy I would have to cut down on my diving significantly.
As far as quitting goes, well I really hope I never have to quit. I doubt it would be voluntary....
Edit: Oops I posted this before I finished.
Anyway, I know a few people who have quit diving and the main reason is they are just not that into it. There is a lot of effort behind going out for a dive where I live - long travel time, it's cold, there are rough seas, strong currents, poor viz, etc, and for some people that outweighs the benefits.