I wonder if the 'giving up' primarily stems from the 'difficulty', or simply a perception that the diving has a low return-on-investment in respect of enjoyment versus discomfort/ investment.
Diving in cold water is significantly more costly than warm-water; in terms of equipment (esp. exposure protection), training for that equipment (i.e. dry suit courses) and generally higher diving/boat fees.
Many people don't enjoy 'brain freeze' or 'stiff fingers' - a minority accept it in return for the chance to dive regularly at home, but many won't - they'll defer diving to warm-water on their vacation.
Being cold or uncomfortable isn't "harder" or "more difficult" diving... not in the respect of necessary environmentally-demanded skill development etc...
I think it really depends on a lot of different things. People who live in colder climes have to make a decision ... dive regularly in local conditions, or occasionally when on vacation. Each choice has its drawbacks ... and each is going to depend on why the person wants to get into diving in the first place. I think most people who elect to dive locally don't perceive it as a low return-on-investment situation ... particularly those who live in areas that offer a relative abundance of things to see ... which encompasses most cold-water coastal areas, I think. Those who dive in fresh-water locales, such as lakes and quarries don't have the abundance of marine life to satisfy their itch to dive ... but many find other areas of enjoyment that make the investment worthwhile. Not everyone dives for the same reasons, after all ... just talk to the folks who live in cave country, or in the Great Lakes region, for example. They don't at all dive for the same reasons as those who dive regularly in tropical or more temperate coastal areas. Return on investment depends a great deal in what aspect of diving interests you.
As for cost ... it costs me more to take a two or three week vacation to someplace like Indonesia than it does to dive locally all year long. Both are worth the investment, but for different reasons.
Water temperature isn't as big a deal as some here make it out to be ... if you're dressed for it. That's a matter of making an investment in equipment, certainly ... it's also a matter of making an investment in learning to use your equipment properly, and choosing the correct equipment. Almost any decent hood will prevent "brain freeze" ... I purchase a $30 hood annually, as a "Christmas present" to myself. Hoods are warmest when they're not worn out from use, and a decent hood will usually last for a few hundred dives before it needs replacing. Diving with gloves is more difficult, but most people don't have much trouble adapting ... humans are amazingly adaptable when they decide to be. The nice thing is that once you get used to manipulating your gear with gloves on, taking them off makes doing so seem ridiculously easy.
And yes, some people are more susceptible to cold than others. Some people just don't have any desire dive in cold water ... or find anything desireable about the notion of diving in cold water. On the other hand, if you live in an area where warm water is a plane ticket away, then the drawback is that you aren't going to be diving very often ... and that usually means your skills will never really progress beyond a certain point. For probably the majority, that's quite OK. For those who get the diving bug in a bad way, whatever it takes to get that next diving "fix" is worth the investment. Some will find reasons to move closer to their environment of choice ... others will learn how to adapt to the environment at hand.
I think, for most, it's less a matter of discomfort or degree of difficulty as it is a matter of priorities. For most local divers in my area, the complaints I hear aren't about cold water, it's about plankton blooms or storm conditions that mess up visibility ... or big tidal exchanges that make the best dive sites temporarily off-limits due to current. For those who come here and want to adapt from the warm-water conditions they were used to, the most common complaint I hear amounts to "I never had this problem diving in ________ (fill in whatever tropical locale they came from)". To which I reply, "Of course not ... and after a dozen dives you won't have it here either".
... Bob (Grateful Diver)