I used to think so too...but don;t take my word for it...here's something form Dr. Jolie Bookspan.
"Helium has greater thermal conductivity than air. Undeniably, you lose more heat when surrounded by helium than by air, because heat conductance is the major factor in skin heat loss. Therefore helium is not used in dry suits. However, respiratory heat loss depends on heat capacity, and not at all on conductance. The thermal capacity of helium per gram is higher than that of air. However, there are fewer grams of helium for the same volume breathed because it is far less dense, making thermal capacity less compared to the same volume of air. Less heat would be lost breathing helium, so it should not chill you to breathe, as commonly thought. In a helmet or full face mask, your face may feel cool, making it hard to separate out the lesser loss through breathing.
Depth affects gas density, and so, heat loss through the breathing medium, and to be more confusing, you also need to account for interactions of respiratory heat loss through convection and evaporation. With helium you may also be more aware of the cold that is so common in diving, than when dulled by narcosis while breathing non-helium mixes. Remember too, it is generally not feasible to breathe air at depths where helium is used, so hard to compare in actual use. The short answer seems to be that breathing mixtures of helium at depths encountered by technical divers does not seem to result in greater cooling than breathing air. Helium feels colder to your skin than air, but it carries away less heat when you breathe it."
I was taught back in the 80's when I was in a commercial diving academy that heat loss was greater when
breathing helium. But after a bit of research learned that is not exactly correct. Just like argon does not insulate us better because it is "denser". Here's a link to someone who can explain that a lot better than I can...
http://www.decompression.org/maiken/Why_Argon.htm
like I said earlier, a bit off topic but beneficial knowledge nonetheless.
dive safe!---brando