Before I took up diving I used to do a lot of backpacking. My pack usually weighed about 35 pounds. After a day of mountain climbing I was exhausted. I began diving and suddenly found myself carrying 80-120 pounds of gear on my back, hiking in a drysuit and entering surf from a rocky beach. After spending 60-120+ minutes in 50F water +- a few degrees, I would climb back out of the water, up the hill and plop down on the tailgate of my truck. I feel the same after a dive no matter what gasses I use.
I agree with this as well but for me it's a different feeling. With one I'm tired as my muscles are sore...for good reason but if you feel like sleeping for 12 hours after two recreational dives with easy shore entry that's a different matter. Also, if slowing the ascent and spending more time shallow doesn't clear things up then you probably are just tired from all the other factors mentioned.
It's two different problems/situations in my experience. Maybe all of your entries are tiring (Calif diving) and maybe your ascents are already slow and you already spend a bit more time shallow then needed so maybe you've always avoided this problem in the first place. I don't have this problem anymore either.