Yeah, for sure. I got an Otterbox and waterproof headphones and solved most of that problem. I also enjoyed the time to think.
I don't know how much lap swimming really helps though. I have always been a strong swimmer, but I grew up playing in the ocean all summer long. Surfing, snorkeling...