I think you have to ask yourself:
"Am I going diving for myself, or beacuse of others", is it an escape to or an escape from. I sometimes have a similair feeling that I am abandoning my wife and kids (well kid, one dives), to go diving. What I have come to realize though, is that I only feel guilty because diving is just sooooo damn rewarding.
Without anyone knowing what your situation, is I doubt there is that much advice to offer. If you didn't dive would you be finding some other reason to take off? If the answer is No, then my guess is you just love diving and are feeling a little guilty. Find a way to make those you are leaving behind "okay" with your departures and eveything will be cool.