It is great that you can take time during the week to do your dives but I wouldn't leave the kids out. I introduced both my sons to the ocean 4 days after their bellybutton healed up, they are also 3 years apart in age. They both could swim and float before they managed to walk. Mask, fins and snorkel were used almost every weekend, as they grew older there were buggy boards, skin boards and surf boards; but mask and snorkels were always used, free diving for stuff, swimming races and your typical horseplay in the water. By the time they were old enough to scuba dive they had no problem with any of the basic skills, they were familiar with dive gear because they grew up around it. Later on they went on to other activities but every so often they still have time for a dive with mom.
I find diving very personal, and I think that taking time during the week to dive will give you mental peace that will allow you to be a better mom (for sure more patient when the kids push the limits) I know my kids are alive thanks to my diving. But also share your water world with them; in the water with them, there are no video games, no text messages or any other distractions; there is physical activities, exposure to a magnificent world, opportunities to teach them all kind of marine stuff about waves, fish wind, respect to environment, avoidance to weirdos, wow the list goes forever. And all that learning while having fun. The cherry on top way later after they left the house, it will be such a treat when they share a dive with you.
Won't be easy sometimes to fit a dive because you'll have many things to do, but if it doesn't affect the financial picture of the house, make it your weekly obligation, the investment will be worth it.