Given that normally divers are really cautious regarding health issues, yes, I suggest to use the same caution when working with lead. It is not instantly a huge hazard, true. But especially of you do it often, there are more healthy things in life than casting lead weights in an improvised fashion.