A few in here have already touched on this point, but I think it should be emphasized a bit more.
Unlike 99.9 percent of all munitions manufactured, depth charges were NOT intended to destroy a submarine by kinetic energy. In other words, no one ever attempted to destroy the sub by shrapnel. Instead depth charges were meant to kill a sub by concussion. As everyone knows, water isnt compressible. Hence, the concussion generated by the depth charge was the lethal force, not the shrapnel like a conventional bomb.
Carrying this to the logical next step, if a diver were to pick up a 5 inch shell and subsequently drop it on the deck of the wreck, probably the least worry he would have would be the shrapnel internal injuries from the concussion alone would kill him/her faster than other, more obvious wounds.
Also as a round corrodes, various areas become saturated at different rates, depending on the amount of corrosion, the place where the corrosion takes place and the type of fill (TNT, Torpex, etc etc) employed. This makes them inherently unstable.
An earlier poster is correct, the most fragile part of any weapon is the fuse it will corrode first. But just because the fuse is totally gone does not, in any respect, render the round safe to handle far from it.
As long as people throw a SCUBA cylinder on their backs, people will pick up UXO. However, I think the best way to prevent yourself from becoming a statistic is to make sure everyone in the dive party knows the dangers and agrees not to touch a bloody thing underwater otherwise, they may have an uplifting experience!