I agree with the above posters. Plaster casts are extremely susceptible to water damage, and even the fiberglass ones suffer if they are very wet or wet for any period of time. You really don't want a cast to become loose or misshapen, because it's what's holding the bones in proper alignment (unless you had surgery). Particularly with hand injuries, you WANT those bones to heal correctly, because even small discrepancies can have significant effects on hand function.
In addition, you would have limited mobility and ability to react in an emergency -- if you got the other hand trapped in something, could you use the broken one to free it?
Luckily, the period of immobilization for most hand injuries is only a few weeks. (Although if somebody told me I'd have to take a few weeks off from diving, you'd hear me howling in Florida!)
Good luck with your recovery.