I used to sail small boats a lot, and saw a lot of crap wash out in to the water after storms. The worst I ever saw was in the south bay after it rained and the Hyperion sewage treatment plant couldn't keep up and just started discharging raw sewage.
A lot of factors determine how long I wait until after getting in the water, at least for me: how long it's been since the last major rainfall (first big rain always washes a lot of crap, literally, into the water), whether there are canyons or just flat areas onshore and up current from where you are in the water, whether it's city or rural / agricultural, and what the swell and tide conditions are like. Rocky coastlines are a lot better than dirt, and coastal landslides are bad. I always err on the side of not being one of the first ones into the water after it rains
When we're in the tropics, where it rains all the time, it doesn't matter nearly as much how long you wait compared to so cal