Well, I'm back! I have to say I was a little disappointed, but I think we probably missed the best parts. We never found the ledge, just cruised along a sloping bottom at about 70 ft. We saw some big rocks, but no ledge. We saw lots of interesting things, like a Lewis's Moon Snail and a Red Irish Lord, but no GPOs or wolf eels.
We entered the water about 90 minutes before high slack at the Narrows, like one of the books recommended. There was hardly any current, but it flowed consistently North the whole time, so we had to swim against it to get to the tree, and we drifted with it, but didn't make it past the second house on the way back.
The walk was a little bit further than I'd like, but it wasn't the worst I've seen by far. I took all my gear down there before I put my suit on, but carried it all back up in one trip.
Bottom line: I'd go back.