Well after reading everyones replies, I went and did some searching of my own. I believe that water quality degradation and fishing are both major causes of fish depletion. I found a study on refuges that contain significant amount of mangroves that told me exactly what I wanted to know. Heres the abstract below. Basically it says that refuges that arent fished contain 2 to 12 times the fish as refuges that are fished. And that these fish do end up swimming to other areas later on in life.
Abstract.Approximately 22% of the waters of the Merritt Island National Wildlife Refuge, which encompasses the Kennedy Space Center, Florida, have been closed to public access and fishing since 1962. These closed areas offer an opportunity to test the effectiveness of no-take sanctuaries by analyzing two replicated estuarine areas. Areas open and closed to fishing were sampled from November 1986 to January 1990 with 653 random trammel-net sets, each enclosing 3,721 m2. Samples from no-fishing areas had significantly (P < 0.05) greater abundance and larger fishes than fished areas. Relative abundance (standardized catch per unit effort, CPUE) in protected areas (6.4 fish/set) was 2.6 times greater than in the fished areas (2.4 fish/set) for total game fish,
2.4 times greater for spotted seatrout Cynoscion nebulosus,
6.3 times greater for red drum Sciaenops ocellatus,
12.8 times greater for black drum Pogonias cromis,
5.3 times greater for common snook Centropomus undecimalis, and
2.6 times greater for striped mullet Mugil cephalus. Fishing had the primary effect on CPUE, independent of habitat and other environmental factors. Salinity and depth were important secondary factors affecting CPUE, followed by season or month, and temperature. The importance of specific factors varied with each species. Median and maximum size of red drum, spotted seatrout, black drum, and striped mullet were also
significantly greater in the unfished areas. More and larger fish of spawning age were observed in the unfished areas for red drum, spotted seatrout, and black drum.
Tagging studies documented export of important sport fish from protected areas to fished areas.
http://afs.allenpress.com/perlserv/?request=get-abstract&doi=10.1577%2F1548-8675(1999)019%3C0436:EOAEEN%3E2.0.CO%3B2