Can't believe that you've never been to the caves, though! They're high on the list of places to go and show.... though don't expect too much if you do go to find them, there's just a bit of topography as opposed to flat silt. The "caves" are more shallow overhangs than anything else - but a good place to spot some eels, particularly at night.
The foolsafe way to find them is from the bouy (at the pumphouse) head towards shore until you hit the weedline. Follow it left until you reach a big pointy rock, shaped a bit like the Matterhorn. Do a 90 degree turn left until at about 14m depth there's a few boulders on your right - turn along the base of these to find the caves.