Don't worry about it. But, just give it a go. Your Instructor will know if your ready.
Maybe I misunderstood the OP's question. Sam appeared to be asking Scubaboard whether he is ready, not asking a Cavern instructor whether he is ready. My advice to him was in fact to consider doing the dive escorted by a qualified guide or instructor. That is a very different matter to doing the dive with another person of similar inexperience.
I actually recommend (and I wish I had done it) the Open water and advanced to be taken together.
This may work for you, but I am happy I followed another course of action. I realize everybody progresses at a different rate, and also that everybody has a different set of choices for their AOW dives. In my case, I did my AOW in Brockville. Every dive was a drift dive, and by drift I mean river current with boating and shipping channels. Every dive was a wreck dive. And if you float off the wreck, you need to deploy an SMB. So when I did my deep dive, I was simultaneously handling depth, navigating around the wreck back to the mooring line, and the river current. Did I mention that learning to use a pony bottle was a requirement on that particular deep dive? And that while the water was warm by Ontario standards, I was wearing heavy layers of wet suit that required lots of compensation at depth?
All together, there was a lot going on for me in my AOW long week-end. YMMV, but I'm glad I followed the path of getting my OW, then my nitrox, then my AOW. Getting back to caverns, if I was going to dive a cavern I would sign up for a cavern course with an agency I trust. I had 40+ OW dives before I felt comfortable walking onto a charter and doing a wreck survey dive without a divemaster or instructor keeping an eye on me, and I would probably want a little guided experience before doing the same in a cavern.
Again, that's just me. getting back to the OP... If I was going to be down there I would absolutely want to try it. But touching back to my original advice and Bobby's advice, find a local dive shop or instructor and ask them if they can guide you in there.
If they refuse to take you, trust their judgment and come back when you have more experience. If they take you, trust that they can guide you through it safely, follow their instructions closely, and have a great time. if caverns interest you and your wife, go one step further and ask the instructor to do your cavern specialty. You can probably apply the dives to your AOW if you choose to follow that route.