Sometimes I regret it when I hear people talking about the cool capabilities that their CCR gives them. But I also enjoy the times when I am ready to relax with a beverage at the end of the day, and they’re still messing with their rig.
It's not really that bad. I'm usually drinking an adult beverage when building my rebreather.
In all seriousness, rebreather build + breakdown is more time consuming than open circuit but not horribly so. I would argue a lot of us can build/prep our rebreather for the next day in about ~15-25 minutes. Of course troubleshooting takes significantly more time than that if you run into an issue.
Back to the OP's question. While I do a fair bit more than 50 dives in a year I do believe that number is a good metric for keeping up with proficiently but honestly that number is quite ambiguous. For me it would depend on the parameters of those dives (depth, runtime, environment, deco obligations, gases used, etc).
When I first started diving CCR I pretty much dove my unit on every dive to build hours, proficiency and muscle memory. As a result my OC skills lapsed quite a bit. For the sake of argument let's call ~50 hours/50 dives your first season on a rebreather. I see a lot of rebreathers go for sale used online with less than that number. To me ~50 dives is basically MOD1 course (entry level rebreather course and maybe a couple weeks of CCR diving). My point here is don't take the 50 dive number too literally but if you are thinking of switching to CCR you should devote some significant hours/dives to developing good habits.
Nowadays I try to split my time equally between CCR and OC but I realize that can be difficult if you're only doing 50 dives a year.
For me a two hour dive in Ginnie (you mentioned CCR cave diving ambitions) does not require a rebreather. This is an arbitrary number but I tend to dive rebreather on dives over 2-2.5+ hours or deeper than 130ft for helium savings. I like diving OC so I have realistic idea of my SAC rate and how long it takes me to swim places on OC so I can use those numbers for CCR bailout planning.
I feel it's really important to have good OC skills (bottle handling with stages + deco gases) before one switches to CCR. It's been stated here on Scubaboard a million times but when your rebreather fails and you have to bailout you become an OC diver again. You better hope you're comfortable managing multiple bottles potentially thousands of feet back in a cave, doing multiple gas switches all while managing the air space of your failed rebreather (counterlung) along with your drysuit, wing, etc.