Like most things in life, I am going to go against the stream on this one.
I am a firm believer in finding the config that will fit you best for the diving you want to do. If you want to cave dive in the SM config, start now. Do not let anyone tell you that you need 100 dives or any other arbitrary numbers they make up.
I did my SM course at the same time I did my RAID Advanced 35 course. I may have had 15 or 20 dives at that point. You see, SM is where I new I wanted to go due to the diving I planned on doing. Plus, I am a big fan of having my gas sources independent of one another.
Was the SM course harder because I was still working on propulsion techniques like frog kick, back kick and helicopter turn? No it wasn't. At no time was I ever out of control even though I was still working on my trim and buoyancy.
It did allow me to feel more comfortable in the water, which not many talk about. You see, when you are new and doing dives in the 30-35m range...nothing makes you feel more safe than having that extra source of gas that is 100% independent of your other gas source. As a matter of fact, I did my first deep dive for the ADV 35 course in single tank bm. The entire time my brain was like this " How much gas do I have left?...ohhhh a wreck...how much gas is left...is that enough to get me to the surface...think Turk...Puffer fish!!! " LOL
My second deep dive for the course as done in the SM config. That dive was incredibly relaxing I still checked my gas as I was supposed to but I was able to relax a bit more knowing I could rely on me to save me in the event of an issue with one cylinder. This allows me breathe more like a normal person rather than sucking down gas like on the previous dive. My SAC rate was actually lower on the second dive.
Comfort is a big part of diving and safety tends to make most of us more comfortable.
I say go for it but be sure to find a great instructor. SM instructors, like all instructors, are not equal. Find someone who dives SM when they are doing fun dives and not just when teaching. Ask to see videos of past students and post questions here about where to find a proper sidemount instructor.
I am always happy to help if needed