Well, that was pretty much what I did. I took a sidemount course with about 50 dives, in preparation to do cavern / Intro soon thereafter in Mexico. I struggled a bit, but the problem was mostly with the sidemount course, which had been taught by an instructor who didn't cave dive and my cave instructor didn't approve of some of the things I had been taught. So if I would do things differently, I'd take a better sidemount course from a cave instructor. I would not want to start with doubles and then switch over shortly thereafter. Nor would I want to take cavern in a single-tank configuration. That's why I would start with the end goal of diving caves in mind, and get proficient with an appropriate configuration for that as soon as possible, and then get started on cavern and cave training.
Backmount doubles are a very appropriate gear configuration for cave diving. People have been cave diving for years in backmount doubles and the safety record is much higher than sidemount.
Name one person that has died from a manifold failure. You can't, it hasn't happened.
Now let's talk about how many people have drowned doing gas switches in sidemount. We can start with one about 11 months ago here in Florida...