When did you make the switch to doubles? What are the factors to make this switch?
I made the switch when I started technical diving training. I do most of my recreational dives in doubles too.
I guess your question is really also about
when to switch to doubles. I guess my answer would be get very comfortable with your single rig first, then ask yourself whether you
need to switch. Sure, one can dive doubles for the sheer heck of it but if I'm diving doubles but all my buddies are on singles then it may be bit pointless.
There are many benefits to using doubles (I use manifold doubles, btw) - redundancy, ease of use (no need to switch tanks between rec dives, rig can be set up at home) and stability are a few I can think of.
There are drawbacks also. I do feel that one does need some training/informal instruction and practice in order to dive with doubles safely. Training/practice and an experienced buddy are crucial ingredients for dealing with failures (valve(s), manifold, etc) underwater.
Since I took the rescue diver class I've realized having extra air is a good think in case an emergency comes up.
Having "extra air" or enough gas to get you and another diver to the surface (in a controlled manner with enough gas to do necessary/minimal stops along the way) is really all about dive planning and setting aside the necessary gas reserve. This is true whether you are diving singles, doubles or more bottles.
How challanging is the switch with regard to buoyancy?
Strapping on and diving a doubles rig was for a bit strange initially. For me, the biggest change was adapting to having to actively add/dump gas to/from my BC as opposed to just using my breathing for buoyancy control like I did with my single tank setup. I was lucky to have more experienced divers accompany me on my initial dives and they helped me get comfortable quickly.