I don't agree, yes you will get lots of boats on Elphinstone but only on the first and last days of the trip. This is because it is the nearest area to Port Ghalib where the southern boats start from.
I have done the Brothers, Elphinstone and Deadalus in 2004 and St Johns in 2004 and 2005...