So this is a bit complicated, but hopefully this makes sense and is how I would do it.
Priority #1 to me would be for you to get banks for nitrox and put a blend stick on your compressor. Gives you a ton of advantages. Compressor doesn't have to run when you fill, don't have to O2 clean tanks/valves, more efficient use of O2, etc etc. Bank EAN30/32/36 or whatever is most convenient for your non-trimix diving locally. That solves your O2 going thru the blend station.
Your panel then becomes Nitrox only with valves for each of the banks and however many outlets you want.
Priority #2 would be to find a booster. Booster can double duty for He, and O2, and can either be driven by a shop compressor, or your nitrox banks. Shop compressor is cheaper if you have one available.
Your booster has a pair of inlets, one O2, one He, with a single outlet.
The O2 bank is fed into the booster when you need anything higher than the mixes provided by your nitrox stick. Typically 100%, 50%, and 30/30 if you dive that mix. Once those are depleted, you then move that bottle over to the nitrox stick where the compressor can scavenge all of the O2 out of the bottle from a regulator. Gives you most efficient use on the booster, and you can use all of your O2.
He is self explanatory, but you get to use all of it and don't have to drain any tanks down if you need to change mix.
Keeping the nitrox fill station separate from the boosters is IMO a good idea. It may sound a bit less streamlined, and it does make it a 2 step process, but I think it is well worth it.