The size of the pony that will work for you depends on the amount of risk you are willing to take on.
6 cf will get you to the surface from 100 feet. You will have no time at depth to solve a problem, and you will be skipping most if not all of your safety stop depending on your ascent rate, but as someone pointed out it is significantly safer than a CESA which is what you will be doing if you don't have it and your buddy is unavailable. But there is 0 margin for error and probably carries a false sense of security. It is what I use in clear tropical water - diving solo but I am aware that it only solves a very limited number of problems and if something not on that list happens - I won't be coming back (the risk I am willing to take on). I am also rarely at 100 feet, most of my diving in that environment is at 50 - 80 feet and I am entirely comfortable doing a CESA from that depth.
A 40 will allow you lots of time to solve a problem at depth, do a leasurly ascent, safety stop and then some. If you are diving where the entanglement risk is high, even that much gas might not be enough. There is always some problem that you might need to solve that will take more gas than you have. We all take that risk every time we dive. Fortunately mostly the risk is extremely small.
You have to decide what problems you are trying to address, how much time you want to spend at depth resolving those problems, what kind of ascent rate you are comfortable with and how long a safety stop you want to do. Once you know the answers to all of these questions you will have your answer. Without your answers to these questions everyone is just giving their opinions based on their own answers.