I think if you are doing a dive where the bottom is not below the rec limit, then sure, you could do it. For example, if the bottom is only 130 feet or something, or at most like 140-ish (or perhaps slightly more than that like 150-160 if you know for a fact you handle narcosis well or dont get it), then you could do it and you wouldn't need to worry about dying or getting nitrogen narcosis (at least not bad enough to where you wouldn't be able to exit).
However, if you dive where the bottom is like 300 feet or more, or something like that... (Something where you DONT want to have a runaway descent situation cuz its too deep).. Then you put yourself in a VERY dangerous situation by not having an inflator hose!!!
For example, let's say that for some reason your BCD has become under inflated (happens to me sometimes when diving) and then you need to inflate it. But, you can't inflate it fast enough to stay level. You keep sinking, trying to inflate, but you are getting out of breath as you sink farther down. You will then be forced to have 2 options:
1) quickly dump your weights and then you'll float to the surface, but perhaps too rapidly and then suffer DCS
2) Keep your weights on, and try to do it manually... and fail... and then float all the way 300+ feet down... and die of nitrogen narcosis, oxygen toxicity perhaps, or both at those depths. Plus the pressure will be insane, and even if you were able to avoid narcosis and toxicity and inflate (extremely unlikely), then you simply wouldn't have enough air to make it back to the top, at least not with doing the proper deep stops and safety stops required at that depth.
You would 90% likely die from something, or at best, maybe if you are super lucky, only suffer severe assss DCS and probably be crippled or something.
My suggestion: Keep your power inflator unless you are in shallower waters (140 or less) or if you have a deathwish.
If you are in a cave, you actually dont NEED a power inflator if you are super super good with your buoyancy... although, it makes it much easier to maintain buoyancy with a power inflator, and the last thing you want to have is a silt-out in a cave because you started sinking and couldn't reach your buoyancy manual inflate valve quick enough. Trust me, I HAVE BEEN IN A SILT OUT ONCE... NOT FUN. And more importantly, even when you remain calm, it's tricky finding your way out and obviously, dangerous. (You could get lost, lose your line, run out of air, bonk your head on a rock and knock yourself out, etc).