If you're only worried about swimming -20 or so lbs up from depth with good fins and average or better leg strength, I say knock yourself out...it's just not that hard, especially since your suit will expand as you ascend, cutting the negative weight. I know for a fact I can kick full double 130s up from 100'+ if I need to, and I NDL dive steel doubles in little to no exposure protection accordingly (though I always have a large, closed circuit SMB on me because I need it for emergencies unrelated to wing failure).
If you're getting into any kind of deco schedule, forget it--some kind of redundant buoyancy that will allow you to complete your stops should the wing fail is absolutely mandatory. That means an extra wing, a double bladder wing, and/or a SMB with sufficient lift and a closed circuit/OPV design for reliability and easy venting. I deco dive wet in steel doubles, but only with at least one form of redundant buoyancy that I've actually used to complete an ascent schedule before.
It also depends on where you're doing this; I wouldn't attempt it on a wall dive where there's no relevant bottom.