I don't think that diving without ditch able weight is a problem assuming you can build a balanced rig. A balanced rig is defined as something that you can swim up if your wing fails from the deepest and heaviest part of your dive while also being just heavy enough to hold your shallowest stop with all of the gas out of your wing.
The heaviest point of your dive is when the tank is completely full and you're at the max depth because of neoprene compression, and the lightest spot is the end of the dive at your shallowest stop.
Assuming you can satisfy those 2 criteria and I imagine you could with only 6lb of weight needed, then you don't need to worry about ditching weight, but it's easy to test next time you're out.