You may notice it more if you dive a wetsuit. In a drysuit you can easily shift the bubble around to remain stable if you have differing weight on either side of your body (due to all the crap you have clipped off). In a wetsuit, you have to rely on the wing/bladder to provide additional loft (initially on the side the pony is on when full, then on the opposite side if it's empty).
Looking at cylinder specs, both the Luxfer and Catalina AL19s are neutral when empty and -1.3lbs when full (with the valve included). That means you'll be a little over a pound heavy on the side you sling the bottle. I would try just keeping more air in that side of the wing/bladder. If you are trying to stay still in perfect trim, and pushing the bubble to the side of the wing the pony is on isn't working, you could shift a pound to the opposite side. That will even you out when it's full, but you'll run back into the same problem when it's empty.
Really though, 1lb shouldn't be that significant.