Numbers make sense, at least very similar to my experiences as well. I'm a couple inches shorter and a couple pounds heavier, and when I'm in AL backplate and an AL80, I'm putting on like 4-6 pounds of lead. Even in 85 degree water I'm using some type of exposure protection, whether thats a 3mm shorty or a thicker rashguard and some neoprene tech shorts. If I wasn't wearing any neoprene at all, I'd be pretty close to neutral in Aluminum. If I dove with no exposure protection and a HP steel tank, I'm sure I'd be slightly negative too, even at 500psi.
Being slightly negative at the end of the dive when you're doing a safety stop is far better than being slightly positive.
All of the suggestions of double-checking your weight needs at like 15 feet with an empty BC and near-empty tanks are spot-on. Once you're sure of your weight requirements wearing your birthday suit, then figure out how much lead you need to add for any exposure protection options. Don't bring any extra lead than you'll need, and if you're diving without any redundant bouyancy, make sure you're less than 10 pounds negative and you can swim it up with full tanks, compressed neoprene, and an empty wing. I'd also suggest using weight pockets that you can easily empty at the surface if you find yourself with a completely unusable wing.
In that scenario where you're -3 pounds at the surface with no lead and no exposure protection, maybe try a pair of neoprene tech shorts? Having pockets to stash your SMB, spool, and gopro is awfully nice.