A full suit, in the M - L zone, is about 2lb per mm. Switching from a 5/4 to a 3, IF both are in reasonably good condition, would be around a 3 lb swing (3 lb less lead).
The bigger swing is freshwater, quarry, to salt. If the tank size/type is the same (diving an AL80 in both cases, for...