My guess is that you aren't in fact neutrally buoyant when you start to play with the SMB. The difference is that normally you will vent the gas from your lungs before you start to drift up and this will correct the added buoyancy of your full lungs (keep in mind your neutral buoyancy is the average of your breathing and you change this when you don't include the low volume part of the cycle).
Concentrating on the SMB also may mean you pay less attention to your position in the water column and by the time you've finished with the SMB you may have moved up enough that the volume change is accelerating your buoyancy change.
Concentrating on the SMB also may mean you pay less attention to your position in the water column and by the time you've finished with the SMB you may have moved up enough that the volume change is accelerating your buoyancy change.