Check and make sure you have the face priority disabled. I've found that if it is enabled and there is no face in the image, it tends to search around looking for one. Also, if you are using c-af tracking, it will search around a lot. If you want to use c-af and not have it search, be sure to...