First off, make sure the latex seal is trimmed to the right diameter. Properly sized latex seals are not very comfortable. If you don't notice the seal on land, it's likely to leak underwater.
Second, review your weighting and your trim. If you are overweighted and having to carry too much air in the suit, and especially if you then dive in a head-up position, you will burp air out the seal, and that results in leaking.
Silicone seals can be tight enough to seal without being as uncomfortable, because they are softer, but they still have to be tight. Neoprene seals are quite likely to leak if you turn your head a lot, no matter how they are sized.