You can try to put a half twist in the belt when you thread it through the weights. Thread through one slot, twist, then thread through the other. You can also get weight keepers to keep everything in place.
I think your idea would work to keep them in place, just be careful drilling the...