The people who comes to the north, work their a$$ off, frequently as slaves, frequently get nowhere. If they do eventually move out of indentured servitude though multiple generations, the mostly likely place in society they will reside is as wage slaves.
I know there are classism in all places, but they are usually isolated to different places. You could have the fat cats spending their summers in the French Riviera, but rarely have seen the separating of classes openly visible in the same resort. I don't see VIP class on the dive boats.
So you are ok with different classes, as long as the patricians don't visit your bunk in steerage? Are you posting from a tent on Wall Street?