It can be an adjustment issue.
And it could be too loose or too tight, depending on your body type.
I have dived 7 different SM systems, including two that I made myself.
One is based on a single piece of webbing harness. That one is the closest to the Ray setup. I've only tried it with smaller cylinders, as it has a deco sidemount bladder with minimal lift without using a chest strap, and it moved too much like you are describing.
So I added the chest strap to it.
A couple of questions I have are, what cylinders are you using it with? Steels? Aluminums?
How big is your chest, and what's your body type?
How many other rigs have you used?
Which ones?
Unless you are working with an instructor who knows how to set up the system, if you don't have a fair amount of experience with different rigs, it could take a while to figure it out.
That's where having an instructor experienced in the system or one similar can save you a lot of time and effort. An hour in the pool with them might save you days trying to set it up yourself. I had a number of students come to me with their rigs saying they didn't work or were ready to get something else.
Except for the guy with the junk ScubaPro rig, I was able to tweak their setups in one pool session to work for them.
The SP one should have been burned in the factory. Not enough adjustment built into it, lousy shaped wing, poor attachment points, etc. Complete garbage.
Even the Hollis SM100 pig could be made to work well enough.
You may need to add a chest strap if your chest is larger or your shoulders have some slope to them.
Adding a chest strap may feel like "cheating" but the beauty of SM is that as long as it doesn't affect your safety, making a mod like adding a chest strap is not a big deal. Nor is it wrong.
I used to assemble BPWs when I was selling gear, and more than a few times had to modify Hogarthian harnesses for people who were barrel-chested, or they would not have been able to reach their backup lights because they were so far under their arms.
I had to add chest straps for thinner people or cross the straps so that they didn't slide down off their shoulders.